DoorDash Senior Data Engineer Interview Preparation Guide

Requirements & constraints:

• Must respect country/region data-residency and consent laws (e.g., EU GDPR, Brazil LGPD, China PIPL).
• Minimize cross-border transfer of raw PII.
• Enable global product availability with staged rollout.
• Provide central analytics without moving raw data.

Phased rollout / region sequencing:

1. Pilot (single low-risk region with flexible regs) to validate infra and workflows.
2. GDPR-region (EU/EEA) next — high compliance bar forces robust controls early.
3. Other regulated regions (UK, Switzerland, Brazil) in parallel.
4. High-restriction regions (China, Russia) after local legal & infra readiness.
5. Remaining markets.

Architecture: federated with isolated region data stores

• Per-region isolated data stores (cloud-region-specific accounts/AZs) to guarantee residency and ease audits.
• Federated control plane: central metadata/catalog, policy engine, and orchestration; no raw-data movement.
• Use immutable event streams locally (e.g., region Kafka/MSK) and local data lake/warehouse (S3 + Athena / ADLS + Synapse / GCS+BigQuery in regional projects).

Consent capture & enforcement:

• Capture consent at point-of-collection with standardized consent schema (versioned).
• Store consent tokens and provenance locally tied to user ID and region.
• Central policy engine maintains consent rules; enforcement happens in-region (via SDKs or API gateways) that check local consent store before processing or sharing.
• Expose tokenized consent proofs to central plane (hashes/consent-IDs) — no PII leaked.

Handling schema divergence:

• Enforce a core canonical schema (stable, minimal required fields) across regions.
• Allow region extensions via namespaced fields and schema registry (Avro/Protobuf) per region.
• Use schema registry supporting compatibility checks; conversion adapters in ETL to map regional variants into canonical view for analytics derivatives.

Central analytics via aggregated derivatives:

• Define privacy-preserving aggregates computed locally (e.g., counts, histograms, differentially-private summaries).
• Local jobs produce normalized, aggregated derivatives and publish to central analytics bucket (only aggregates and metadata, no raw records).
• Use techniques: differential privacy, k-anonymity thresholds, cryptographic MPC or secure aggregation for sensitive metrics.
• Central analytics layer ingests aggregates for global dashboards, ML model training on aggregated features, and federated learning approaches where models are trained locally then gradients/weights are aggregated centrally (secure aggregation).

Operational & governance:

• Automated compliance checks, region-specific monitoring, and audit logs stored locally with exportable signed proofs.
• CI/CD for schema, consent policies, and pipelines with staged gates per region.
• Trade-offs: isolated stores increase operational overhead; federated control plane adds complexity but offers compliance and centralized governance benefits.

Example tech stack:

• Regional: Kafka/MSK, S3/GCS/Azure Blob in-region, Spark/Flink for local ETL, Postgres/BigQuery regional.
• Central: Metadata store (Data Catalog), policy engine (OPA), orchestration (Airflow/Kubernetes), analytics (Looker/BigQuery on aggregates), federated learning frameworks.

This design balances legal requirements with ability to run global analytics by keeping raw data local, enforcing consent locally, and moving only privacy-preserving derivatives to the center.

Direct answer
My mentoring approach starts from diagnosing where someone actually is, not applying one fixed style, and it balances technical guidance with career development by treating them as two separate but connected tracks: technical guidance closes the gap between where they are and what the work in front of them needs right now, while career conversations look further out at where they're trying to go. The mix between the two shifts substantially depending on how experienced the person already is.

Structured elaboration

Diagnosing before applying a style

The first move with any new mentee is figuring out their actual starting point and goals, not assuming based on title or tenure. Two people at the same level can need very different things: one might need technical unblocking, another might already be technically strong but stuck on visibility or scope.

Balancing technical guidance and career development

• Technical guidance tends to dominate early in a relationship or when someone's working in genuinely new territory; it's concrete, has fast feedback loops, and builds the trust that makes career conversations land later.
• Career development becomes a larger share of the time as technical competence stabilizes; someone who's already reliable on the day-to-day work benefits more from conversations about scope, visibility, and where they're headed than from more line-by-line guidance.
• The two aren't fully separable in practice: a well-run technical conversation often surfaces the real career question underneath it (they're not struggling with the code, they're struggling with whether this kind of work is even what they want to be doing).

How the approach changes: newer teammate vs. experienced one

• A newer teammate typically needs a tighter structure: explicit expectations, closer review, and a higher ratio of technical to career conversation, because there usually isn't yet a track record to have a grounded career conversation about.
• A more experienced teammate usually needs the opposite ratio: less hands-on technical guidance (often none at all on execution, more on judgment calls and trade-offs), and more time spent on career and scope, sometimes including the expectation that they take on some mentoring of their own, since that's often the actual next step in their growth.

Worked example

Applying the philosophy

With a newer teammate, most of an early 1:1 might genuinely be spent walking through a specific technical decision they made, only pivoting to career topics once they'd built enough of a track record to have something concrete to talk about. With a more experienced teammate on the same team, the same 1:1 slot might be spent almost entirely on a scope or visibility question, with technical guidance limited to a quick sanity check on a hard trade-off they'd already mostly worked out themselves.

Signal of it working

The clearest sign the ratio was right in either case wasn't a specific number, it was whether the conversation actually used the full time productively: a newer teammate's 1:1 running long on technical questions because they had real ones was a good sign; the same happening with an experienced teammate, repeatedly, usually meant something else was being avoided, often a harder career conversation neither of us had opened yet.

Trade-offs & pitfalls

• Applying the same ratio to everyone regardless of experience. A fixed philosophy that doesn't flex by seniority isn't really a philosophy, it's a script, and it under-serves experienced mentees while potentially overwhelming newer ones.
• Letting technical conversations become a permanent default because they're easier. Technical questions have clear right answers and fast feedback; career conversations are ambiguous and can feel uncomfortable. A senior mentor notices when technical talk has become an avoidance pattern rather than what's actually needed.
• Treating career conversations as an occasional add-on rather than a real track. If career development only comes up during formal review cycles, it usually means the day-to-day mentoring relationship isn't actually addressing it.

Requirements clarification:

• Country-level residency: raw PII must never leave originating country/region.
• Central analytics: only aggregated, non-identifiable metrics allowed centrally.
• Ingestion: 5 TB/day (assume global aggregate).
• Minimize cross-region egress and provide auditable controls.

High-level architecture (regional-first, central-aggregate):

1. Edge ingestion (per-country/region)

   • Regionally deployed ingestion layer (APIs / Kafka / Kinesis / PubSub) collects events and writes to a regional hot store (object store + streaming).
   • Ingest -> regional raw data lake (S3/GCS/Azure Blob in-region) partitioned by country/tenant; use Schema Registry for consistent schemas.

2. Regional processing & enforcement

   • Regional ETL (Spark/EMR/Dataflow) performs:
     • PII detection & classification (DLP)
     • Tokenization / encryption of direct identifiers using region-specific Customer-Managed Keys (CMKs)
     • Anonymization/aggregation transforms that produce GDPR-safe metrics (differential privacy, k-anonymity, or hashed buckets + noise)
   • Raw PII stays in regional cold store with strict access controls and retention lifecycle.

3. Central analytics tier

   • Only aggregated, non-identifiable datasets (pre-approved aggregates) are exported to central analytics region(s).
   • Central analytics runs OLAP/warehouse (Snowflake/Redshift/BigQuery) fed by regionally-pushed aggregated files or federated queries to regional read-only aggregate endpoints.
   • Use message queues or signed manifests to transfer aggregates; all transfers are of non-PII.

4. Governance & audit plane

   • Global catalog (read-only metadata of data locations, lineage) with per-region access controls (Lake Formation / Data Catalog).
   • Audit logs (ingest events, key usage, data access, transfers) forwarded to a secure SIEM; retention policies per regulation.
   • Policy engine (OPA or cloud policy service) enforces residency before any transfer.

Partitioning strategy

• Primary partition by country_code and ingestion_date for fast residency isolation and retention enforcement.
• Secondary partition by tenant/product for parallelism.
• Use region-local shards for streaming topics to align with country partitions.

Encryption & key management

• TLS for in-transit.
• At-rest encryption using region-scoped CMKs (KMS) per country; keys cannot be exported.
• Envelope encryption: data encrypted with data keys; data keys encrypted by regional CMKs.
• Central aggregated datasets use separate CMKs and are audited; aggregated exports are verified to exclude PII before encryption and transfer.

Replication strategy

• Raw PII: no cross-region replication. Replicate within-country zones only (AZ-level) for durability.
• Aggregated metrics: replicated to central analytics region(s); use asynchronous transfer after anonymization.
• Metadata, schema, and audit logs: replicate globally but strip PII; maintain provenance linking to region-of-origin.

Handling 5 TB/day & cost optimization

• Ingest partitioning + parallel streaming clusters sized per regional load; use autoscaling.
• Minimize egress by:
  • Doing heavy transforms regionally and only transferring compact aggregates (orders-of-magnitude smaller than raw).
  • Where possible run federated queries (BigQuery/Athena federated connectors) so central compute pulls only required aggregated slices.
  • Use compression (Parquet/ORC, snappy) and columnar formats.
  • Batch transfers to avoid per-request overheads.
• Trade-offs:
  • Centralized compute (easier analytics) vs regional compute (lower egress). Prefer regional transforms + smaller central aggregator to minimize egress.
  • Stronger anonymization (more privacy) may reduce analytic fidelity — choose DP parameters with stakeholders.
  • Cold storage vs more frequent recomputation: store raw PII regionally cheaper vs replicating; recompute aggregates when needed.

Auditable controls & compliance

• Immutable audit trail: log every ingest, access, transfer, key usage (CloudTrail-like), stored regionally and archived to WORM storage.
• Role-based access + attribute-based policies; approvals and data export workflows with recorded attestations.
• Periodic policy enforcement audits, automatic tests (canary queries) verifying exported datasets contain no PII (DLP scan + statistical checks for uniqueness).
• Retention & right-to-erasure flows: implement delete/expire pipelines that purge raw and derived datasets within retention windows and propagate to catalogs & indexes.

Why this design

• Ensures legal residency by keeping raw PII in-country and enforcing key separation.
• Minimizes egress by designing regional-first transforms and transferring only compact aggregates.
• Supports central analytics while preserving privacy via cryptographic controls and anonymization.
• Provides full auditability and policy enforcement with minimal impact on analytical capability.

Implementation notes

• Use infrastructure-as-code for reproducibility; automated data classification and unit tests for anonymization guarantees; SLA-driven monitoring for ingestion/backpressure.
• Choose cloud services that support regionally-scoped keys and fine-grained IAM; if operating in restricted countries, consider dedicated regional accounts/tenants.

Edge cases

• Countries that forbid cross-border transfer of even aggregated stats — maintain analytics within country and provide federated dashboards.
• Real-time analytics needs — implement regional streaming materialized views and send downsampled summaries centrally.

The design: a checkpoint table tracks which partitions are done, and each partition's update is its own transaction, so the backfill can crash and resume without re-doing completed work or leaving a partition half-updated.

CREATE TABLE backfill_checkpoint (
  event_date DATE PRIMARY KEY,
  status VARCHAR,
  rows_updated INT
);

Per-partition loop (pseudocode driving the SQL):

-- for each partition not already marked 'done':
BEGIN TRANSACTION;

UPDATE analytics_fact
SET amount_usd = round(amount * 1.10, 2)
WHERE event_date = :partition AND amount_usd IS NULL;

INSERT INTO backfill_checkpoint VALUES (:partition, 'done', :rows_updated)
ON CONFLICT (event_date) DO UPDATE SET status = 'done', rows_updated = excluded.rows_updated;

COMMIT;

Verified end to end, backfilling a derived amount_usd column across 5 date partitions:

Run 1 (crash simulated after partition 3 of 5):
  partition 2026-01-01: updated 20 rows
  partition 2026-01-02: updated 20 rows
  partition 2026-01-03: updated 20 rows
  !! simulated crash !!
  checkpoint after crash: 2026-01-01, 02, 03 all 'done'; 40 rows still NULL

Run 2 (resume, no changes to the driver code):
  partition 2026-01-04: updated 20 rows   <- correctly picked up where it left off
  partition 2026-01-05: updated 20 rows
  amount_usd remaining NULL: 0

Correctness check: 0 rows where amount_usd != round(amount * 1.10, 2)

Idempotency check: re-running the ENTIRE backfill again (checkpoint cleared to force
a full pass) updates 0 rows in every partition; sum(amount_usd) before and after
the re-run is identical.

Why this minimizes locking: each transaction's UPDATE is scoped by event_date, touching only rows in one partition, so it holds locks on a bounded slice of the table rather than the whole thing, letting concurrent reads (and writes to other partitions, if your workload has any) proceed unblocked. Batching by partition rather than backfilling row-by-row or in one giant transaction across the whole table is what turns a multi-hour table-locking operation into a sequence of short, low-contention ones.

Why it avoids duplicates: the WHERE amount_usd IS NULL guard on the UPDATE means a row that's already been backfilled (this run or a previous one) simply doesn't match the WHERE clause and isn't touched again, so re-running a partition, whether because you're resuming after a crash or deliberately re-running everything, is always a safe no-op on already-completed rows.

Why it resumes cleanly after failure: the checkpoint table is the source of truth for "what's done," checked before each partition starts, and only written after that partition's update has actually committed. A crash mid-partition leaves that partition's checkpoint row absent (not marked done), so a resume correctly retries it from scratch, while every already-committed, already-checkpointed partition is skipped.

Final cutover step: once every partition shows status = 'done' in the checkpoint table, verify the row-count and null-count invariants (as shown above: amount_usd IS NULL should be 0), then drop or repurpose the checkpoint table and treat the backfill as complete; downstream consumers reading amount_usd from this point forward see the fully-backfilled column with no special-casing needed.

Direct answer

I'd start by pulling the itemized billing export (AWS's Cost and Usage Report, or the equivalent from Azure or Google Cloud) rather than the headline invoice total, break the delta down by service, region, and tag, and look for whether the jump is broad (many services up a little) or concentrated (one or two line items driving most of it), because that single split determines whether I'm hunting for a config change or a runaway process. In parallel, I'd put an emergency budget alert and a hard cap or throttle in front of the likeliest culprit so I'm not still bleeding money while I investigate, since stopping the bleeding and finding the root cause are two different, simultaneously-runnable tasks, not a sequence.

Structured elaboration

Step 1: Scope the delta before hypothesizing about the cause
Pull the granular billing export for both months and diff them by service, region, account or project, and tag. A 40% jump could be one resource type up 400% or every service up a proportional amount, and those point to completely different investigations, so this diff is the first thing to run, before forming any hypothesis about root cause.

Step 2: Narrow to the resources actually driving the delta
Match the billing line items with the largest deltas back to specific resource IDs and tags. Look specifically for: new or unusually large resource counts, a spike in a service that's normally near-zero for this account, unattached storage volumes that shouldn't exist, or a jump in data-transfer line items, since egress spikes are a common and easy-to-miss cause of a sudden bill jump.

Step 3: Correlate against what changed operationally, in the same window
Cross-reference the spike window against deployment history and CI/CD logs (did a release ship right before the jump started), autoscaling events (did a scaling policy misfire and hold capacity high), and application/infrastructure logs for anything that looks like a runaway job or a misconfigured batch process. This is where "a new deploy shipped a debug flag that left instances scaled up" or "a scheduled job got triggered twice" typically surfaces.

Step 4: Form and test a specific hypothesis
By this point there's usually a short list of candidates (a specific deploy, a specific job, a specific resource type). Validate each by reproducing the cost against the actual usage: does the resource's usage-hours or data-scanned metric, for the spike window, actually account for the dollar delta observed. A hypothesis that doesn't reproduce the dollar amount isn't the (whole) root cause, and the investigation isn't done until the numbers actually add up.

Step 5: Stop further unexpected spend while still investigating
This runs in parallel with steps 1 through 4, not after them: set an emergency budget alert on the affected account or service immediately, and for the most likely culprit, apply a targeted control (scale the autoscaling group's max down to a safe ceiling, pause the suspect job, revoke a deployment credential if a bad deploy is implicated) rather than a blanket freeze that would stop legitimate traffic too. The goal is to bound the damage on the specific thing under suspicion without causing a second incident by shutting down something that turns out to be unrelated.

Worked example

A team's monthly bill jumps from a baseline of $42,000 to $58,800, a 40% increase, or $16,800 in absolute terms. The billing-export diff shows the increase isn't broad: EC2 (Amazon Elastic Compute Cloud) compute is up only 4%, but a single service, a managed data-warehouse query engine, is up $11,200, and S3 (Amazon Simple Storage Service) data-transfer/egress charges are up $4,100, together accounting for $15,300 of the $16,800 delta, with the remaining $1,500 spread thinly across normal month-over-month growth.

Matching the query-engine spike to usage shows a single ad-hoc analytics query scanned roughly 2 TB of data three separate times during the window, run by an analyst testing a new dashboard against a table that wasn't partitioned the way they assumed, at a rate where scanning that much unpartitioned data repeatedly accounts for the bulk of the $11,200. The egress increase correlates with a deploy that changed a batch export job's destination from an in-region bucket to a bucket in a different region, adding a per-GB cross-region transfer charge that hadn't existed before that release.

Root cause: two independent, unrelated changes landing in the same billing period, an expensive unpartitioned ad-hoc query pattern and a misconfigured cross-region export destination, together explaining $15,300 of the $16,800 (91%) increase, with the remainder attributable to normal growth.

Trade-offs and pitfalls

The most common mistake is jumping straight to a hypothesis (usually "it's autoscaling" or "it's a new feature") before actually running the billing diff, which wastes time chasing a plausible-sounding cause that the data doesn't support. A second is applying a blanket spend freeze while investigating, which stops the bleeding but can also take down legitimate production traffic, turning a cost incident into an availability incident. A third is declaring victory once you've found a cause that "sounds right" without checking that the dollar amount it explains actually adds up to the observed delta. As this example shows, a real spike is often more than one thing happening at once, and stopping the investigation after finding the first plausible cause can leave a second, still-active cost driver running unnoticed.

Approach
I would make checkpointing a consistent snapshot of everything needed to resume training, not just model weights. That means saving model parameters, optimizer state, learning-rate scheduler state, random number generator seeds, current epoch and batch cursor, and any data sampler state.

Design

• Write checkpoints atomically: first write to temporary storage, validate checksums, then publish a manifest pointer.
• Keep checkpoints incremental when possible, but always make the latest one self-contained for fast restart.
• Trigger an immediate checkpoint on spot interruption notice, then resume from the last good manifest.
• Store code version, config, and data version so the run is reproducible.

Worked example
If I checkpoint every 10 minutes and a preemption happens at minute 37, the restart only redoes at most 7 minutes of work, not the whole job.

Why this works
The manifest guarantees consistency, the RNG and sampler state keep replay deterministic, and the atomic publish prevents half-written checkpoints from being used.

Direct answer

Anchor the whole program on baselining first, not target-setting first: measure where the org actually is on a small number of leading and lagging signals, then set targets as a fraction of that measured baseline rather than an invented number, and phase in enforcement gradually so the program earns credibility before it asks for hard compliance. What keeps it from becoming a compliance exercise is that the scoreboard is outcomes, fewer incidents, faster recovery, faster and safer delivery, not activity, percentage of teams that attended training, percentage of services with a badge.

Structured elaboration

What to measure, leading and lagging, and why both matter:

• Lagging (outcomes): incident frequency, mean time to detect and mean time to recover, production bug rate per release. These are what you are actually trying to improve, but they move slowly and are easy to game by redefinition.
• Leading (practice signals that predict the lagging ones): percentage of critical services with basic tracing, metrics, and logging in place; pull-request cycle time; whether critical services have an on-call runbook that has actually been used, not just written. These move faster and tell you whether the program is working before the lagging metrics catch up.

Governance and phasing, quarterly and staged, not a big-bang mandate:

1. Baseline quarter: measure current state honestly on a couple of pilot teams before committing to any target. A program that skips baselining ends up defending invented numbers.
2. Pilot quarter: apply the practice changes, tracing, basic service-level objectives (SLOs, the specific reliability targets a team commits to), continuous-integration (CI) checks, to the pilot teams, and treat the pilot's own before-and-after change, not an assumed industry figure, as the evidence for the rest of the rollout.
3. Expand quarter: roll out to the rest of the org with the templates and tooling the pilot proved out, and introduce lightweight enforcement, CI gates for the checkable items, not a review board for everything.
4. Sustain quarter: fold the surviving metrics into normal quarterly review and retire the special program status. A program that never ends is not a program, it is a permanent tax, and teams notice the difference.

Avoiding the compliance-exercise trap specifically: tie the metrics to things a team would want anyway, fewer 2am pages, less time firefighting, and use the pilot team's own real improvement as the pitch, not a mandate from a council. Never let "percentage compliant" become the headline metric, a team can be fully compliant on checklist items and still have bad reliability if the checklist does not map to real behavior, so keep the lagging outcome metrics as the actual scoreboard and treat leading and practice metrics as diagnostic. Build in an explicit, lightweight exception path so teams with a real reason to deviate do so openly; a program with no legitimate way to say "not yet, and here's why" produces gaming instead of honest exceptions.

Governance structure: a small standing group, a rotating set of senior engineers plus one reliability-focused owner, that reviews the metrics and playbook quarterly and has authority to revise the standard, not just enforce it. A body that can only enforce and never revise loses credibility the first time a rule turns out to be wrong for some team's real context.

Worked example

Illustrative baseline-and-target methodology, shown so a reviewer can reproduce the reasoning, not a claimed historical outcome. Suppose the pilot's baseline quarter measures a current mean time to recover of 90 minutes across the pilot's critical services. Rather than asserting an arbitrary target such as "cut it 40 percent," the program sets the target from what the specific interventions plausibly buy: adding basic distributed tracing and a documented, tested runbook. If the pilot's own after-state for the services that got both changes comes in at, say, 55 minutes, that pilot number, not a projected industry average, becomes the evidence used to set the expand-quarter target for the rest of the org. The program's credibility rests entirely on that number being the pilot's real, reproducible before-and-after, not an assumed percentage stated up front.

Trade-offs and pitfalls

• Setting a target before baselining, mandating "cut mean time to recover 40 percent" org-wide on day one, is the fastest way to make this read as a compliance exercise, because nobody can tell if the number is real.
• Over-indexing on leading or practice metrics, badges, checklist completion, without ever checking whether they moved the lagging outcome metrics lets a team look compliant while reliability does not improve.
• A governance body with only enforcement power and no ability to revise the standard loses legitimacy the first time a rule is wrong for a real team's context, and teams stop engaging honestly once that happens.
• Treating this as a 12-month sprint that ends on schedule regardless of outcome, instead of sustaining the surviving pieces past the program, is how these programs regress within a year of "completion".

Definition & core behavior:

• Consistent hashing: maps keys to a ring; nodes own token ranges; good for uniform distribution and low-movement rebalancing.
• Range sharding: partitions by contiguous key ranges (e.g., time, lexicographic); excellent for range scans because locality is preserved.

Trade-offs:

• Locality for range scans: range sharding wins — contiguous keys live on same shard, enabling efficient sequential reads and server-side scans. Consistent hashing destroys contiguous layout (keys are scattered), so range queries require fan-out to many nodes or scan over the ring.
• Load balance: consistent hashing excels at even distribution (especially with virtual nodes). Range sharding can create hotspots when data distribution is skewed (e.g., recent timestamps).
• Rebalancing cost: consistent hashing minimizes key movement on node add/remove (~O(k/n) keys); range sharding may require moving large contiguous ranges or splitting/merging ranges (higher data transfer).
• Operational complexity: range sharding requires dynamic split/merge logic and hotspot mitigation; consistent-hash systems need mechanisms for range scans (indices, coordinated scanning).

Hybrid approach (preserve locality, avoid hotspots):

1. Hierarchical sharding: first range-shard by coarse-grained window (e.g., day/week) to preserve locality for typical scans. Within each range shard, use consistent hashing (or virtual nodes) to distribute load across multiple physical nodes. Example: keys => day bucket => vnode ring per bucket.
2. Adaptive intra-range partitioning: monitor per-range load; if a range becomes hot, split it into sub-ranges and reassign vnodes; leverage consistent-hash placement to move only vnodes (small units).
3. Read-path optimization: maintain per-range metadata (min/max, replica list) and bloom filters to route range scans to the minimal set of nodes; allow coordinator to perform parallel ordered merge when sub-shards exist.
4. Tiered storage: keep recent hot ranges on many replicas or write-heavy nodes; colder ranges compacted to fewer nodes for scan efficiency.

Why this works:

• Coarse ranges preserve scan locality and allow server-side sequential reads.
• Using consistent hashing inside a range gives even distribution and low-cost rebalancing at vnode granularity.
• Dynamic split/replicate handles hotspots without moving entire historical datasets.

Operational notes:

• Choose coarse range size to match common query windows to limit fan-out.
• Ensure metadata service is highly available for routing.
• Monitor split thresholds (QPS, size) and automate splits/merges to avoid manual balancing.

I would frame the discussion around user impact, risk, and reversibility. A workaround is a temporary fix that reduces pain now, while the root issue is the underlying cause we still need to solve. I would ask: how many users are affected, how severe is the problem, and how risky is the workaround itself?

If the workaround is low risk and reversible, I would lean toward shipping it now and scheduling the root fix immediately after. For example, if users are blocked by a broken validation rule and we can safely relax it, I would ship the workaround, monitor errors, and commit to the deeper fix in the next cycle. If the workaround could corrupt data or create a bigger support burden, I would slow down and fix the root issue first.

I would make the decision explicit, document the trade-off, and assign an owner for the follow-up fix. That way the team is not pretending the workaround is the final answer, and users get relief as soon as it is safe to do so.

Goal: support median order value per customer approximate and exact on demand. Strategies:

1. Approximate: per-customer sketches. Maintain streaming quantile sketches (e.g., t-digest or q-digest) per customer updated as orders arrive. Store serialized sketch in a customer_agg table; querying median = merge sketches and extract 0.5 quantile — very fast and low storage.
   Schema:

• customer_metrics(customer_id PK, orders_count INT, total_amount NUMERIC, tdigest BYTEA, last_updated)
  Writes: update tdigest in place or via append-and-merge in background for concurrency.

2. Exact on-demand: maintain recent order list or sampled materialized table. Options:

• Precompute exact medians for high-activity customers by maintaining an ordered data structure (e.g., skiplist) in an external store (Redis sorted set) to compute exact median in O(log n).
• For on-demand exact compute: run a database query using percentile_cont over partitioned data or use external MapReduce job; limit cost by computing only for requested customers and caching results.
  Hybrid approach:
• Use t-digest for interactive dashboards and alerts.
• For ad-hoc exact needs, run exact computation asynchronously and cache result; for large customers keep exact structure.
  Trade-offs:
• t-digest: fast, low memory, small error (configurable), merges easily—good for production dashboards.
• Exact computation: expensive for large histories; use only for small/high-value customers or infrequent reports.
  Implementation notes:
• Provide TTL and compaction for sketches to bound storage; ensure deterministic merging and versioning of sketch library.
• Evaluate approximation error empirically and surface confidence intervals to consumers.