DoorDash Staff Data Engineer Interview Preparation Guide

Direct answer

A degenerate dimension is an identifier that lives directly on the fact table with no corresponding dimension table, because it has no descriptive attributes of its own beyond the identifier itself, for example an order number on an order-line fact table.

Structured elaboration

• Why it's "degenerate": a normal dimension has descriptive attributes (a customer has a name, an address). An order number, by itself, describes nothing beyond identifying which fact rows belong to the same order; it has no attributes worth storing in a separate table.
• Why keep it on the fact table instead of building a dimension: building a order_number_dim(order_number_key, order_number) table with no other columns adds a join for zero descriptive benefit; the identifier is just as useful, and cheaper to query, sitting directly on the fact table.
• When it stops being degenerate: if the business later wants to attach real descriptive attributes to the order itself (an order-level status, a fulfillment channel, an order-level discount code) that aren't already captured elsewhere, that's a sign the order deserves its own real dimension (or an order-level fact table), not that it should stay degenerate.

Worked example

order_line_fact(order_line_id, order_number, product_key, customer_key, date_key, quantity, unit_price) keeps order_number directly as a plain column, used to group line items belonging to the same order (SELECT order_number, SUM(quantity * unit_price) FROM order_line_fact GROUP BY order_number computes order totals) without needing to join anywhere.

Trade-offs and pitfalls

The common mistake in the other direction is treating a genuinely descriptive identifier as degenerate to avoid building a dimension, then later needing to add real attributes and discovering the fact table has no natural place for them without a schema change and a backfill. Before deciding an identifier is degenerate, check whether the business already has, or will likely soon need, descriptive attributes attached to it; if so, model it as a proper dimension from the start.

Direct answer

At 20,000 events/sec with a sub-5-second ingestion latency target, the ingestion layer needs only a small number of partitions to handle the raw throughput comfortably, and the harder engineering problem is actually the read side: getting thousands of dashboard queries per second down to sub-second latency, which a caching layer in front of the serving store solves cheaply if the read pattern is repetitive (as dashboard traffic usually is).

Structured elaboration

Sizing the ingestion layer: with an average event size and a per-partition throughput ceiling (both stated assumptions here, since real limits depend on your specific broker and hardware), compute the number of partitions from both the message-rate cap and the raw-bytes-per-second cap, and provision for the larger of the two plus headroom for skew and growth.

Processing layer: a stream processor consuming those partitions, doing whatever light transformation/aggregation the metrics require, with parallelism matched to the partition count so no single consumer instance becomes a bottleneck. At this volume (about 19 MB/s), this is a modest cluster, not a large one; the sub-5-second latency bar is comfortably achievable without exotic tuning.

Storage layer for serving: choose a real-time-oriented store (a fast OLAP engine like ClickHouse/Druid, or a low-latency key-value/columnar store) over a traditional data warehouse for the serving path, since warehouses are optimized for large scans, not thousands of small, low-latency point/aggregate queries per second.

Caching strategy for sub-second reads: dashboard read traffic is highly repetitive (many viewers polling the same small set of metric keys), which makes it an excellent caching target. Put a cache (in-memory, short TTL matched to your freshness bar) in front of the serving store, so only a small fraction of reads (cache misses) actually hit the store, and even those hit a store designed for low-latency point reads.

Worked example

Partition sizing (assumptions: 1 KB average event size; a single partition sustains 10,000 msgs/sec or 10 MB/sec, whichever binds first):

EVENTS_PER_SEC, AVG_EVENT_BYTES = 20_000, 1_000
PER_PARTITION_MSG_CAP, PER_PARTITION_MB_CAP = 10_000, 10

throughput_mb_s = EVENTS_PER_SEC * AVG_EVENT_BYTES / (1024*1024)   # 19.07 MB/s
partitions_by_msg_rate = -(-EVENTS_PER_SEC // PER_PARTITION_MSG_CAP)  # ceil -> 2
partitions_by_mb_rate = -(-throughput_mb_s // PER_PARTITION_MB_CAP)   # ceil -> 2

print(f"throughput_mb_s={throughput_mb_s:.2f}")
print(f"partitions_by_msg_rate={partitions_by_msg_rate}")
print(f"partitions_by_mb_rate={partitions_by_mb_rate}")

Executed output: raw throughput is 19.07 MB/s; both the message-rate and byte-rate caps independently require 2 partitions; provisioning 3 (50% headroom) comfortably absorbs skew and near-term growth without needing to repartition soon.

Cache sizing (assumptions: 500 distinct dashboard metric keys, 50 viewers per key, polling every 5 seconds, 98% cache hit rate, 150ms store read latency vs. 3ms cache read latency):

read_qps = 500 * 50 / 5                                    # 5,000 reads/sec, all viewers
store_qps = read_qps * (1 - 0.98)                          # 100 reads/sec actually hit the store
blended_latency_ms = 0.98*3 + 0.02*150                      # 5.94 ms

print(f"read_qps={read_qps:.0f}")
print(f"store_qps={store_qps:.0f}")
print(f"blended_latency_ms={blended_latency_ms:.2f}")

Executed output: 5,000 total read QPS collapses to only 100 QPS actually reaching the serving store once the cache absorbs 98% of reads, and the blended expected latency across all reads is 5.94 ms, comfortably under a one-second target even though a single uncached store read (150ms) is far from instant on its own.

Trade-offs and pitfalls

The mistake to avoid is over-provisioning the ingestion layer out of an instinct that "20,000 events/sec sounds like a lot," when the arithmetic shows it's a genuinely modest throughput for a modern streaming platform; the real engineering effort here is on the read side, where naively hitting the serving store for every dashboard poll (5,000 QPS with no cache) would either require a much more expensive store or blow the sub-second latency target under load. The other pitfall is setting the cache TTL longer than the stated freshness requirement to squeeze out a higher hit rate; the cache's TTL has to be bounded by the same 5-second latency promise the whole system is built to keep, not chosen purely to optimize the cache-hit number.

Direct answer. Parameter sniffing happens because a compiled, cached plan is built using the specific parameter value it first saw, and that plan's shape (which join algorithm, which access method) is only optimal for parameter values with a similar selectivity; a wildly different parameter value reused against that same cached plan can get a badly mismatched strategy instead of a plan tailored to it.

Structured elaboration. Compiling a plan is genuinely expensive, so reusing a cached plan across executions with different parameter values is usually the right default, since recompiling every single execution would cost more in aggregate than occasionally running a slightly suboptimal cached plan. The problem is specifically when the DISTRIBUTION of a filtered column is skewed: a plan compiled for a parameter value that matches very few rows (favoring an index-heavy, nested-loop-friendly strategy) can be catastrophically wrong when reused for a parameter value that matches a huge fraction of the table (which would have wanted a scan-and-hash-join strategy instead), and vice versa.

Three mitigation strategies, with their own trade-offs: (1) recompiling for specific problematic queries (an explicit "always compile fresh for this statement" hint) trades the compilation cost, paid every time, for guaranteed plan-appropriateness, which is worth it for a query executed rarely enough that compilation cost is a rounding error, and wrong for one executed thousands of times a second where that cost would dominate; (2) compiling against a deliberately "average" or "unknown" assumption about the parameter, rather than the literal first-seen value, trades away plan-appropriateness for EITHER extreme in exchange for a plan that's reasonably okay across the whole range, which suits a column whose skew isn't too extreme; (3) maintaining multiple cached plans keyed by parameter-value bucket (where the engine supports it) gets closer to "the right plan for the right value" without paying full recompilation cost every time, at the cost of more cache memory and more operational complexity to reason about.

Worked example. A stored procedure looking up orders by status, where 95% of rows are 'completed' and 1% are 'disputed': a plan first compiled and cached while looking up 'disputed' orders (an index-heavy plan, since that value is highly selective) reused later for 'completed' orders would try to walk an index for what's now 95% of the table, dramatically slower than the scan-based plan that value actually deserves.

Trade-offs and pitfalls. There's no universally correct default among the three mitigations above; the right one depends on how skewed the underlying column actually is, how often the query executes, and how much compilation overhead the workload can tolerate, which is exactly why this remains a judgment call rather than something to fix once and forget.

Direct answer

Availability is whether the service is up and responding right now, the percentage of time requests get a correct response. Reliability is whether the service does the correct thing every time over a longer horizon, even if that means taking longer or failing loudly rather than silently. A service can be highly available (always responds) while being unreliable (frequently returns wrong or incomplete results), and vice versa.

How they're measured differently

• Availability: uptime percentage, request success rate (successful responses over total requests), and latency, all measured in real time against a rolling window.
• Reliability: job or transaction success rate over time, data-loss incidents, mean time between failures, and correctness checks like reconciliation counts, none of which are visible from a single point-in-time health check.

Worked example: an HTTP API versus a background worker

An HTTP API's job is to respond fast and stay up, so availability is the priority metric. Suppose the API calls three dependencies in sequence to serve a request: an auth service at 99.95% availability, a database at 99.9%, and a cache at 99.99%. Because a single request needs all three to succeed, the composed availability is the product of the three:

That's under three nines even though every individual dependency is at or above three nines, because failures compound across a serial chain. In annual downtime terms:

compared to a single 99.9% dependency on its own:

Chaining three otherwise-strong dependencies serially costs over 300 extra minutes of downtime a year versus just one of them alone. This is why an API-focused architect pushes hard on redundancy at each hop. To see how strong that lever is even when the underlying component is weaker, consider a hypothetical, cheaper cache tier, deliberately worse than the 99.99%-rated cache used above, where each individual replica only hits 99% availability on its own: two independent, parallel replicas of that weaker cache layer already beat any single component in the chain, the strong 99.99% cache included:

A background worker processing a queue of jobs, by contrast, doesn't need to respond within milliseconds; what matters is that every job eventually completes correctly, with no silent data loss, which is a reliability property, not an availability one. If the worker is down for ten minutes and then resumes and correctly processes every job that queued up during that window, availability took a hit but reliability didn't; if the worker stays "up" the whole time but drops or duplicates 0.01% of jobs due to a bug, availability looks perfect while reliability has quietly failed.

Trade-offs & pitfalls

Optimizing for availability alone can mask reliability problems: a service that always responds quickly, even by returning stale or wrong data rather than waiting for a correct answer, looks perfect on an uptime dashboard while silently corrupting downstream state. The practical approach is deciding, per component, which property is actually load-bearing: user-facing APIs generally prioritize availability with graceful degradation for correctness-adjacent risk, while systems of record and background processing prioritize reliability, often accepting higher latency or even temporary unavailability rather than risk an incorrect or lost write.

Direct answer

Without formal authority, influence has to come entirely from credibility and voluntary buy-in instead of any ability to assign work or shape a review. That changes the approach toward explicit opt-in, keeping every session clearly worth their time, and respecting that they can walk away at any point without consequence.

What actually changes

• No mandate over cadence or topics. You can't schedule a recurring 1:1 and assume it happens, each session has to earn its place on their calendar.
• No visibility into their formal goals. You're advising without the context a manager has, so advice has to stay conditional ("here's what I'd consider, given what I know") rather than directive.
• No enforcement of follow-through. They can take or leave anything you suggest with no consequence, which is a feature, not a problem, but it means you can't measure success the way you would with a direct report.
• A boundary with their actual manager. Advice that touches their team's norms, priorities, or performance is their manager's territory. Staying in a peer-advisor lane means flagging that explicitly rather than quietly overriding it.
• No natural checkpoint. A direct-report relationship gets reviewed on a cycle; an informal one only continues as long as both sides keep choosing it, so it's worth periodically checking whether it's still useful rather than assuming it is.

Worked example

A colleague on a different team reached out about a specific hard decision they were facing. The first move was an explicit, opt-in question rather than assuming continued access: whether they wanted a recurring conversation or just help with this one thing. Advice stayed framed as "here's what I'd weigh" rather than a recommendation to just do X, and anything that touched their team's priorities or their manager's likely call was flagged as outside this lane, with a suggestion to raise it with their manager directly instead. A few sessions in, a light check-in confirmed it was still useful before continuing.

Trade-offs and pitfalls

A common mistake is treating an informal mentee like a direct report: being directive, assuming continued access, and not checking whether it's still wanted. The more durable version treats it as an ongoing, consent-based relationship, and requires being comfortable that some advice will simply be ignored with no way to enforce it, which is normal here, not a sign of failure. The other real pitfall is overstepping into another manager's territory, giving performance-adjacent feedback that should go through the person's actual chain instead.

Direct answer

An automated skew-detection-and-mitigation system needs three layers working together: a lightweight, continuous METRICS pipeline collecting per-stage task-duration and shuffle-size distributions from every job (not sampling, since skew is inherently about outliers a sample can miss); explicit, conservative THRESHOLDS for what counts as actionable skew (tuned to avoid reacting to normal variance); and a SAFE remediation layer that only applies well-understood, reversible mitigations (dynamic salting, on-the-fly repartitioning via Adaptive Query Execution) with cooldown/hysteresis logic to prevent oscillation. The hardest design constraint is not detecting skew, it is avoiding a system that reacts to noise and makes things worse.

Structured elaboration

Metrics to collect. Per-stage, per-task: task duration, shuffle-read bytes/records, shuffle-spill bytes (memory and disk), and GC time, collected via Spark's own SparkListener API (the same event stream that powers the Spark UI) rather than parsing UI HTML or logs after the fact, giving a structured, queryable feed. Derived per-stage SUMMARY metrics: the ratio of max-to-median task duration (a direct, simple skew signal: a healthy stage has this ratio close to 1, a skewed one shows it far above 1), and the coefficient of variation of shuffle-read bytes across tasks in the same stage (a statistically more robust skew signal than a single max/median ratio, since it accounts for the whole distribution's spread, not just the two most extreme points).

Thresholds for action. A single fixed threshold (say, "max/median duration ratio > 5") risks both false positives (a stage with naturally uneven work, not true data skew, like an intentionally imbalanced join by design) and false negatives (skew below the threshold that still meaningfully hurts a specific critical pipeline's SLA). A more robust design uses a threshold PLUS a minimum absolute impact filter: only flag a stage where the skew ratio exceeds the threshold AND the absolute wasted time (the difference between the actual stage duration and what an evenly-distributed stage of the same total work would have taken) exceeds a meaningful floor (say, several minutes), so the system does not fire on a stage that is technically skewed but too short in absolute terms for the skew to matter practically.

Safe automated remediations.

• Dynamic salting, applied automatically ONLY to a specific job's specific skewed join/aggregation key, requires the system to be able to safely REWRITE the query (or inject a salting transformation) without changing its semantics; this is safe to automate for a KNOWN, common pattern (a groupBy/join on a single flagged column) but risky to automate blindly for an arbitrary, complex query where an automated rewrite might not preserve correctness, so a production system should restrict automated salting to a well-tested, narrow set of recognized query shapes, not attempt to rewrite arbitrary SQL.
• On-the-fly repartitioning via Adaptive Query Execution is the SAFEST automated remediation available, since it is not a custom system at all, it is Spark's own built-in, extensively-tested runtime feature (spark.sql.adaptive.skewJoin.enabled); an automated system's most valuable "remediation" action for many detected cases may simply be confirming AQE is enabled and correctly configured for the flagged job, and alerting a human only when AQE's own skew handling did not fully resolve the detected issue (since AQE has its own thresholds and does not catch every skew shape, particularly aggregation skew outside of joins).

Avoiding oscillation and excessive cost. Any AUTOMATED remediation that changes a running job's configuration (not just a recommendation) needs explicit hysteresis: a cooldown period after applying a fix before re-evaluating whether further action is needed, and a record of what was already tried, so the system does not repeatedly reapply (or worse, alternate between) conflicting mitigations for the same recurring job run over run. A remediation's OWN cost (added shuffle from a salting rewrite, added complexity) needs to be weighed against the skew's actual cost; a system that "fixes" a mildly skewed stage by adding meaningful overhead everywhere, including on runs where the skew was mild and would have resolved acceptably via AQE alone, is a net loss even though it technically reduced skew.

Worked example

A production Spark job runs nightly; the automated system's SparkListener captures per-task metrics for every stage. On night N, stage 4 (a groupBy aggregation) shows a max/median task-duration ratio of 18x and an absolute wasted-time estimate of 22 minutes (the stage took 25 minutes; an evenly-distributed equivalent, estimated from total shuffle bytes divided by available parallelism, would have taken roughly 3 minutes), well past both the ratio threshold and the absolute-impact floor.

The system's first response is NOT to auto-salt; it first checks whether spark.sql.adaptive.skewJoin.enabled was active for this run (it was, but this is an AGGREGATION skew, which AQE's skew-join handling does not address, since that feature specifically targets JOIN skew, not groupBy skew) and logs this distinction. Since this specific query shape (a single-column groupBy aggregation) matches the system's recognized, pre-validated pattern for automated salting, it applies a salting rewrite for THIS job's THIS query going forward, tags the change with a timestamp and the triggering metrics, and sets a 14-day cooldown before re-evaluating whether the salted version still shows skew (which would indicate the underlying key distribution has shifted further and needs re-tuning, e.g., a different number of salt buckets) rather than re-triggering on every subsequent run's minor variance.

Trade-offs and pitfalls

• Common mistake: building a system that reacts to EVERY detected skew ratio above a threshold, without an absolute-impact floor; this produces noisy, low-value interventions on short stages where the skew, while statistically real, does not matter in absolute terms, eroding trust in the system's alerts and remediations.
• Common mistake: allowing automated remediation to apply to ARBITRARY query shapes rather than a narrow, well-tested set of recognized patterns; an automated rewrite that subtly changes a complex query's semantics (not just its performance) is a correctness risk far more costly than the performance problem it was meant to fix.
• Aggregation skew and join skew need different detection AND different remediation logic, as the worked example shows; a system that only checks whether AQE's join-skew handling is enabled will silently miss aggregation skew entirely, a scope gap worth being explicit about rather than assuming one feature flag covers "skew" generically.
• The cooldown/hysteresis design is what actually prevents oscillation, but it also means the system is deliberately SLOW to react to a genuinely NEW skew pattern that appears shortly after a cooldown began for a different pattern; this is a deliberate trade of responsiveness for stability, worth stating explicitly as a design choice rather than an oversight.

Direct answer: A window function keeps one row per product while still attaching the category's total, because PARTITION BY does not collapse the rowset the way GROUP BY does. Aggregate revenue to one row per product first, then compute SUM(product_revenue) OVER (PARTITION BY category_id) for the category total and divide. A plain GROUP BY can give you category totals or product totals but not both on the same row: you would need two separate aggregate queries joined back together, which is exactly what PARTITION BY does in a single pass without ever dropping the product-level grain.

Approach

Aggregate to one row per product first (so line-level sales don't get double counted), then add a window SUM partitioned by category_id for the category total, and divide.

WITH product_revenue AS (
  SELECT
    p.category_id,
    s.product_id,
    SUM(s.revenue) AS product_revenue
  FROM sales s
  JOIN products p ON p.product_id = s.product_id
  GROUP BY p.category_id, s.product_id
)
SELECT
  category_id,
  product_id,
  product_revenue,
  ROUND(
    100.0 * product_revenue / NULLIF(SUM(product_revenue) OVER (PARTITION BY category_id), 0),
    2
  ) AS pct_of_category
FROM product_revenue
ORDER BY category_id, pct_of_category DESC;

Key points

• Aggregate first, then window: windowing over raw, unaggregated sales rows would inflate the category total by however many order lines each product happened to have.
• SUM(...) OVER (PARTITION BY category_id) computes the category total on every row of that category without collapsing the product grain, which is the entire reason to reach for a window function instead of a second GROUP BY query.
• NULLIF(denominator, 0) turns a divide-by-zero into a clean NULL result instead of an error, for a category whose total happens to be zero.

What a plain GROUP BY loses: GROUP BY category_id alone drops the product_id column (or makes it arbitrary, depending on dialect, since it isn't a grouping key). GROUP BY category_id, product_id keeps per-product rows but has no way to also carry the category's total on that same row without a second aggregate query and a join back to the detail rows. That extra join is exactly the cost the window-function version avoids.

Worked example

Products: (1, category 10), (2, category 10), (3, category 20), (4, category NULL). Sales revenue by product: product 1 = 150, product 2 = 50, product 3 = 80, product 4 = 40.

Verified in DuckDB:

Category 10's two products correctly sum their percentages to 100 (75 + 25). Product 4, with a NULL category_id, still gets a result: 100%.

What changes with a NULL category_id: PARTITION BY treats NULL as a valid, groupable key: unlike a JOIN condition, where NULL = NULL is never true, two rows with a NULL category_id are still treated as the same partition for window-function purposes. So an uncategorized product does not error out or vanish; it lands in its own "NULL partition." If there is only one such product it trivially reports 100%, and if there are several, they get percentages of each other's totals under an "uncategorized" bucket, which may or may not be what the business actually wants. Flag this explicitly to a stakeholder rather than silently shipping a NULL-partition percentage as though it were a real category.

Supporting COUNT(DISTINCT x) in the same window: some dialects reject DISTINCT inside a window aggregate (COUNT(DISTINCT customer_id) OVER (...) is invalid syntax there). The standard workaround is DENSE_RANK(): rank the distinct values within the partition (gapless, so ties get the same rank and the next distinct value gets the next integer), then take MAX() of that rank over the same partition. The highest dense rank equals the count of distinct values.

WITH ranked AS (
  SELECT category_id, customer_id,
    DENSE_RANK() OVER (PARTITION BY category_id ORDER BY customer_id) AS dr
  FROM orders
)
SELECT category_id,
  MAX(dr) OVER (PARTITION BY category_id) AS distinct_customers_in_category
FROM ranked;

Verified in DuckDB against orders(category_id, customer_id) = (10,1), (10,1), (10,2), (20,3): category 10 correctly returns 2 distinct customers, category 20 returns 1.

Complexity
The product-level aggregation is O(m log m) or O(m) depending on whether the engine sorts or hashes to group (m = number of sales rows); the subsequent window pass over the resulting one-row-per-product set is O(p log p) to sort by category_id (p = number of products) plus an O(p) incremental sum. This is materially cheaper than running two separate GROUP BY queries and joining, which would require building and probing a join on top of both aggregates.

Edge cases

• A category total of exactly zero: handled by NULLIF, returns NULL rather than raising a division error.
• Products with multiple category memberships (many-to-many): this pattern assumes one category per product; a many-to-many relationship needs an allocation rule (e.g., split revenue proportionally across categories) before this query applies at all.
• Negative revenue (refunds/returns): the percentages still compute correctly as signed numbers, but a category with revenue near zero can produce a percentage that swings wildly or exceeds 100%, worth sanity-checking before it reaches a dashboard.

Trade-offs & pitfalls

The same PARTITION BY-and-divide pattern generalizes cleanly to sibling framings that a stakeholder is likely to ask for next: swap the partition key from category_id to product_id (with the row grain widened to product-month) to get percent of that same product's own total, or swap it to DATE_TRUNC('month', order_date) to get percent of that month's total instead of percent of category. All three are the identical shape, one row per detail record carrying its own value plus a group-level denominator computed by a window, just with a different partition key.

Direct answer

Build the splitter around scikit-learn's grouping and stratification machinery, generating candidate fold assignments that respect group boundaries first, then greedily adjusting group-to-fold assignment to bring each fold's label distribution as close as possible to the overall distribution, since satisfying both constraints EXACTLY is not always possible.

Structured elaboration

import numpy as np
from collections import defaultdict

class SimpleStratifiedGroupKFold:
    """API: split(X, y, groups) -> yields (train_idx, test_idx).
    Greedily assigns each GROUP (not each row) to whichever fold currently has the
    lowest positive-rate deficit, keeping label balance close across folds while
    guaranteeing no group is split across folds."""
    def __init__(self, n_splits=5, random_state=None):
        self.n_splits = n_splits
        self.random_state = random_state

    def split(self, X, y, groups):
        y = np.asarray(y)
        groups = np.asarray(groups)
        rng = np.random.default_rng(self.random_state)

        # aggregate label counts per group
        unique_groups = np.unique(groups)
        rng.shuffle(unique_groups)  # process in random order to avoid systematic bias
        total_size = len(y)
        target_share = total_size / self.n_splits
        group_pos_count = {g: int(y[groups == g].sum()) for g in unique_groups}
        group_size = {g: int((groups == g).sum()) for g in unique_groups}

        fold_pos = np.zeros(self.n_splits)
        fold_size = np.zeros(self.n_splits)
        group_to_fold = {}

        for g in unique_groups:  # greedy: assign each group to the fold it helps balance most
            candidate_scores = []
            for f in range(self.n_splits):
                new_pos = fold_pos[f] + group_pos_count[g]
                new_size = fold_size[f] + group_size[g]
                label_penalty = abs(new_pos / max(new_size, 1) - y.mean())
                # without a size term, the greedy rule can chase label balance right into an
                # empty or tiny fold; penalizing distance from the target fold SIZE too keeps
                # fold sizes from collapsing while still preferring the label-closest fold
                size_penalty = abs(new_size - target_share) / total_size
                candidate_scores.append(label_penalty + size_penalty)
            best_fold = int(np.argmin(candidate_scores))
            group_to_fold[g] = best_fold
            fold_pos[best_fold] += group_pos_count[g]
            fold_size[best_fold] += group_size[g]

        row_fold = np.array([group_to_fold[g] for g in groups])
        for f in range(self.n_splits):
            test_idx = np.where(row_fold == f)[0]
            train_idx = np.where(row_fold != f)[0]
            yield train_idx, test_idx

groups = np.array([1,1,1,2,2,3,3,3,3,4,4,5,5,5])
y = np.array([0,0,1,0,1,0,0,1,0,1,0,0,1,0])
skf = SimpleStratifiedGroupKFold(n_splits=3, random_state=0)
for i, (train_idx, test_idx) in enumerate(skf.split(None, y, groups)):
    print(f"fold {i}: test groups={sorted(set(groups[test_idx]))}, test pos rate={y[test_idx].mean():.2f}")

The greedy algorithm: process groups in a randomized order, and for each group, assign it to whichever fold minimizes a COMBINATION of two penalties, how far that fold's positive rate would land from the overall dataset's positive rate, and how far that fold's resulting size would land from its equal target share of the data. The size term is not optional: a purely label-rate-driven greedy rule can walk straight into a degenerate solution (an empty or near-empty fold) whenever that happens to minimize label-rate deviation locally, which is exactly the kind of quietly-wrong behavior that only shows up by actually running the code, not by reading the label-balancing logic in isolation. This is a greedy, not globally optimal, heuristic, since finding the truly optimal group-to-fold assignment is a combinatorial problem, but it works well in practice and runs in roughly linear time in the number of groups.

Limitations of plain GroupKFold: it guarantees no group is split across folds but makes NO effort to balance label proportions across folds at all, so with a skewed label distribution and unevenly-sized groups, some folds can end up with meaningfully different positive rates than others purely by which groups happened to land where, exactly the gap this custom implementation exists to close.

Worked example

Running the code above on 5 groups (sizes 3, 2, 4, 2, 3) with a mix of label rates over an overall positive rate of $5/14\approx 0.357$, the 3-fold split produces test sizes of 4, 5, and 5 rows (no empty or near-empty fold) with per-fold positive rates of 0.25, 0.40, and 0.40, all within a reasonable band around the true 0.357 rate rather than either collapsing to a degenerate empty fold or drifting to an extreme rate the way an unweighted, size-blind greedy rule did before the size penalty was added.

Trade-offs and pitfalls

When groups vary wildly in size (one enormous group, several tiny ones), even this improved greedy heuristic can struggle to achieve good stratification, since assigning the one enormous group anywhere dominates both the size and label balance of whichever fold it lands in, regardless of how the smaller groups are subsequently balanced around it; in that specific situation, it's worth checking the group-size distribution before trusting the stratification quality, rather than assuming the greedy algorithm always closes the gap.

Direct answer

Multi-region observability under a data-sovereignty constraint (raw telemetry must stay within its originating region) means storing each region's raw telemetry locally, only ever exporting AGGREGATED or ANONYMIZED summaries across region boundaries, and giving engineers a federated query interface that can dispatch a query to each region's local store and merge the results, rather than centralizing raw data anywhere.

Structured elaboration

• Regional telemetry storage: each region runs its own independent, full-fidelity telemetry store (metrics, logs, traces), with no raw data replicated to a central location, satisfying the sovereignty constraint by construction, the raw data simply never crosses the border.
• Federated querying: a query layer that accepts a single query from an engineer and internally dispatches it to each relevant region's local store in parallel, merging the results before returning them, so from the engineer's perspective, querying "error rate across all regions for the last hour" feels like querying one unified system, even though the underlying execution is distributed and each region's raw data stays put.
• What CAN cross region boundaries: pre-aggregated, non-identifying summary metrics (a regional error-rate percentage, not raw log lines containing user data) can be safely replicated centrally for a global health overview dashboard, since an aggregate that's been reduced to a summary statistic no longer carries the sensitive raw content the sovereignty rule is protecting.
• Cross-region debugging without moving raw telemetry: for an engineer who needs to actually LOOK at raw telemetry from a specific region (not just an aggregate), the query executes IN that region (a query is sent to the region, executed there, and only the specific, relevant RESULT, not the region's whole raw dataset, is returned to the engineer), so the engineer accesses the data without the underlying raw telemetry ever being copied out of the region wholesale.

Worked example

Concretely: an engineer investigating a cross-region latency issue queries "p99 latency by region for the last 2 hours." The federated query layer sends this query to each region's local store, each region computes its own p99 locally (no raw trace data leaves that region), and only the small, aggregated p99 NUMBER per region is returned and merged into a single view for the engineer. If the EU region's number looks anomalous and the engineer needs to look at specific traces to understand why, a follow-up query for "show me the 20 slowest traces in the EU region for this window" is dispatched TO the EU region's store, executed there, and only those 20 trace RECORDS (not the region's full trace volume) are returned, still a targeted, minimal cross-border data movement, and one that would itself need to be evaluated against the sovereignty policy (whether even returning specific trace content to an engineer outside that region is permitted, which is a legal/policy question, not just an architecture one).

Trade-offs and pitfalls

The federated query approach genuinely adds latency and complexity versus querying a single centralized store, cross-region query fan-out and merge takes longer than a single local query, and the query layer itself becomes a piece of critical infrastructure that needs its own reliability engineering. The pitfall to flag explicitly is that "cross-region debugging" and "data sovereignty" can be in real tension depending on the exact legal requirement, whether even RETURNING specific record-level content (not just aggregates) to an engineer physically located outside the region is compliant is a genuine legal/policy question that varies by jurisdiction and data type, this is architecture that ENABLES compliant access, but the specific access pattern (aggregate-only versus record-level) still needs sign-off from whoever owns the actual compliance requirement, not just an engineering judgment call.

Direct answer

For a workflow spanning around 30 services, the real choice is not orchestration versus choreography as a single binary decision for the whole workflow; it is which steps need a component that can prove ordering and drive compensations (orchestration), and which steps can react to events with no central authority at all (choreography). Orchestration puts one coordinator in charge of calling each step and firing compensations in a known sequence; choreography has each participant publish an event when its own step completes and react to others' events, with no single place holding the overall plan.

Orchestration

A coordinator persists the saga's state as an explicit record (an event-sourced log or a saga_state table with a status per step), calls each participant directly, and on a failure at step k issues compensating calls for steps 1..k-1 in reverse order. Because the plan lives in one place, ordering and auditability are straightforward to reason about; the coordinator itself must be made durable and, typically, run as a small number of replicas, since it is now a component the whole workflow depends on.

Choreography

No coordinator exists. Participant N completes its local step and emits a domain event; participant N+1 subscribes to that event and reacts; a failure is just another event (e.g. ShippingFailed) that any interested participant can subscribe to and use as its own trigger to compensate. This removes the central dependency but means "what state is this workflow in" is a property of the whole event graph rather than one component's state, which is harder to reconstruct when debugging.

Compensating actions

A compensating action is the business-meaning inverse of a step, not a literal undo: refunding a settled charge is not "un-charging" it, and cancelling a shipped order needs a return flow, not a rollback. Compensations must be idempotent (safe to invoke more than once with the same effect), because a coordinator restart or a redelivered event can cause the same compensation to be issued twice.

Idempotency of each step

Every forward and compensating action is invoked with a natural key, typically (saga_id, step), that the receiving service stores alongside the resulting effect. If the same key arrives again, the service returns the already-recorded result instead of re-applying the effect (charging twice, releasing stock twice). This is what makes it safe for either a restarted orchestrator or a redelivered choreography event to retry a step it cannot be sure completed.

Detecting and recovering a mid-protocol crash

Orchestration: the coordinator's saga state is durable, so on restart it scans for sagas stuck in an in-flight status past an expected time bound, reads the last completed step from that record, and resumes forward execution or begins compensation from there. Because every action is idempotent, resuming is safe even in the worst case (crash after a participant executed but before the coordinator recorded it): the only possible cost is one duplicate no-op call.

Choreography: there is no single resume point. Each participant instead needs its own local timeout: for example, the inventory service reserves stock with an expiry, and if it never receives a downstream "payment confirmed" event within that window, it independently emits its own "reservation expired" event to trigger compensation across whatever already acted. Detecting "stuck" is decentralized and has to be designed per-participant rather than once, centrally.

Worked example: order O-500 across Payment, Inventory, Shipping

Orchestration trace:

sequenceDiagram
    participant C as Coordinator
    participant P as Payment
    participant I as Inventory
    participant S as Shipping
    C->>P: charge(step=1)
    P-->>C: success
    C->>I: reserve(step=2)
    I-->>C: success
    Note over C: crash before calling Shipping
    Note over C: restart, reads saga_state
    C->>S: schedule(step=3)
    S-->>C: fail
    C->>I: release(step=2)
    C->>P: refund(step=1)

1. saga_state(saga_id=S-500, step=1, status=STARTED).
2. Coordinator calls Payment.charge(saga_id=S-500, step=1, key=S-500:1); succeeds; saga_state updated to step=1, status=DONE.
3. Coordinator calls Inventory.reserve(saga_id=S-500, step=2, key=S-500:2); succeeds; saga_state updated to step=2, status=DONE.
4. Coordinator crashes before calling Shipping (step 3).
5. Coordinator restarts, reads saga_state for S-500: last DONE step is 2, step 3 was never started, so it resumes at step 3 and calls Shipping.schedule(saga_id=S-500, step=3, key=S-500:3).
6. Shipping fails permanently (undeliverable address).
7. Coordinator runs compensations in reverse for the completed steps: Inventory.release(saga_id=S-500, step=2), then Payment.refund(saga_id=S-500, step=1).
8. If the coordinator crashes again mid-compensation and retries Inventory.release(step=2) a second time, Inventory recognizes the key S-500:2 was already applied and returns the recorded result instead of releasing stock twice.

Choreography, same scenario: Payment emits PaymentCharged(S-500); Inventory, subscribed to it, reserves stock and emits InventoryReserved(S-500); Shipping, subscribed to that, tries to schedule and fails, emitting ShippingFailed(S-500); Inventory and Payment, both subscribed to ShippingFailed, independently run their own compensations on receiving it. If Shipping crashes before ever publishing ShippingFailed, no coordinator exists to notice the gap; Inventory only recovers because its own reservation carries a TTL (time-to-live, an expiry after which it self-cancels; say 15 minutes), and on expiry with no follow-up event it self-triggers its own compensation.

Trade-offs & pitfalls

A common pitfall is writing a compensation that isn't actually the semantic inverse of the forward action, which produces a technically-completed rollback that is still wrong for the business. In practice, a workflow like this is often a hybrid: strict, auditable steps (payment, billing) run under orchestration because ordering matters and correctness is expensive to get wrong, while more tolerant downstream steps (inventory, shipping) are choreographed since they are naturally eventual and cheaper to compensate if something goes wrong.