DoorDash Software Engineer (Mid-Level) Interview Preparation Guide

Direct answer
safeAdd has to detect overflow before it returns, not after: compute the sum in a type wide enough to hold the true result, then check that value against the 32-bit signed range before narrowing back. The three named cases each probe one boundary: no overflow at the positive ceiling, overflow just past the positive ceiling, and overflow (underflow) just past the negative floor.

Structured elaboration
Two implementation strategies:

1. Widen and compare. Promote both operands to long (64-bit), add, and check the 64-bit result against Integer.MIN_VALUE/Integer.MAX_VALUE before narrowing back to int. Simple and correct whenever a wider primitive is available.
2. Sign-check without widening. When there is no wider primitive to promote into (for example, adding two longs, since Java has no 128-bit integer type), check the operand signs directly: if b > 0 and a > Integer.MAX_VALUE - b, the addition would overflow; if b < 0 and a < Integer.MIN_VALUE - b, it would underflow. This is the same logic as approach 1, just applied one level down without a wider type to lean on.

Either way, the design decision that matters as much as the arithmetic is what the function does on overflow: a custom unchecked OverflowException (shown below) documents the contract explicitly without forcing every caller to add a try/catch; a checked exception would force explicit handling at every call site, which is heavier but harder to silently ignore; returning a sentinel (like Optional.empty()) avoids exceptions entirely but pushes the "did this succeed" check onto every caller and is easy to forget.

Worked example (executed)

public class OverflowException extends RuntimeException {
 public OverflowException(String msg) { super(msg); }
}

public static int safeAdd(int a, int b) {
 long sum = (long) a + (long) b;
 if (sum > Integer.MAX_VALUE || sum < Integer.MIN_VALUE) {
 throw new OverflowException("Integer overflow for " + a + " + " + b);
 }
 return (int) sum;
}

import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.*;

class SafeAddTest {
 @Test
 void maxPlusZero() {
 assertEquals(Integer.MAX_VALUE, safeAdd(Integer.MAX_VALUE, 0));
 }

 @Test
 void maxPlusOneThrows() {
 assertThrows(OverflowException.class, () -> safeAdd(Integer.MAX_VALUE, 1));
 }

 @Test
 void minPlusNegativeOneThrows() {
 assertThrows(OverflowException.class, () -> safeAdd(Integer.MIN_VALUE, -1));
 }
}

The plain driver below invokes the exact production method above (a JUnit runner was not available in this sandbox, so this is the reproducible equivalent):

public class Driver {
 public static void main(String[] args) {
 System.out.println("safeAdd(MAX_VALUE, 0) = " + safeAdd(Integer.MAX_VALUE, 0));
 try {
 safeAdd(Integer.MAX_VALUE, 1);
 } catch (OverflowException e) {
 System.out.println("safeAdd(MAX_VALUE, 1) threw OverflowException: " + e.getMessage());
 }
 try {
 safeAdd(Integer.MIN_VALUE, -1);
 } catch (OverflowException e) {
 System.out.println("safeAdd(MIN_VALUE, -1) threw OverflowException: " + e.getMessage());
 }
 }
}

Running this driver against the exact safeAdd implementation above produces:

safeAdd(MAX_VALUE, 0) = 2147483647
safeAdd(MAX_VALUE, 1) threw OverflowException: Integer overflow for 2147483647 + 1
safeAdd(MIN_VALUE, -1) threw OverflowException: Integer overflow for -2147483648 + -1

As a contrast, Integer.MAX_VALUE + 1 computed with plain int arithmetic (no check at all) silently wraps to -2147483648, which is exactly the class of bug this method exists to prevent: a positive-looking calculation flipping to a large negative number with no exception and no log line.

Complexity and edge cases
safeAdd is O(1) time and O(1) space regardless of implementation strategy, it does a fixed, constant amount of work per call. Edge cases: the no-overflow boundary (MAX_VALUE + 0), the positive-overflow boundary (MAX_VALUE + 1), the negative-overflow (underflow) boundary (MIN_VALUE + (-1)), and by extension any pair whose true sum falls outside [Integer.MIN_VALUE, Integer.MAX_VALUE].

Trade-offs & pitfalls
The widen-and-compare approach is the one to reach for by default, it reads clearly and is easy to review. The sign-check alternative matters once you're adding two values that are already at the widest available primitive (long + long), where there is nothing left to widen into; the same sign-check pattern applies there with Long.MAX_VALUE/Long.MIN_VALUE. A common mistake is checking only the positive overflow case and forgetting Integer.MIN_VALUE + (-1), since the negative floor is easy to forget when the mental model is "sums get too big," not "sums get too negative." Another pitfall: Java's built-in Math.addExact(int, int) already does exactly this check and throws ArithmeticException, so a from-scratch implementation is usually only justified when the API needs a custom exception type or message.

Direct answer

When the timeline must shrink dramatically, the job is not to work faster on everything, it is to make an explicit, defensible call about what stops being in scope, using consistent criteria (business value, risk, dependencies, customer impact) rather than gut feel or cutting whatever is easiest to remove.

Structured elaboration

1. Score the backlog against criteria: business value (revenue or retention impact), risk (what breaks or what exposure appears if this is cut), dependencies (does other committed work depend on it), and customer impact (who notices, how loudly).
2. Use the speed, quality, cost lens explicitly when that is how the ask is framed: lay out what could be traded in quality (test coverage, polish, edge-case handling), in cost (temporary contractors, overtime), or in scope, as separate levers rather than assuming scope is the only one available.
3. Negotiate with two or three concrete packages, each naming what it buys and what it costs, rather than deciding alone and revealing the cut plan only at the deadline.
4. Revise and communicate: update the plan with what is cut or deferred and why, get written sign-off from whoever asked for the timeline change, and tell the wider team the same story so nobody discovers the cut informally later.

Worked example

A 12-week internal analytics dashboard replatform has three planned pieces: A, the core feature (must-have, 5 weeks), B, a secondary feature (4 weeks), and C, a nice-to-have polish pass (3 weeks). Leadership asks for either a 30% scope cut or half the time (6 weeks), and the two asks require different depth of cutting.

If the ask is a 30% cut of the 12-week plan, the target is roughly 8.4 weeks. Deferring C alone (3 weeks) lands at 9 weeks, still short of target; trimming a week of B's edge cases brings it to 8 weeks, meeting the target with a small margin. That is the 30%-cut path: defer C, thin B.

If instead the ask is literally half the time, 6 weeks, deferring C only reaches 9 weeks, three weeks over budget. Reaching 6 weeks also requires deferring B entirely: 12 minus 3 (C) minus 4 (B) is 5 weeks, a week under budget, so a thin slice of B's most critical piece (about 1 week) is added back, landing exactly at 6 weeks. That is the half-time path: ship only A plus a thin slice of B, defer C and the rest of B.

A survives both cuts because it scores highest on business value and dependencies; the arithmetic shows the 30%-cut plan and the half-time plan are genuinely different depths of cutting, not the same plan reused.

Trade-offs and pitfalls

The clearest pitfall is treating "cut 30% of scope" and "cut to half the time" as interchangeable; they demand different depth of cuts, and conflating them produces a plan that misses whichever target was not actually checked against the arithmetic. Another pitfall is negotiating only on scope when a quality or cost lever might serve the business better, for example, temporary contractor cost might be cheaper than a customer-facing cut. A real trade-off: cutting the same "nice to have" item first every time preserves trust, but if a deferred item keeps sliding indefinitely, it needs a genuine re-review date, not a silent, permanent drop.

Direct answer. Look past 'does it pass tests' to readability, failure handling, API/contract impact, and test coverage of the CHANGE itself -- and phrase feedback as a specific, actionable suggestion rather than a vague complaint.

The core checks, with an example each

1. Readability: would a teammate unfamiliar with this PR understand the intent from the code and names alone? Example feedback: 'This function does three unrelated things (parses, validates, persists) -- can we split it so each has one job? Happy to pair on this if useful.'
2. Test coverage of the CHANGE, not just the file: does the diff itself have tests exercising the new behavior, especially edge cases the author might not have thought of? Example: 'This handles the happy path well -- what happens if items is empty? Could we add a test for that?'
3. API/contract impact: does this change a public function's signature, a response shape, or a database schema in a way that breaks existing callers? Example: 'This response now omits the status field that the mobile client depends on -- can we confirm nothing downstream reads it, or keep it for backward compatibility?'
4. Failure handling: what happens when a dependency is slow, unavailable, or returns something unexpected? Example: 'If this external call times out, does the caller get a clear error, or does this hang indefinitely?'
5. Scope: is this PR doing ONE reviewable thing, or is a refactor bundled with a behavior change, making it hard to tell which lines matter for the actual bug fix? Example: 'Could we split the formatting-only changes into a separate PR so the logic change is easier to review on its own?'

Why 'beyond correctness' matters

A PR that passes its own tests can still be a liability: a correct-today function that's unreadable becomes the next person's bug because they misunderstand it while extending it; a correct-today API change that silently breaks a downstream caller is correct in isolation and wrong in context. Reviewing for maintainability is reviewing for the NEXT change, not just this one.

Trade-offs and pitfalls

• Reviewing everything at maximum scrutiny on every PR doesn't scale -- calibrate depth to risk (a payment-path change deserves more scrutiny than a copy tweak) rather than applying a uniform checklist regardless of stakes.
• Vague feedback ('this could be cleaner') is nearly useless; specific, actionable feedback with a suggested alternative respects the author's time and moves the review forward instead of triggering a guessing game.
• Nitpicking style that a formatter/linter should catch automatically wastes review bandwidth on human judgment that should be spent on the checks above; automate what can be automated so reviewers focus where judgment actually adds value.

Direct answer
Speaking up when you notice leadership or business behavior running against a stated principle, or discovering a values-violating practice yourself, is a career-risk-aware judgment call. The strongest answers show that you assessed the risk of speaking up honestly, chose a channel and framing proportionate to the issue, and can describe a concrete outcome, even a partial or mixed one.

Structured elaboration

• Assessing: what made you decide this was worth raising rather than letting go, whether it was a one-off or a pattern, and how material the impact was.
• Channel: who you raised it with first, and why (a direct manager rather than jumping straight to a skip-level or a formal channel, unless the severity warranted it).
• Framing: leading with concrete impact or evidence rather than an accusation, which is what makes an objection hearable rather than confrontational.
• Outcome: what actually changed, or didn't. An honest "it partially worked" or "nothing changed and here is what I did next" is a legitimate and often more credible answer than a perfectly clean resolution.
• The self-discovered variant: if you found the issue yourself, in your own work rather than someone else's, the same shape applies, but the story should show you didn't just quietly fix it and move on. Escalating a self-discovered gap through the proper channel, rather than silently patching it, is the part that demonstrates the competency.

Worked example
While reviewing a data-handling process they had built, a candidate noticed it retained a category of information longer than the stated retention policy required. Rather than quietly deleting the excess and saying nothing, they flagged the specific gap to their manager and the relevant policy owner along with a proposed fix, since a silent fix would have hidden that the gap had existed and might recur elsewhere. The fix was implemented, and the review also surfaced one other process with the same gap that would not have been found otherwise.

Trade-offs and pitfalls
Escalating everything regardless of materiality can read as poor judgment rather than integrity; the strongest answers show calibration about what is worth raising. An outcome of "nothing changed" is realistic and acceptable, but the answer should still show a proportionate attempt, not that you gave up after one try or escalated aggressively without cause. Framing a self-discovered gap as "I caught someone doing something wrong" when the honest version is closer to "I found a gap in a process I owned" overstates the story; the self-discovered version is common and doesn't need to be dressed up as catching someone else.

Direct answer

A stateful operator maintaining per-key running sums needs three things to checkpoint safely: state and the corresponding input offset checkpointed TOGETHER as one atomic unit (never one without the other), the checkpoint write itself made atomic via a stage-then-publish pattern so a reader (or a restart) never observes a partially-written checkpoint, and a restore step that reads the latest VALID checkpoint and resumes from its recorded offset. Below is a full implementation, executed, proving both correctness (restored state matches ground truth) and atomicity (a checkpoint interrupted mid-write is correctly ignored on restore, never mistaken for valid).

Structured elaboration

Why offset and state are checkpointed as one unit. If state were checkpointed without also recording exactly which input offset it reflects, a restart would not know where to resume reading, risking either reprocessing already-reflected events (double-counting sums) or skipping events the checkpoint does not yet reflect (undercounting), the same general state-and-source-position coupling any checkpoint design needs, applied here concretely to a per-key-sum operator.

Atomic snapshotting. Serialize the full current state to a TEMPORARY file, fsync it, then atomically publish it (here, os.replace, which is atomic on POSIX filesystems: a reader sees either the complete old file or the complete new file, never a partial write mid-replace). This mirrors the guarantee S3 itself gives for a single object PUT: the object is either fully the old version or fully the new version, never observable half-written. No live AWS S3 is used in this sandbox; the local-filesystem stand-in exercises the identical stage-then-atomically-publish logic a real S3 client performs, and this is stated explicitly rather than presented as a live cloud call.

Restoring state on restart. On construction, find the LATEST checkpoint file present (by filename, which encodes the offset it covers) and load it. Because only successfully, atomically published checkpoints ever appear as a checkpoint-NNNN.json file (an interrupted write leaves only a .tmp file, which the restore glob deliberately does not match), restore never accidentally picks up a half-written checkpoint.

Consistency guarantees. The restored state is exactly the state as of the last successfully completed checkpoint, no more, no less: any events processed between that checkpoint and a subsequent crash are lost from the operator's own state (this is why, in a full production pipeline, a WAL sits alongside checkpointing to make that gap recoverable via replay; this exercise focuses on the checkpoint/restore mechanism itself).

Checkpoint frequency trade-offs. More frequent checkpoints bound the gap above tighter (less state lost/needing WAL replay on crash) at the cost of more frequent I/O and, for a full-state snapshot approach like this one (as opposed to an incremental approach that persists only what changed), a write cost that scales with total state size on EVERY checkpoint, which becomes the dominant cost once per-key state grows large, exactly why production systems move to incremental checkpointing once state size crosses that threshold.

Worked example

"""
A stateful operator maintaining per-key running sums, with atomic
checkpointing to durable storage and restore-on-restart. "Atomic snapshot to
S3" is modeled with a local directory standing in for a bucket: write to a
temp key, then atomically publish via os.replace, mirroring S3's own atomic
single-object PUT visibility. No live S3 is available in this sandbox.
"""
import json, os, shutil, glob


class StatefulSumOperator:
    def __init__(self, checkpoint_dir):
        self.checkpoint_dir = checkpoint_dir
        os.makedirs(checkpoint_dir, exist_ok=True)
        self.state = {}
        self.last_offset = -1
        self._restore_latest()

    def process_event(self, offset, key, value):
        if offset <= self.last_offset:
            return
        self.state[key] = self.state.get(key, 0) + value
        self.last_offset = offset

    def checkpoint(self):
        checkpoint_id = self.last_offset
        final_path = os.path.join(self.checkpoint_dir, f"checkpoint-{checkpoint_id:010d}.json")
        temp_path = final_path + ".tmp"
        payload = {"last_offset": self.last_offset, "state": self.state}
        with open(temp_path, "w") as f:
            json.dump(payload, f)
            f.flush()
            os.fsync(f.fileno())
        os.replace(temp_path, final_path)  # atomic on POSIX
        return checkpoint_id

    def _latest_checkpoint_path(self):
        paths = sorted(glob.glob(os.path.join(self.checkpoint_dir, "checkpoint-*.json")))
        return paths[-1] if paths else None

    def _restore_latest(self):
        path = self._latest_checkpoint_path()
        if path is None:
            return
        with open(path) as f:
            payload = json.load(f)
        self.state = payload["state"]
        self.last_offset = payload["last_offset"]


def main():
    ckpt_dir = os.path.join(os.path.dirname(__file__), "s29_checkpoints")
    if os.path.exists(ckpt_dir):
        shutil.rmtree(ckpt_dir)

    op = StatefulSumOperator(ckpt_dir)
    events = [(0, "a", 5), (1, "b", 3), (2, "a", 2), (3, "c", 10), (4, "a", -1),
              (5, "b", 7), (6, "a", 1), (7, "c", 4), (8, "b", -2), (9, "a", 6)]
    for offset, key, value in events:
        op.process_event(offset, key, value)
        if offset in (4, 9):
            cid = op.checkpoint()
            print(f"Checkpointed at offset {offset} (checkpoint id {cid}); state so far: {op.state}")

    state_before_crash = dict(op.state)
    print(f"\nState before simulated crash: {state_before_crash}")

    # Simulated crash: FRESH instance on the SAME checkpoint dir, no in-memory carryover.
    op2 = StatefulSumOperator(ckpt_dir)
    print(f"Restored state after simulated restart: {op2.state}, last_offset={op2.last_offset}")
    assert op2.state == state_before_crash
    assert op2.last_offset == 9

    ground_truth = {}
    for _, key, value in events:
        ground_truth[key] = ground_truth.get(key, 0) + value
    print(f"Ground truth (summed directly from all events): {ground_truth}")
    assert op2.state == ground_truth
    print("Assertion passed: state restored purely from disk exactly matches ground truth.")

    # Atomicity proof: simulate a crash MID-checkpoint (an incomplete .tmp file exists,
    # os.replace never ran), confirm restore falls back to the last GOOD checkpoint.
    op3 = StatefulSumOperator(ckpt_dir)
    op3.process_event(10, "a", 100)
    incomplete_path = os.path.join(ckpt_dir, "checkpoint-0000000010.json.tmp")
    with open(incomplete_path, "w") as f:
        f.write('{"last_offset": 10, "state": {"a": ')  # deliberately truncated

    op4 = StatefulSumOperator(ckpt_dir)
    print(f"\nAfter a simulated crash mid-checkpoint (incomplete .tmp present, never "
          f"published): restored state = {op4.state}, last_offset={op4.last_offset}")
    assert op4.state == ground_truth
    assert op4.last_offset == 9
    print("Assertion passed: the incomplete .tmp file is correctly ignored; restore")
    print("falls back to the last GOOD checkpoint.")

    shutil.rmtree(ckpt_dir)


if __name__ == "__main__":
    main()

Output (actually executed with python3):

Checkpointed at offset 4 (checkpoint id 4); state so far: {'a': 6, 'b': 3, 'c': 10}
Checkpointed at offset 9 (checkpoint id 9); state so far: {'a': 13, 'b': 8, 'c': 14}

State before simulated crash: {'a': 13, 'b': 8, 'c': 14}
Restored state after simulated restart: {'a': 13, 'b': 8, 'c': 14}, last_offset=9
Ground truth (summed directly from all events): {'a': 13, 'b': 8, 'c': 14}
Assertion passed: state restored purely from disk exactly matches ground truth.

After a simulated crash mid-checkpoint (incomplete .tmp present, never published): restored state = {'a': 13, 'b': 8, 'c': 14}, last_offset=9
Assertion passed: the incomplete .tmp file is correctly ignored; restore
falls back to the last GOOD checkpoint.

The restored state after a simulated crash matches both the last checkpoint's recorded state AND the ground truth computed independently by summing every event directly, confirming the checkpoint mechanism captured state correctly. The atomicity test is the deliberately non-vacuous proof: a genuinely incomplete, truncated .tmp file is left on disk (simulating a crash mid-write, before os.replace ran), and restore correctly ignores it entirely, falling back to offset 9's last GOOD checkpoint rather than either crashing on the malformed JSON or silently adopting a partial state.

Trade-offs and pitfalls

• Common mistake: writing directly to the final checkpoint filename instead of a temp-file-then-rename. A crash mid-write to the final filename directly leaves a corrupted or truncated file exactly where restore expects a valid one, forcing restore to either crash or (worse) silently load malformed state; the temp-then-atomic-rename pattern above avoids this by construction, since the final filename never exists in a partial state.
• Common mistake: checkpointing state without the corresponding offset, or vice versa. Either one alone is insufficient information for a correct restore, exactly the coupling this implementation deliberately keeps as one JSON payload written in one atomic operation.
• Full-state snapshotting (this exercise's approach) does not scale indefinitely. Every checkpoint here re-writes the ENTIRE state, cheap for a handful of keys, but for state with millions of keys this becomes the dominant cost; production systems (Flink's incremental RocksDB checkpoints) checkpoint only the DELTA since the last checkpoint, once state size crosses that threshold.
• This implementation checkpoints state alone; it does not by itself give exactly-once output to a downstream sink. If this operator's sums are themselves written onward to an external system, that write needs the same idempotent, keyed-write discipline any downstream sink write does, since checkpoint/restore correctness for the OPERATOR's own state is a necessary but not sufficient condition for end-to-end correctness of everything downstream of it.

Goal: Reduce cloud spend by 30% within 3 months while maintaining or improving p95 latency and availability.

Assumptions (base for estimates)

• Current monthly cloud spend: $100k (adjust proportionally)
• 60% of compute is steady-state web/API, 40% spikey batch/ingress
• 70% of storage is infrequently accessed objects; DB is a managed RDS-like service with 3 read replicas
• SLA: p95 latency <= current baseline; availability >= current baseline

Prioritized action plan (measurable, 3-month roadmap)

1. Quick wins (0–2 weeks) — target 8–12% savings

• Rightsize VMs/instances with automated recommendations (Cloud Advisor): reduce oversized by 20% → est. 4% savings.
• Turn off non-prod environments nights/weekends via scheduler → est. 2–4% savings.
  KPIs: % of instances resized, hours non-prod off, cost delta.

2. Compute optimization (2–6 weeks) — target 10–12% savings

• Mixed reserved + spot strategy: purchase 1–3yr reserved for 40% steady-state (save ~35% vs on-demand) and move up to 30% stateless tiers to spot with fallback (save ~50% of that portion). Combined compute saving ~10–12%.
• Autoscaling tuning: change scale-in/scale-out cooldowns, predictive scaling for traffic patterns to reduce overprovisioning ~3% more.
  KPIs: on-demand vs reserved vs spot %, average CPU headroom, scale events per day, p95 latency.

3. Caching & CDN (2–8 weeks) — target 6–10% savings + performance gain

• Add CDN for static assets and cacheable API responses (cache TTLs, cache-control). Expect bandwidth & origin compute reduction ~4–6%; p95 latency improves significantly.
• In-memory caches (Redis/Memcached) for hot DB queries, with cache hit target >80% → reduces DB load and DB cost ~2–4%.
  KPIs: CDN hit ratio, cache hit ratio, origin requests, p95 latency.

4. Storage tiering & lifecycle (1–6 weeks) — target 2–4% savings

• Move cold objects to infrequent/archival tiers via lifecycle policies; compress snapshots, enable object versioning cleanup. Estimate 2–4% overall.
  KPIs: GB in hot vs cool vs archive, monthly storage cost.

5. Database optimization (3–10 weeks) — target 4–8% savings

• Analyze slow queries, add indexes, use read replicas appropriately, move analytics to separate cluster or data warehouse, rightsize instance class. Expect 3–6% saving plus performance stability.
• Consider reserved instances for DB baseline usage (save ~30% of DB compute portion).
  KPIs: DB CPU, query p95, number of slow queries, replica lag.

6. Longer-term & guardrails (ongoing) — target 2–4% cumulative

• Implement cost-aware CI/CD (artifact retention), tagging + chargeback, anomaly alerts for spend spikes.
• Regular runbooks for spot interruptions, chaos testing for resilience.

Estimated cumulative savings (conservative):

• Quick wins: 8–12%
• Compute + autoscaling: 10–12%
• Caching/CDN: 6–10%
• Storage: 2–4%
• DB: 4–8%
• Guardrails/other: 2–4%
  Total (overlap-aware conservative): 30–35% achievable.

Key risk mitigations

• Preserve performance by A/B testing cache TTLs, phased spot rollout with warm pools, rollback playbooks.
• Track KPIs daily; stop any change that increases p95 beyond 5% of baseline.

Execution: Sprints with measurable milestones, weekly cost dashboard, and an executive checkpoint at 6 and 12 weeks.

Direct answer. The causal chain here is: service-a is calling service-b, which is calling redis, and redis connections are timing out, which stalls service-b, which stalls service-a; the error message ('connect timeout to redis') is the concrete evidence that redis connectivity, not application logic, is the proximate cause.

Structured elaboration.

1. Build the chain from the evidence given, not assumption. The trace shows gateway calling service-a (600ms), which calls service-b (580ms), which calls redis (560ms). Almost all of service-a's 600ms is inside its call to service-b, and almost all of service-b's 580ms is inside its call to redis. Combined with the log line 'ERR connect timeout to redis host:6379' repeated in service-b, the chain reads: redis is not responding to new connections in time, so every request through service-b (and therefore through service-a) pays that timeout cost, and 10% of them fail outright.
2. Distinguish plausible causes of a redis connect timeout. This symptom is consistent with several different underlying problems: redis itself is overloaded or down, the network path to redis is degraded, redis's connection limit has been reached so new connections queue or get refused, or service-b's own connection pool to redis is exhausted or misconfigured (for example too few connections for its current traffic, or connections not being released properly).
3. Use additional signals to narrow it down. Redis's own metrics (CPU, memory, connected-clients count, command latency) would show whether redis itself is unhealthy. Service-b's connection-pool metrics would show whether it's a client-side exhaustion problem rather than a server-side one. A redis-cli PING or a synthetic connection test from the same network path as service-b would show whether the network path itself is the issue, separate from whether redis is overloaded.
4. Immediate remediation. Options depend on what step 3 finds: if redis is overloaded, shedding load or failing over to a replica; if it's a client-side connection-pool exhaustion, increasing the pool size or adding a circuit breaker so service-b fails fast instead of piling up timeouts; if it's the network path, routing around the degraded path if there's an alternative.
5. Long-term fixes. Add a circuit breaker around the redis call so a redis outage degrades service-b gracefully (serving stale or default data, or failing fast) instead of cascading its own latency upstream into service-a and the gateway. Add explicit alerting on redis connection-timeout rate specifically, since it's a distinctive and actionable signal that this trace shows was already present before the wider error rate and latency alerts fired.

Worked example. If redis's own dashboard shows connected_clients pinned at its configured maximum throughout the incident window while CPU and memory on the redis host stay normal, that specifically points to connection exhaustion rather than redis being overloaded: either too many clients are opening connections without closing them (a leak somewhere, possibly in service-b or another caller), or redis's maxclients setting is simply too low for current legitimate traffic. Checking service-b's own connection-pool configuration would tell you which: if service-b's pool size looks reasonable and connections are being released properly, the fix is raising redis's maxclients; if service-b is opening far more connections than its configured pool size suggests it should, the fix is finding and closing the leak in its client code.

Trade-offs and pitfalls. The trap in this kind of RCA is stopping at 'redis connect timeout' and treating that as the root cause, when it's really a symptom with at least four different plausible underlying causes; each one implies a different fix, and applying the wrong one (for example scaling up redis when the real problem is a client-side connection leak) won't resolve the incident. It's also worth being explicit in the writeup about WHY the fix you propose follows from the evidence you gathered, not just from a guess, since that's what lets someone else validate or challenge the conclusion.

Requirements / goal: measure user progress from install → create account → complete profile → first core action; detect regressions quickly and track activation.

Funnel metrics (definitions measured per cohort / day):

• Step counts:

  • Installs: number of unique device installs (first open).
  • Accounts Created: unique users who successfully register (email/SSO/phone verified).
  • Profiles Completed: users who fill required profile fields.
  • First Core Action: users who perform the app’s main action (e.g., post, purchase, send message).

• Step conversion rates:

  • Install → Account = Accounts Created / Installs
  • Account → Profile = Profiles Completed / Accounts Created
  • Profile → First Action = First Core Action / Profiles Completed

Overall activation metric:

• Activation Rate = First Core Action / Installs (captures end-to-end success).

Guardrail metrics (one per step to detect regression):

• Install: Crash rate on first session (crashes / first sessions) — spike may reduce installs→account.
• Account Created: Email/OTP deliverability failure rate or verification rejection rate.
• Profile Completed: Average time to complete profile and abandonment rate on profile screen (drops indicate UX issue).
• First Core Action: Time-to-first-action median (sudden increase = friction) and success/error rate of the action (API failures).

Implementation notes:

• Instrument events with stable names and attributes; send user_id/device_id, timestamp, cohort tag.
• Compute metrics daily and alert on relative drops (e.g., >10% week-over-week) and statistical significance.
• Use A/B cohorts and retention windows (e.g., 7-day) to avoid noise.

Direct answer. A shape-mismatch error tells you two tensors disagreed in dimension somewhere in the forward pass, but the traceback often points at the operation that FAILED, not the operation that introduced the wrong shape several layers earlier, so the debugging process is really about walking the shape forward from the input until it diverges from what you expect.

Step-by-step approach.

1. Print the input shape first, and compare it against what the first layer actually expects. A surprising number of shape bugs are simply "the input isn't shaped the way I assumed," not a bug in the model at all.
2. Use a model summary tool (or manually print .shape after each layer in a quick forward pass) to see the shape at every stage in one pass, rather than binary-searching by commenting out layers one at a time.
3. Add explicit shape assertions inside custom layers, at the point where a specific shape is assumed (assert x.shape[-1] == self.expected_dim, f"got {x.shape}"). This turns a downstream, confusing shape error into an immediate, precisely-located one the next time the bug is triggered, which pays for itself the first time someone else hits a variant of the same bug.
4. Write a small unit test with a known, fixed input shape that exercises just the suspect layer or block in isolation, rather than the whole model, so you can iterate on the fix without paying the cost of a full forward pass through everything else.

A concrete example of why step 1 matters. A very common real case: a model expects batch-first input (batch, seq_len, features) but receives (seq_len, batch, features) from a data loader or a different framework's convention. The shapes are individually valid tensors, nothing crashes until several layers in when a dimension that "coincidentally" matched for a while finally doesn't, at which point the error message points at a layer far from the true cause (the data loader).

The unit test that prevents recurrence. Something as small as:

def test_encoder_output_shape():
    x = torch.randn(4, 10, 32)  # (batch=4, seq_len=10, features=32), the CONTRACT this layer expects
    out = encoder(x)
    assert out.shape == (4, 10, 64), f"expected (4, 10, 64), got {out.shape}"

run in CI on every change to the layer or anything upstream of it, catches this class of bug the moment a shape contract is violated, rather than three deploys later when someone finally notices predictions look wrong.

Trace and span

A trace is the full record of one request's journey through a distributed system, everything that happened to serve it end to end. A trace is made up of spans: each span is one unit of work, like a single RPC call or a single database query, with its own start time, end time, name, and metadata (tags/attributes).

Parent-child relationships

Spans form a tree (or more generally a DAG): a span is the child of whichever span invoked it. For example, an API gateway's span is the parent of the auth-service span it calls, which might itself be the parent of a database-query span. That tree structure is what makes a trace useful beyond "the total time was X": it shows whether work happened sequentially (one span starts only after its parent's other children finish) or in parallel (several child spans overlap in time), which is exactly what you need to reason about where wall-clock time actually went.

Why context propagation matters

Context propagation is passing the trace's identifying information (a trace ID, the current span ID, and the sampling decision) along with the request as it crosses process and network boundaries, typically as HTTP headers (like the W3C traceparent header) or gRPC metadata. Without it, each service only knows about its own local work; there's no way to stitch five separate services' telemetry back into one coherent trace of a single user request.

Common sampling strategies

• Head-based sampling: decide whether to keep a trace right at the start of the request (e.g., keep a random 1%). Cheap and simple, but you're gambling before you know whether the request turns out to be interesting.
• Tail-based sampling: buffer the full trace and decide whether to keep it only after it completes, so you can apply rules like "always keep traces with errors or above a latency threshold." Much better at catching rare, interesting events, but requires a collector that can buffer and hold spans before deciding.
• Adaptive/rate-limited sampling: target a roughly fixed volume of kept traces per second regardless of traffic swings, so sampling doesn't silently drop to near-zero during a traffic spike or flood a backend during a burst.

What tracing helps answer

Which specific hop a regression is coming from, why one particular slow request was slow (as opposed to just "the p99 is high"), whether time was spent sequentially or in parallel, and where a request spent idle/waiting time versus actually doing work.

Trade-offs of high sampling rates and high-cardinality data

Storage and ingestion cost scales roughly with how many traces (and how much detail per trace) you keep, so sampling everything at high fidelity gets expensive fast. High-cardinality attributes on spans (like a full URL with query parameters, or a raw user ID as a tag) multiply that cost further, since backends often index on tag values, and a tag with millions of distinct values blows up that index. The practical trade-off is: sample richly enough (often via tail-based sampling specifically for errors and slow traces) to reliably catch the rare tail events you actually care about, without paying full cost to store the enormous volume of unremarkable, fast, successful traces that make up the bulk of traffic.