Error Handling and Defensive Programming Questions

Direct answer
Optimistic locking assumes conflicts are rare and detects them at write time (via a version number or timestamp comparison), rejecting a write if the record changed since it was read; pessimistic locking assumes conflicts are common and prevents them upfront by holding a row lock for the duration of the read-modify-write, blocking other writers until release.

Structured elaboration

• Optimistic locking: read a record along with its version column; on write, UPDATE ... SET data=?, version=version+1 WHERE id=? AND version=? (the original version read); if zero rows are affected, someone else updated it in between, and the caller must re-read and retry. No lock is held during the 'think time' between read and write, so it scales well under low contention and doesn't risk a stuck lock from a slow client.
• Pessimistic locking: SELECT ... FOR UPDATE acquires a row lock at read time, held until the transaction commits or rolls back; other transactions trying to read (with their own FOR UPDATE) or write the same row block until the lock releases. Guarantees no conflict can occur, at the cost of reduced concurrency and the risk of a slow or hung client holding a lock far longer than necessary.
• When to favor which: optimistic locking suits low-contention scenarios (most updates to DIFFERENT rows, occasional collision) like updating a user's own profile; pessimistic locking suits high-contention, correctness-critical scenarios where a retry loop under real contention would thrash (a hot inventory counter during a flash sale) or where the cost of a failed optimistic retry is unacceptable.
• Complementary techniques under contention: idempotency tokens (so a retried write after an optimistic-lock conflict doesn't double-apply a side effect) and compensating transactions (to undo a partially-applied multi-step change) both remain relevant regardless of which locking strategy is chosen, since they address a different axis (safe retry, safe rollback) than conflict detection/prevention itself.

Worked example
Updating a user's account balance to avoid a double-spend under concurrent retries: optimistic locking with a version check catches TWO concurrent updates cleanly (one wins, the other retries with fresh data), but under heavy, sustained contention on the SAME row (many concurrent requests for the same account), most attempts fail their optimistic check and have to retry repeatedly, wasting work; pessimistic locking (SELECT balance FOR UPDATE) serializes access to that one row directly, trading throughput for guaranteed progress without a retry storm.

Trade-offs and pitfalls
Optimistic locking's retry loop needs a bound (max retries, with backoff) or a client can spin indefinitely under sustained contention; pessimistic locking's held lock needs a bound too (a statement timeout, so a stuck transaction doesn't hold the lock forever and block every other writer), and the choice of ISOLATION LEVEL interacts with both strategies in ways that are easy to get subtly wrong (read-committed vs repeatable-read changes what 'the version I read' even means across the transaction).

Direct answer
Coordinate retries across a whole fleet of clients (not just within one client) using either client-side randomized backoff (simple, no new infrastructure, statistically effective at scale but not deterministic), a centralized rate limiter (deterministic and tunable, but a new single point of failure and added latency), or a token-bucket approach (a middle ground: bounded, predictable throughput without a per-request round trip to a central service).

Structured elaboration

• Client-side randomized backoff: each client independently jitters its own retry timing; at fleet scale, the LAW OF LARGE NUMBERS smooths the aggregate retry rate even though no single client coordinates with any other. Simple and resilient (no new dependency), but gives you no hard guarantee on the aggregate retry rate, and a partial outage that resolves right as many clients' jittered windows happen to overlap can still produce a smaller thundering herd.
• Centralized rate limiter: every client checks in with a shared service before retrying, which gives a precise, enforceable cap on aggregate retry rate; the cost is the round trip's added latency on every retry decision, and the rate limiter itself becomes a new dependency that must be more available than the thing it's protecting, or you've just moved the single point of failure.
• Token bucket (typically local, refilled from a shared quota periodically, or fully distributed via a shared store): clients draw from a bucket that refills at a controlled rate; this bounds the aggregate rate without a synchronous round trip on every single retry decision, at the cost of the bucket's refill logic itself needing to be correctly distributed (or accepting some imprecision from local-only buckets, similar to the local-vs-shared circuit-breaker trade-off).
• Per-dependency policies: different downstream dependencies have different real capacity; a client library configuring one global backoff policy for every dependency it calls under-protects a fragile dependency and over-throttles a robust one, so policies should be keyed per dependency, ideally informed by that dependency's own observed error budget and latency (adapting backoff parameters dynamically rather than using a fixed constant).

Worked example
A payment gateway experiences a brief outage; 10,000 clients across a fleet all get a 503 within the same second. With pure client-side jitter alone (say, a 0-5 second uniform window), roughly 2,000 clients still retry within any given one-second slice, which can still be a meaningful spike; layering a centralized token bucket capped at 500 requests/second specifically for THIS dependency smooths that into a bounded, predictable ramp, letting the gateway recover without a second self-inflicted overload from the retry wave itself.

Trade-offs and pitfalls
The common mistake is treating this as an either/or choice: client-side jitter and a centralized (or token-bucket) rate limit are complementary, not competing, layers, since jitter reduces the SEVERITY of any coordination gap and the rate limiter provides the hard guarantee jitter alone can't. Operational complexity scales with the coordination mechanism chosen: pure jitter needs no new infrastructure at all, while a centralized limiter needs its own high-availability design, ironically often more demanding than the dependency it's protecting.

Direct answer
Design a resume/restart mechanism around durable checkpoints written at a controlled granularity, so a crashed job restarts from the last checkpoint rather than from scratch, and make each unit of work between checkpoints idempotent (or transactionally paired with its checkpoint write) so replaying it on resume doesn't duplicate side effects.

Structured elaboration

• Checkpoint granularity: checkpoint too frequently (after every record) and the I/O overhead of writing state dominates throughput; too infrequently (once an hour) and a crash near the end of a window discards a large amount of completed-but-uncommitted work. Choose based on measured throughput and an acceptable 'worst case redo' window.
• Idempotency across resume: if a unit of work has a side effect (writing a row, sending a notification), either make writing that side effect idempotent (an upsert keyed by record id, not a blind insert) or record 'this record's side effect definitely completed' as PART of the same checkpoint write, so a resumed job can skip records already marked done rather than blindly redoing them.
• Transactional updates: where the destination supports it, write the checkpoint marker and the actual output data in the SAME transaction, so a crash between 'wrote the output' and 'wrote the checkpoint' can't happen; if the destination doesn't support that (a streaming sink with no transactions), design for at-least-once output with idempotent writes instead, accepting that resume may reprocess the last incomplete batch.
• Throughput vs checkpoint frequency: this is a direct trade-off; a job processing millions of records can amortize checkpoint overhead by batching (checkpoint every N records or every T seconds, whichever comes first) rather than a fixed record count alone, which adapts to varying per-record processing time.
• State management for a DAG with dependent tasks: a single linear job only needs one progress marker, but a multi-stage DAG needs per-node state (pending, running, succeeded, failed) plus the dependency edges themselves, since resuming correctly means re-running only the nodes that were incomplete AND every downstream node that depends on a node whose output can't be trusted as complete; a node's checkpoint therefore has to record not just 'did this node finish' but 'did it finish with output that its dependents can safely consume', and the resume logic walks the DAG from the crash point forward, skipping any subtree whose upstream inputs are already confirmed durable and re-running any subtree whose upstream state is ambiguous.

Worked example
A nightly batch job processing 10 million records checkpoints its progress (last committed record id) every 50,000 records, in the SAME database transaction as the batch's own output writes. On a crash at record 6.3 million, resume reads the last checkpoint (6.25 million, the last full batch committed), and re-processes records 6,250,001 through 6,300,000; because the output writes are idempotent upserts keyed by record id, reprocessing that small overlap window produces the same end state as if it had never crashed, rather than duplicate rows.

Trade-offs and pitfalls
The single most common bug in this pattern is writing the checkpoint AFTER the side effect but NOT in the same transaction/atomic unit: a crash between the two leaves the side effect applied but the checkpoint not advanced, so resume redoes work that already happened, which is only safe if that redo is genuinely idempotent; if it isn't (a non-idempotent notification send, say), this ordering silently produces duplicate notifications on every crash-and-resume cycle.

Direct answer
A strong answer to this question is specific about the actual code-review COMMENT you left (not just 'I suggested better error handling'), how the discussion actually went (including any pushback, which shows genuine collaboration rather than a one-sided lecture), and what durable practice or pattern resulted from it, since a one-off comment on a single PR is a much weaker story than one that changed how the team works going forward.

Structured elaboration

• The specific comment: name the actual pattern you flagged (a swallowed exception, a missing timeout, an error message that would leak internal details) and what you suggested instead, concretely enough that someone could picture the actual diff.
• The author's response: a genuine story includes some texture here, not just instant agreement; did they push back with a reasonable counterpoint you had to address, or did the comment reveal a broader gap in shared understanding that was worth resolving explicitly rather than just fixing the one line?
• What became a team pattern: did this become a documented convention, a linter rule, a section in an onboarding doc, or just a one-off fix? The strongest version of this story is one where the SAME class of issue is now systematically caught (a rule, a checklist item) rather than just this one PR being improved.

Worked example (a filled skeleton, illustrating the level of specificity expected)
"I reviewed a PR that caught a downstream API's exception and returned None on any failure, which the caller then didn't check before using the result (a null downstream a few lines later). I left a comment showing the specific failure mode this would produce (a confusing null-pointer exception three files away from the actual root cause) and suggested either raising a specific, typed exception the caller would be forced to handle, or explicitly documenting why swallowing was intentional here if it genuinely was. The author initially pushed back that raising would be 'too disruptive' to the existing calling code; we looked at the actual callers together and found none of them checked for None anyway, meaning the swallow was already effectively causing silent failures downstream, which settled the discussion in favor of raising. As a result, we added 'no swallowed exceptions without an explicit comment justifying it' to our team's PR-review checklist, and I later found and fixed two similar pre-existing instances elsewhere in the codebase using the same pattern."

Trade-offs and pitfalls
A review comment that's purely stylistic ('I'd prefer you write it this way') is a much weaker answer than one grounded in a CONCRETE FAILURE MODE the pattern would produce; interviewers listening for depth want to hear that you can articulate WHY a pattern is dangerous, specifically, not just that you have a style preference.

Direct answer
Prevent PII and secrets from leaking into logs and error messages with layered defense: identify which fields are sensitive up front, redact or tokenize them at the point of logging (not as an afterthought), sanitize exception messages specifically (since they often echo raw input verbatim), and catch anything that slips through with an automated CI or runtime scan before it becomes a real incident.

Structured elaboration

• Identifying PII fields: maintain an explicit allowlist or a schema annotation (@sensitive on a field) rather than relying on developers remembering which fields are sensitive; a schema-driven approach means a NEW sensitive field added later is caught by the same mechanism automatically.
• Redaction at ingestion time: strip or mask known-sensitive fields in the logging library itself (a formatter/filter that runs on every log call), so the redaction is enforced centrally rather than trusted to every individual log statement across the codebase.
• Sanitizing exception messages specifically: an exception like ValueError(f"invalid email: {user_email}") embeds the raw PII directly in the message; the fix is either not embedding the raw value at all (ValueError("invalid email format"), letting a separate, redacted context field carry the value if needed) or running the same redaction filter over exception messages before they're logged.
• Tokenization/hashing for correlation: when you need to correlate the SAME user across log lines without exposing their identity, hash or tokenize the identifier (with a stable salt) rather than logging it raw; this preserves debuggability (same user, same token, every time) while removing the raw PII.
• Differential privacy for ML-system logs specifically: for ML logs that aggregate over many users (feature-distribution summaries, per-batch statistics used for monitoring or drift detection), hashing a single user's id is not enough on its own, since an aggregate statistic computed over a small enough cohort can still leak information about a specific individual; adding calibrated noise (a differential-privacy mechanism) to aggregate counts or distribution summaries before they are logged bounds how much any single user's data can influence what's visible in the log, which matters specifically for aggregate ML telemetry in a way that per-request hashing does not address.
• CI/test-time safeguards: a linter or test suite that scans for known-sensitive field names appearing in log/exception-message code paths, and a periodic audit of ACTUAL log output (sampled) against a PII-detection pattern set, catches what code review alone misses.

Worked example
A payment service's validation error originally read f"card number {card_number} failed Luhn check"; redacted to "card number failed Luhn check" with the actual (masked, last-4-digits-only) card reference carried separately in a context field that's itself excluded from the default log sink and only accessible via a restricted, audited query path.

Trade-offs and pitfalls
Debuggability and privacy compliance are in genuine tension: aggressive redaction can strip exactly the detail an engineer needs to diagnose a subtle bug (was the email actually malformed, or did the validation logic have a bug?); the resolution is usually a TWO-TIER approach, redacted-by-default logs for general operational visibility, plus a separate, access-controlled, audited path (not general logs) for the rare case where a specific engineer investigating a specific incident genuinely needs the raw value, with that access itself logged.