Error Handling and Defensive Programming Questions

Direct answer
Optimistic locking assumes conflicts are rare and detects them at write time (via a version number or timestamp comparison), rejecting a write if the record changed since it was read; pessimistic locking assumes conflicts are common and prevents them upfront by holding a row lock for the duration of the read-modify-write, blocking other writers until release.

Structured elaboration

• Optimistic locking: read a record along with its version column; on write, UPDATE ... SET data=?, version=version+1 WHERE id=? AND version=? (the original version read); if zero rows are affected, someone else updated it in between, and the caller must re-read and retry. No lock is held during the 'think time' between read and write, so it scales well under low contention and doesn't risk a stuck lock from a slow client.
• Pessimistic locking: SELECT ... FOR UPDATE acquires a row lock at read time, held until the transaction commits or rolls back; other transactions trying to read (with their own FOR UPDATE) or write the same row block until the lock releases. Guarantees no conflict can occur, at the cost of reduced concurrency and the risk of a slow or hung client holding a lock far longer than necessary.
• When to favor which: optimistic locking suits low-contention scenarios (most updates to DIFFERENT rows, occasional collision) like updating a user's own profile; pessimistic locking suits high-contention, correctness-critical scenarios where a retry loop under real contention would thrash (a hot inventory counter during a flash sale) or where the cost of a failed optimistic retry is unacceptable.
• Complementary techniques under contention: idempotency tokens (so a retried write after an optimistic-lock conflict doesn't double-apply a side effect) and compensating transactions (to undo a partially-applied multi-step change) both remain relevant regardless of which locking strategy is chosen, since they address a different axis (safe retry, safe rollback) than conflict detection/prevention itself.

Worked example
Updating a user's account balance to avoid a double-spend under concurrent retries: optimistic locking with a version check catches TWO concurrent updates cleanly (one wins, the other retries with fresh data), but under heavy, sustained contention on the SAME row (many concurrent requests for the same account), most attempts fail their optimistic check and have to retry repeatedly, wasting work; pessimistic locking (SELECT balance FOR UPDATE) serializes access to that one row directly, trading throughput for guaranteed progress without a retry storm.

Trade-offs and pitfalls
Optimistic locking's retry loop needs a bound (max retries, with backoff) or a client can spin indefinitely under sustained contention; pessimistic locking's held lock needs a bound too (a statement timeout, so a stuck transaction doesn't hold the lock forever and block every other writer), and the choice of ISOLATION LEVEL interacts with both strategies in ways that are easy to get subtly wrong (read-committed vs repeatable-read changes what 'the version I read' even means across the transaction).

Direct answer
A batch/ETL job that exits with status zero but produces missing or empty output is invisible to exit-code-based monitoring by definition, so detection needs to look at the OUTPUT itself: row counts against an expected watermark, checksums, heartbeats during long-running processing, and manifest files that explicitly declare what SHOULD have been produced, so a comparison against the manifest catches a silent gap.

Structured elaboration

• Row-count/watermark checks: compare the job's output row count against a rolling historical baseline or an explicit expected minimum (derived from the input's own row count, for a 1:1 or bounded-ratio transformation); a sudden drop to zero or a small fraction of the expected count is a strong silent-failure signal even though the process exited cleanly.
• Checksums: for a job that should be deterministic given its input, a checksum mismatch against a previous run's output (when the input is known to be unchanged) flags a silent processing bug even when row counts happen to look normal.
• Heartbeats: a long-running job that emits a periodic 'still alive, processed N so far' signal lets you detect a job that's silently STUCK (not crashed, not producing output, but also not progressing) well before its eventual timeout, distinct from a job that completes 'successfully' with empty output.
• Manifest files: an explicit, separate file declaring what output files/partitions THIS run was supposed to produce, written as the job's LAST step; a downstream consumer or monitor checks the manifest against what's actually present, catching a job that silently skipped a partition without producing an error.
• Idempotent re-run strategy: once a silent failure is detected, the remediation is typically 'safely re-run the job for the affected window', which requires the job to be idempotent (re-running for the same input window produces the same correct output, not duplicated or corrupted data), tying this directly back to the idempotency survivors.

Worked example
A nightly ETL job normally processes ~2 million rows and writes a manifest listing the expected output partitions; one night, an upstream schema change causes every row to fail a silent validation branch that logs nothing and just skips the row, so the job 'succeeds' with zero rows written, and the exit code is 0. A row-count check comparing against the 7-day rolling average (2M expected, 0 actual) fires an alert immediately after the job completes, well before any downstream consumer of the (empty) output would have discovered the gap on their own, likely hours or days later via a completely unrelated symptom.

Trade-offs and pitfalls
A row-count threshold set as a fixed number rather than relative to a rolling baseline breaks the first time legitimate traffic/data volume genuinely changes (a real, expected growth in input volume shouldn't trip a stale fixed threshold); use a relative, rolling comparison, and revisit the threshold periodically as the baseline naturally shifts. Manifest-based detection only works if the manifest write is the ABSOLUTE LAST step and is itself verified as part of the check (a job that crashes AFTER writing a correct manifest but BEFORE actually writing all the data would falsely appear complete); order operations so the manifest genuinely reflects verified, completed output, not just intended output.

Direct answer
Build alerting on SLOs (not raw error counts) so pages correlate with actual user-facing pain: use an error BUDGET to gate how aggressively you alert, distinguish symptom alerts (something is actually broken for users right now) from cause alerts (a likely contributing factor, lower urgency), and require a minimum sample size before firing so low-traffic noise doesn't page anyone.

Structured elaboration

• Metrics feeding the SLO: typically a ratio (successful requests / total requests) measured against an explicit target (99.9% success over a rolling window), NOT a raw error count, since raw counts don't distinguish '10 errors out of 100 requests' (10% error rate, bad) from '10 errors out of 1,000,000' (negligible).
• Symptom vs cause alerts: a symptom alert ('user-facing error rate exceeds SLO threshold') should page immediately, since it directly reflects user pain; a cause alert ('CPU usage is elevated') is a likely contributing factor but not itself proof of user impact, and should typically be visible on a dashboard or trigger a lower-urgency notification rather than a page, since elevated CPU alone doesn't necessarily mean anyone is affected.
• Error budgets gating alerts: track how much of the SLO's allowed error budget has been consumed over the current period; alert more aggressively (shorter time windows, lower thresholds) as the budget depletes, since burning through budget fast (a 'fast burn') threatens the SLO commitment much sooner than a slow, steady trickle of errors within otherwise-normal bounds.
• Minimum sample size: error_ratio > 1% computed over 5 requests is meaningless noise; requiring a minimum request count (e.g. 1000) before evaluating the ratio prevents low-traffic periods (overnight, for a low-volume service) from producing statistically meaningless alerts.
• Grouping/sampling for a noisy downstream integration: a specific downstream integration that is chronically flaky in a way that is well understood and already tolerated (a third-party API with a known baseline 2% error rate that the product already accounts for) will otherwise dominate a shared SLO's error budget and drown out signal from everything else calling into the same service; carve that dependency's calls into their OWN separate SLO/alert group (or exclude its known-tolerated error class from the primary SLO's numerator entirely) and sample its errors at a lower rate for logging/tracing purposes than a genuinely unexpected error type, so the noisy-but-understood integration doesn't consume alert budget or storage that a rare, unexpected failure needs.
• Combining static thresholds with anomaly detection: a static threshold (error ratio > 1%) is simple and predictable but can't distinguish 'normal daily variance' from a genuine anomaly for a service whose baseline error rate itself fluctuates; layering an anomaly-detection signal (current rate significantly deviates from the same time-of-day/day-of-week historical baseline) catches issues a fixed threshold misses without needing to hand-tune a threshold per time-of-day.

Worked example
PromQL-style expression: sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 0.01 and sum(rate(http_requests_total[5m])) * 300 > 1000 fires only when the 5-minute error ratio exceeds 1% AND the 5-minute window saw at least 1000 total requests, avoiding a false alarm during a low-traffic window where a handful of errors could otherwise spike the ratio meaninglessly.

Trade-offs and pitfalls
An SLO-based alert threshold set too close to the actual SLO target itself pages on every minor blip that doesn't genuinely threaten the SLO commitment over its real measurement window; the fast-burn/slow-burn distinction (alert urgently on a fast burn rate that would exhaust the budget in hours, alert less urgently on a slow burn that would exhaust it over weeks) is what lets you page appropriately without either missing a real emergency or drowning on-call in noise from ordinary variance.

Direct answer
Build a structured, JSON-serializable log record from an exception and a context dict using only the standard library, so it can be handed directly to a JSON-based logging pipeline without any custom serialization step, and keep the message field free of anything sensitive by construction (never dumping raw request payloads into it).

Structured elaboration

• Pull the essential fields directly from the exception object (type(exc).__name__, str(exc)) rather than requiring the caller to pass them redundantly, since the caller might forget or get them wrong.
• Accept a context dict for caller-supplied fields (service name, correlation id) that the exception object itself has no way to know.
• Keep short_stack bounded (truncate) so a pathological exception with a huge repr doesn't blow up log line size.

Worked example (executed; all assertions passed, including the JSON-serializability check)

def format_error_log(exc, context):
    return {
        "timestamp": now_iso(),
        "level": "ERROR",
        "service": context.get("service", "unknown"),
        "correlation_id": context.get("correlation_id"),
        "error_type": type(exc).__name__,
        "message": str(exc),
        "short_stack": repr(exc)[:200],
    }

Verified: raising ValueError("bad input: age=-5") and formatting it with {"service": "user-api", "correlation_id": "req-42"} produces a dict that json.dumps() successfully serializes (confirming no non-serializable objects, like a raw exception or a datetime object, leaked into the record) with error_type == "ValueError" and the correlation id correctly threaded through from the context.

Trade-offs and pitfalls
Using str(exc) directly as the message field trusts that whoever RAISED the exception didn't embed sensitive data in its message text; this function can't retroactively sanitize a bad message, so the discipline has to live upstream, at every raise site, not here. Truncating short_stack to 200 characters is a pragmatic default but can cut off the most useful part of a deep stack trace for a complex failure; pair this lightweight per-line record with a separate, fuller stack-trace capture (sent to an error-tracking service, not inline in every structured log line) for genuinely complex failures.

Direct answer
A layered test strategy for error paths runs unit tests that simulate a dependency failure in isolation, integration tests using test doubles for more realistic multi-component interaction, contract tests that verify a downstream dependency's actual failure-mode behavior matches what your code assumes, and failure-injection/chaos tests against real dependencies, gated in CI as blocking (fast, deterministic tiers) or non-gating/scheduled (slower, real-dependency tiers).

Structured elaboration

• Unit tests, simulated failure: mock a single dependency to raise a specific exception, asserting the function under test handles it exactly as designed (falls back, retries, propagates a specific error); fast, deterministic, runs on every commit.
• Integration tests, test doubles: exercise a fuller call path (multiple real internal components, but an external dependency still faked via a test double) so you catch integration bugs a pure unit test's narrow mock can't (an internal component genuinely NOT calling the retry logic the unit tests assumed it would use, say).
• Contract tests: verify that your MOCK of an external dependency's failure behavior actually matches the dependency's REAL behavior (a contract test run periodically against the real dependency's staging environment, or a shared, versioned contract both sides agree to and test against independently); without this, a unit test can pass forever against an increasingly-stale assumption about how the real dependency actually fails.
• Failure-injection/chaos tests: exercise the REAL dependency under an injected real fault (see the companion chaos-game-day survivor), the only tier that can catch a bug in the ACTUAL integration, not just your code's internal logic or a test double's approximation of the real thing.
• CI gating: unit and integration tests (fast, deterministic) gate every merge; contract and chaos tests (slower, sometimes dependent on external infrastructure availability) typically run on a schedule or a separate, non-blocking pipeline, with failures triaged and acted on promptly rather than being blocking-but-flaky in the main merge path.

Worked example
Testing a client's retry-on-503 logic: the UNIT test mocks the HTTP client to return a 503 once then a 200, asserting the retry fires and the final result is correct; the INTEGRATION test runs the real client code against a local test-double HTTP server configured to behave the same way, catching any bug in how the client code is actually WIRED UP (a config value not actually being passed through, say); the CONTRACT test periodically confirms the real downstream service genuinely returns a 503 (not, say, a 500 or a connection reset) for the specific failure condition your retry logic assumes, since a downstream API change could silently invalidate that assumption; a CHAOS test in staging injects a REAL transient network failure against the real dependency and confirms the whole system, end to end, recovers as expected.

Trade-offs and pitfalls
The contract-test tier is the one most commonly skipped, and its absence is exactly what lets a downstream dependency's behavior silently drift away from what your unit tests still (incorrectly) assume, with every unit test continuing to pass the whole time; treat contract-test coverage of your genuinely load-bearing failure-mode assumptions as a first-class investment, not an afterthought.