Error Handling and Defensive Programming Questions

Direct answer
Debug periodic OOM crashes in a distributed service or container by collecting a memory-growth trend (not just a single snapshot) alongside heap dumps and allocation metrics at the moment of highest pressure, since the SHAPE of the growth (a steady climb versus a sudden spike) is the single most useful signal for narrowing down whether it's a leak, a traffic spike, or a specific request pattern, while applying quick defensive mitigations (memory limits, backpressure) to reduce immediate customer impact during the investigation.

Structured elaboration

• Logs, metrics, dumps to collect: a time-series of process/container memory usage (RSS) at fine granularity across the period leading up to each crash; a heap dump captured automatically when memory crosses a high-watermark threshold (BEFORE the OOM kill, since a dump taken after the process is killed is useless); request-rate and request-SIZE metrics for the same window, to correlate against.
• Analyzing GC logs and heap dumps: for a garbage-collected language, GC logs reveal whether the collector is running MORE frequently and reclaiming LESS each time (a classic leak signature) versus running normally but simply facing more live objects than expected (a traffic or data-size issue, not a leak); a heap dump's object-retention graph shows WHAT is being retained and, often, WHY (an ever-growing cache with no eviction, a listener list nobody unregisters from).
• Quick defensive changes for immediate risk reduction: a hard memory limit with a scheduled graceful restart BEFORE hitting the limit (proactive, controlled restart versus a reactive, uncontrolled OOM-kill) buys time to investigate without repeated hard crashes; backpressure (reject or queue new work once memory crosses a soft threshold) prevents the SYMPTOM from getting worse while the root cause is still being found. Separately from a proactively-chosen memory-limit watermark, register an explicit OOM-condition handler where the platform supports it (a cgroup memory.oom_control eventfd notification on Linux, a container-runtime OOM hook, or catching the language/framework's own out-of-memory exception at a top-level boundary) so the process gets a chance to log its current state and exit cleanly BEFORE the kernel's OOM killer sends an untrappable SIGKILL, which fires at the actual point of resource exhaustion rather than an earlier, proactively-chosen threshold.
• Long-term fixes: once the heap dump/GC-log analysis identifies the actual retained-object pattern, the fix follows directly (bound the cache, unregister the listener, fix the reference cycle); add a regression test specifically asserting memory usage stays bounded over a long-running simulated workload, so the same class of bug doesn't silently reappear later.

Worked example
Memory RSS climbs steadily over roughly 36 hours between each crash, NOT tied to any specific traffic spike; a heap dump taken at the 30-hour mark (via an automated high-watermark trigger) shows a request-scoped cache object growing without bound, retained by a static reference that was meant to be per-request-scoped but was accidentally declared at module/singleton scope; GC logs corroborate this (increasing time spent in GC, decreasing memory reclaimed per cycle, the classic leak signature), confirming a genuine leak rather than a traffic-driven memory footprint increase.

Trade-offs and pitfalls
A scheduled proactive restart is a legitimate SHORT-term mitigation while investigating, but shipping it as the PERMANENT fix (rather than finding and fixing the actual leak) trades a loud, diagnosable crash for a quiet, recurring restart that masks the underlying bug indefinitely and adds real cost (dropped in-flight work on every restart, reduced effective capacity); treat a scheduled restart as a stopgap with a tracked follow-up, not a solution.

Direct answer
Handle SIGTERM/SIGINT by registering a signal handler that flips a shutdown flag, then: stop accepting new inference requests, wait for in-flight requests to finish (or hit a timeout), save a checkpoint atomically, release GPU resources, and notify the orchestrator before exiting, coordinating multiple concurrent checkpoint-savers (if applicable) so they don't deadlock or corrupt each other's output.

Structured elaboration

• Signal handling: a signal handler should do minimal work directly (set a flag, or push an event) rather than perform the shutdown logic itself inline, since signal handlers run in a restricted context where many operations (allocating memory, acquiring certain locks) are unsafe; the actual shutdown logic runs in the main loop, checking the flag.
• Draining in-flight requests: same pattern as the general graceful-shutdown survivor, bounded by a timeout appropriate to typical inference latency (usually much shorter than a general web service's timeout, since inference requests are normally fast).
• Atomic checkpoint save: write to a temp path then rename (see the checkpoint-atomicity survivor), so a shutdown that's itself interrupted partway doesn't leave a corrupted checkpoint file behind.
• Releasing GPU resources safely: explicitly free GPU memory/context (torch.cuda.empty_cache(), releasing any device handles) before exiting, since an abrupt process kill can sometimes leave GPU memory in a state the next process on that device has to work around, and a clean release avoids that.
• Avoiding deadlocks with concurrent checkpoint savers: if multiple processes/ranks in a distributed job all try to save a checkpoint on shutdown, coordinate via a barrier or a single-writer designation (only rank 0 writes the checkpoint) rather than letting every process race to write the same path, which risks a corrupted or partially-overwritten file even with the temp-then-rename pattern, since two processes' temp-then-rename sequences can still interleave badly on the SAME final path.
• Context-manager logic: wrap GPU-resource acquisition/release in a context manager (__enter__/__exit__ or @contextlib.contextmanager) rather than a bare acquire/release pair, so the release runs on every exit path, including an exception raised during the shutdown sequence itself (a failed checkpoint write, say), not just the happy path.
• Exit codes: exit 0 for a clean, fully-completed shutdown (drain finished, checkpoint verified, GPU released); exit with a distinct non-zero code if the shutdown had to force-terminate in-flight work past the drain timeout or the checkpoint save failed, so the orchestrator's restart/alerting logic can distinguish a clean planned shutdown from one that completed in a degraded state, rather than treating every SIGTERM exit identically.

Worked example

import signal, threading, sys
from contextlib import contextmanager

shutdown_event = threading.Event()

def handle_signal(signum, frame):
    shutdown_event.set()  # minimal work in the handler itself

signal.signal(signal.SIGTERM, handle_signal)
signal.signal(signal.SIGINT, handle_signal)

@contextmanager
def gpu_context():
    acquire_gpu()
    try:
        yield
    finally:
        release_gpu()  # always runs, even if shutdown itself raises

def main_loop():
    with gpu_context():
        while not shutdown_event.is_set():
            serve_one_request(timeout=1.0)  # periodically re-checks the flag
        drained = drain_in_flight(timeout=10.0)
        checkpointed = save_checkpoint_atomically()
        notify_orchestrator_ready_to_terminate()
    sys.exit(0 if (drained and checkpointed) else 1)

The handler itself does nothing but set an event; all the actual shutdown logic (draining, checkpointing, releasing) runs in the normal program flow once the main loop notices the flag, which is the safe pattern for what a signal handler is allowed to do.

Trade-offs and pitfalls
Performing heavy work (I/O, complex locking) directly INSIDE the signal handler is a classic bug source, since signal handlers can interrupt the main thread at an arbitrary, unpredictable point, and calling non-reentrant-safe functions from within one can corrupt state or deadlock; the minimal-flag-plus-main-loop pattern shown above sidesteps this entirely and should be the default, not an optimization to reach for only if something breaks.

Direct answer
Safely roll out a new model version with a canary that serves a small percentage of traffic first, monitored against explicit health/correctness/quality metrics with automated rollback triggers, and shadow-evaluate against held-out labels before trusting the canary's live metrics alone, since a model can look 'healthy' (no errors, normal latency) while quietly producing worse predictions.

Structured elaboration

• Health and correctness checks: standard service health (error rate, latency) PLUS ML-specific correctness checks (output shape/range sanity, prediction-distribution comparison against the previous version) before the canary even receives real traffic.
• Automated rollback triggers: error rate or latency regression (standard service-health triggers) AND a model-quality delta metric (if ground truth is available quickly enough, an accuracy/precision delta versus the previous version; if not, a proxy like prediction-distribution shift or downstream business-metric movement).
• Data and label shadowing for verification: route a copy of live traffic to the canary WITHOUT serving its response to users (shadow mode), comparing its predictions against either the current production model's predictions or, once available, actual ground-truth labels, which is the only way to validate real-world quality BEFORE the canary affects any real user.
• Traffic ramp: start at a small percentage (1-5%), hold and monitor for a defined bake period, then ramp incrementally, rather than jumping straight to 50/50, so a bad canary's blast radius stays small during the highest-uncertainty early phase.
• Handling incidents during business-critical hours: define an explicit policy (no canary ramps during a known high-traffic event, an accelerated rollback threshold during those windows) rather than treating every hour identically, since the cost of a bad canary is asymmetric across time.
• Avoiding false-positive rollbacks from transient infra problems: distinguish a genuine model-quality regression from a transient infrastructure blip (a brief network partition, a downstream dependency hiccup, a deploy-unrelated traffic spike) that happens to coincide with the canary window; require a rollback trigger to be sustained across multiple consecutive measurement windows, not a single noisy sample, AND require the divergence to be specific to the canary versus the control group, not an absolute threshold that a shared, upstream infra issue could trip for BOTH canary and control simultaneously.
• Staging model artifacts: promote a model artifact through explicit, versioned stages (a build/validation registry, then a staging environment running the exact serving image, then canary, then full production) so the artifact that gets canaried is byte-identical to what was validated in staging, never rebuilt on the fly at canary time, which would reintroduce a version-skew risk between what was tested and what actually serves traffic.

Worked example
A new recommendation model canary starts at 2% of traffic; automated monitors track click-through-rate delta against the 98% control group over a 4-hour bake window, alongside standard error-rate/latency checks; if CTR drops more than 5% relative to control with statistical significance, or error rate/latency regresses past a threshold, the canary auto-rolls-back and pages the owning team; if metrics hold steady or improve, traffic ramps to 10%, then 25%, then 100% over subsequent bake windows, with each step requiring the SAME health bar to be met before proceeding.

Trade-offs and pitfalls
A canary bake window that's too short can miss a quality regression that only manifests under a specific, less-common traffic pattern (a weekday-vs-weekend behavior shift, say); too long delays shipping genuine improvements and ties up on-call attention longer than necessary. The CTR-delta trigger specifically needs a STATISTICAL SIGNIFICANCE check, not a raw threshold, since a 2% canary sample size can show meaningful random noise in CTR that a naive fixed-threshold trigger would misinterpret as a real regression, causing unnecessary rollback churn.

Direct answer
A good error log entry needs enough fields to reconstruct WHAT happened, WHERE, WHEN, and to WHOM, without needing to reproduce the bug: a timestamp, the service and version, a correlation/trace id, the error type and message, severity, and enough context (which user/request) to investigate, while explicitly excluding PII from the message body.

Structured elaboration

• Minimum fields: timestamp, service, version/deploy_id, correlation_id/trace_id (ties this log line to the specific request and to distributed traces), error_type, message, severity, and a context map for anything structured (which endpoint, which downstream call).
• Why each matters: correlation_id is what lets you find EVERY log line for one failing request across multiple services, which is the single most valuable field for incident investigation; service+version tells you whether a recent deploy correlates with the failure; error_type (not just the message string) lets you aggregate and count occurrences of the SAME underlying bug even if the message includes variable data.
• stdout print vs structured logging: a bare print() produces unstructured text that a log pipeline can only full-text search; a structured JSON logger produces queryable, aggregatable fields, letting you ask 'how many ValidationErrors in the last hour, broken down by field' instead of grepping raw text.
• Avoiding PII: never put raw user-identifying data (email, name, full request body) directly in the message string; reference it by an opaque id (user_id) that can be looked up separately by someone with appropriate access, and redact or hash anything that could be sensitive before it's logged at all.

Worked example

{"timestamp": "...", "level": "ERROR", "service": "checkout-api", "version": "v2.14.1", "correlation_id": "7f3e-9c", "error_type": "PaymentGatewayTimeout", "message": "payment gateway did not respond within 5s", "context": {"user_id": "u_88213", "order_id": "o_44210"}}

A log query for correlation_id: 7f3e-9c across every service instantly reconstructs the full cross-service story of this one failing request; a query for error_type: PaymentGatewayTimeout over the last hour tells you if this is a one-off or a sustained outage, without anyone needing to grep raw text.

Trade-offs and pitfalls
Over-logging (dumping the full request/response body into every error log 'just in case') both risks leaking PII and bloats log storage/cost dramatically at scale; log the minimum structured context that's actually useful for triage, and reference larger payloads by id (stored elsewhere, access-controlled) rather than inlining them. Under-logging (a bare message string with no correlation id) is the more common failure in practice, and it's the one that turns a five-minute investigation into a multi-hour one.

Direct answer
Detect feature-wise data drift with statistical tests (KS test for continuous features, chi-squared for categorical) comparing the current window against a trusted baseline, tuned to balance sensitivity against false positives from ordinary seasonal variance, and route the response by confidence: automated retraining/canary evaluation for clear, well-understood drift, human-in-the-loop review for ambiguous cases.

Structured elaboration

• Statistical tests: the Kolmogorov-Smirnov test compares two continuous distributions' cumulative distribution functions and produces a p-value for 'these are drawn from the same distribution'; chi-squared serves the analogous role for categorical feature distributions. Both need a defined baseline window (a trusted, 'normal' period) to compare against.
• Baseline windows: a rolling baseline (the last N days) adapts to genuine, gradual, benign shifts (seasonal trends) automatically but can mask a slow, GENUINE drift that happens gradually enough to keep pace with the rolling window's own adaptation; a fixed baseline (frozen at model-training time) catches slow drift more reliably but requires explicit, periodic re-baselining as the world genuinely and legitimately changes, or it eventually flags EVERYTHING as drifted.
• Thresholds tuned to reduce false positives: a p-value threshold set too sensitively fires on ordinary day-to-day or week-to-week variance; validate the threshold against HISTORICAL data where you know drift didn't meaningfully occur, tuning until the false-positive rate on that known-clean period is acceptably low.
• Automated retraining pipelines with validation and canary evaluation: for a well-understood, previously-seen drift pattern (a known seasonal shift the model handles fine after retraining), an automated pipeline can retrain, validate against a held-out set, and canary-deploy the new model with minimal human involvement; for a NOVEL or large-magnitude drift, route to a human data scientist to confirm the drift is genuine and understand its likely cause before committing to an automated retrain, since retraining on top of a not-yet-understood distribution shift can itself introduce new problems.
• Reducing false positives: combine drift detection with a CORRELATION check against known events (a real product launch, a new user segment coming online) so a drift alert that correlates with an explainable business event is treated with appropriately lower urgency than an unexplained one.

Worked example
A feature's KS-test p-value drops below the tuned threshold, flagging drift; correlating with a business calendar shows this coincides with a new market launch that legitimately shifted the user population's feature distribution; a human reviewer confirms this is expected, EXPLAINED drift (not a data pipeline bug) and approves an automated retrain against the new, broader distribution, rather than treating it as an anomaly requiring deep investigation.

Trade-offs and pitfalls
A rolling baseline that adapts too quickly can silently absorb a genuine, gradual data-quality degradation (a slowly-breaking upstream pipeline) as 'the new normal', never triggering an alert since each day's shift from the previous day is individually small; periodically comparing against a MUCH older, fixed reference point (not just the rolling recent baseline) catches this class of slow-boil drift that a purely rolling comparison structurally can't.

Schema mismatches and the on-call runbook
Input schema mismatches (a field renamed, removed, or its type changed upstream) are a distinct failure mode from statistical drift and need their own, faster detection path: validate incoming data against an explicit, versioned schema at ingestion, rejecting or quarantining a record that fails validation rather than silently coercing or dropping a field, and alert immediately on any validation-failure rate above a near-zero baseline, since a schema mismatch is rarely 'ordinary variance' the way a slow distribution shift can be.

On-call runbook for triage:

1. Identify which alert fired: a schema-validation failure spike (a structural break, usually caused by a fast-moving upstream deploy) versus a drift alert (a statistical distribution shift, usually a slower-moving phenomenon).
2. For a schema mismatch: check the upstream data producer's recent deploys/schema changes first, since this is almost always caused by an upstream change rather than a gradual phenomenon; the fix is typically updating the schema/pipeline to handle the new format, or asking upstream to revert.
3. For a drift alert: follow the retraining/human-review path above (check for a correlating known business event first, then decide automated retrain versus human review based on the drift's novelty and magnitude).
4. In both cases, page the on-call owner with the SPECIFIC feature/field and the SPECIFIC magnitude of the anomaly, not just 'drift detected', so triage doesn't start from zero.