Error Handling and Defensive Programming Questions

Covers designing and implementing defensive, fault tolerant code and system behaviors to prevent and mitigate production failures. Topics include input validation and sanitization, null and missing data handling, overflow and boundary protections, exception handling and propagation patterns, clear error reporting and structured logging for observability, graceful degradation and fallback strategies, retry and backoff policies and idempotency for safe retries. Also address concurrency and synchronization concerns, resource and memory management to avoid exhaustion, security related input checks, and how to document and escalate residual risks. Candidates should discuss pragmatic trade offs between robustness and complexity, show concrete defensive checks and assertions, and describe test strategies for error paths including unit tests and integration tests and how monitoring and operational responses tie into robustness.

MediumTechnical

21 practiced

Explain the differences between logs, metrics, and traces for ML observability. For production error handling, which signals would you rely on for alerting vs for debugging, and how would you correlate them to investigate an inference-time failure?

Sample Answer

Logs, metrics, and traces serve different but complementary roles in ML observability.

- Logs: Textual, high-cardinality records (prediction inputs/outputs, model version, stack traces, warnings). Best for root-cause debugging because they capture detailed context and errors.- Metrics: Aggregated numerical time-series (latency, throughput, error rate, prediction distributions like p50/p95 latency, input feature drift scores, accuracy or surrogate labels). Best for alerting and SLIs/SLOs because they're cheap to query, thresholdable, and easy to visualize.- Traces: Distributed, span-based records that show request flow across components (API gateway → preprocess → model server → postprocess). Best for diagnosing performance bottlenecks and correlating where in the pipeline an error or latency spike occurred.

Alerting vs debugging:- Alerting: rely primarily on metrics (error rate, request latency p95, model-confidence drop, input-feature drift exceeding threshold). Use aggregated anomaly-detection on metrics.- Debugging: use logs and traces to investigate incidents the metrics surfaced. Use logs for stack traces, input/output payloads; use traces to locate the failing service/component and timing.

Correlation workflow for an inference-time failure:1. Alert fires on metric (e.g., error rate ↑ or p99 latency spike).2. Query traces for the alert window to find slow/errored spans and affected downstream components.3. Pull correlated logs for trace IDs, request IDs, and model version tags to inspect input features, model outputs, and stack traces.4. Check metrics for related signals (feature drift, upstream latency, hardware/CPU/GPU utilization) and compare model version rollout timeline.5. Form hypothesis (e.g., new model introduced numerical instability, upstream data schema changed) and reproduce with captured inputs from logs in a staging environment.

Best practices:- Attach trace-id/request-id to every log and include model_version, input hash, and confidence in logs.- Export feature distributions and drift metrics regularly.- Maintain alert runbooks linking metric thresholds to the most relevant logs/traces and ownership.

EasyTechnical

24 practiced

Input coming from external systems can include malicious or malformed values. For ML systems that accept feature payloads or serialized artifacts, describe security-oriented input checks you would put in place to defend against injection, deserialization attacks (e.g., pickles), and excessively large payloads.

Sample Answer

Start with a threat model: untrusted external inputs can carry injection payloads (e.g., SQL, shell), crafted feature values to trigger model bugs, malicious serialized artifacts (pickles), or resource-exhaustion via huge payloads. For an ML engineer I’d implement the following practical, defense-in-depth checks:

Input validation & schema enforcement- Strictly validate incoming feature payloads against a schema (types, ranges, required fields). Use JSON Schema / Protobuf with enforced types and reject unknown fields.- Whitelist acceptable fields and value ranges; reject or coerce everything else.

Safe serialization- Avoid unsafe serializers (pickle, eval). Use JSON, Protobuf, or ONNX for artifacts. If legacy pickles must be accepted, only load in an isolated, least-privileged sandbox and verify provenance (signatures).- Require cryptographic signatures (HMAC / public-key) on model artifacts and only accept artifacts with valid signatures.

Deserialization & execution controls- Deserialize only into known classes; disallow arbitrary object construction.- Run any deserialization or model-loading in a constrained process/container with no network access and limited filesystem/privileges.

Payload size & resource limits- Enforce strict max payload size and per-request CPU/memory timeouts.- Apply rate limits and quotas per client/IP, and validate feature vector dimensionality to prevent oversized inputs.

Injection protections- Treat any downstream commands/DB calls separately; use parameterized queries and avoid shelling out with user data.

Logging, monitoring, and fail-safe behavior- Log validation failures and anomalous patterns; alert on spikes.- On invalid input, fail closed (reject) or return safe default predictions rather than attempting risky processing.

Testing & supply-chain hygiene- Fuzz inputs and adversarial tests during CI/CD.- Scan artifacts for known vulnerabilities and maintain strict artifact provenance and versioning.

These controls combined—schema validation, safe formats, signing, sandboxed deserialization, resource limits, and monitoring—provide layered protection against injection, deserialization attacks, and oversized payloads in ML systems.

MediumTechnical

26 practiced

Explain concrete runtime strategies to avoid GPU memory exhaustion during large-model inference or training. Cover: batching size adaptation, gradient accumulation (for training), model sharding, mixed precision, monitoring/gauging memory, and safe failure handling when memory is insufficient.

Sample Answer

Start with an overall principle: trade compute/time for memory and detect memory pressure early so you can adapt before OOM.

Batch size adaptation- Dynamically reduce batch size when memory high. Start with a target batch and probe upward/downward.- Example heuristic: try batch, catch CUDA OOM, halve batch and retry; or binary-search batch size at startup.- Benefits: simplest immediate fix; downside: affects throughput and gradient statistics.

Gradient accumulation (training)- Keep effective large-batch training without increasing per-step memory:

# PyTorch sketch: accumulate gradients over N steps
loss = model(x).loss
loss = loss / accum_steps
loss.backward()
if (step+1) % accum_steps == 0:
    optimizer.step(); optimizer.zero_grad()

- Use with learning-rate scaling. Increases compute/latency but not peak memory.

Model sharding / partitioning- Tensor/model parallelism: shard parameters across GPUs (ZeRO Stage 1/2/3, DeepSpeed, FairScale).- Offload infrequently used parameters to CPU or NVMe (DeepSpeed ZeRO-Offload).- Use pipeline parallelism to split layers across devices.

Mixed precision- Use FP16/AMP to halve activation/parameter memory:

scaler = torch.cuda.amp.GradScaler()
with torch.cuda.amp.autocast():
    output = model(input); loss = loss_fn(output, target)
scaler.scale(loss).backward(); scaler.step(optimizer); scaler.update()

- Watch for numerical stability and tune loss scaling.

Monitoring / gauging memory- Proactively sample torch.cuda.memory_allocated() / memory_reserved() and nvidia-smi.- Track peak memory with torch.cuda.max_memory_allocated() and reset_peak_memory_stats().- Use lightweight probes to estimate extra memory headroom before large ops.

Safe failure handling- Catch OOMs (RuntimeError) and implement graceful fallback: reduce batch, retry, log diagnostics, and surface alerts.- Example pattern:

try:
    train_step()
except RuntimeError as e:
    if 'out of memory' in str(e):
        torch.cuda.empty_cache(); reduce_batch(); retry_or_abort()
    else: raise

- Implement circuit breakers to avoid repeated retries; fall back to CPU inference for small requests if needed.

Trade-offs and operational notes- Mixed precision + sharding gives biggest wins but increases complexity.- Always validate numeric fidelity (FP16) and tune optimizer/learning rates when changing effective batch.- Automate probing at startup and expose telemetry so orchestrator can reschedule or scale resources.

MediumTechnical

27 practiced

You have an external inference service that sometimes fails transiently. Design a retry and idempotency strategy for an ML client that must avoid duplicate downstream side-effects (e.g., billing events). Describe how to generate idempotency keys, where to store them, TTL considerations, and how to handle concurrent retries from multiple clients.

Sample Answer

Clarify goal: ensure retries of requests to an external inference/billing service do not cause duplicate downstream side-effects while tolerating transient failures.

High-level approach- Client includes an idempotency key with every request. The server (or a proxy) deduplicates using an idempotency store and returns the same result for repeated keys.

Idempotency key generation- For user-initiated actions: deterministic key = HMAC(secret, user_id || action_type || action_timestamp_bucket || model_version || input_hash). This allows clients to retry without collisions for identical logical actions.- For single-call tasks where determinism isn’t possible: generate a random UUIDv4 per logical operation and persist it client-side until completion.

Where to store keys- Central idempotency store accessible to all requesters/servers, e.g., Redis (single-shard or clustered) or a strongly-consistent DB.- Schema: key -> {status: pending/success/failure, response_payload/receipt, created_at, ttl}.- Use Redis for low latency; persist important receipts to durable DB asynchronously if required for audits/billing.

TTL considerations- TTL should cover maximum time clients may retry + audit window. Example: 24–72 hours for billing actions; shorter (minutes) for ephemeral predictions.- Choose TTL = max(expected retry window, legal/audit retention requirement). Extend TTL on access if necessary.

Handling concurrent retries / race conditions- Use atomic operations: - Redis SETNX(key, placeholder) + EXPIRE to claim processing right (returns true for first actor). - If SETNX succeeds: process request, store final response and set status=success. - If SETNX fails: read stored status: - status=pending: either wait/poll with backoff for result or read a cached response if available. - status=success: return stored receipt/result immediately.- Alternatively use compare-and-swap (CAS) or DB transactions to ensure single-writer semantics.- For multi-client scenarios, clients should poll for final status with exponential backoff and jitter; avoid busy-wait loops.

Retry strategy- Exponential backoff with jitter; limit retries (e.g., 5–7 attempts).- Retry only idempotent-safe errors (network timeouts, 5xx); do not retry on 4xx that indicate client error.- Include the idempotency key and a client request timestamp with every retry.

Edge cases and best practices- Ensure idempotency key scope is well-defined (per-user, per-account, per-model).- Protect keys: sign keys (HMAC) so server can validate authenticity and avoid malicious replay.- Log events (claims, successes, duplicates) for observability and reconciliation.- For long-running processes, store a lightweight receipt immediately to allow clients to observe progress.- For highly critical financial side-effects, write to durable storage synchronously once claim succeeds before sending downstream billing event.

Example (Redis claim + process, Python sketch):

python

import uuid, time
import redis
r = redis.Redis()

key = f"idemp:{uuid.uuid4()}"
if r.setnx(key, "pending"):
    r.expire(key, 86400)  # 24h TTL
    # perform external call (with retries)
    result = call_inference()
    r.hmset(key, {"status":"success","resp":result,"ts":time.time()})
else:
    record = r.hgetall(key)
    # return existing resp or poll until status != pending

This design ensures a single effective action per logical operation, low-latency deduplication, safe concurrent retries, and configurable retention to meet business/audit needs.

EasyTechnical

21 practiced

Numeric features sometimes contain extreme values or out-of-range inputs that can cause overflow or unstable behavior in models (e.g., exp overflow). Describe defensive protections you would implement when preprocessing numeric features and in model code to avoid overflow, underflow, or catastrophic activation values.

Unlock Full Question Bank

Get access to hundreds of Error Handling and Defensive Programming interview questions and detailed answers.

Join thousands of developers preparing for their dream job.