Error Handling and Defensive Programming Questions

Covers designing and implementing defensive, fault tolerant code and system behaviors to prevent and mitigate production failures. Topics include input validation and sanitization, null and missing data handling, overflow and boundary protections, exception handling and propagation patterns, clear error reporting and structured logging for observability, graceful degradation and fallback strategies, retry and backoff policies and idempotency for safe retries. Also address concurrency and synchronization concerns, resource and memory management to avoid exhaustion, security related input checks, and how to document and escalate residual risks. Candidates should discuss pragmatic trade offs between robustness and complexity, show concrete defensive checks and assertions, and describe test strategies for error paths including unit tests and integration tests and how monitoring and operational responses tie into robustness.

MediumTechnical

23 practiced

Implement an LRU cache decorator in Python to cache model predictions keyed by an input hash. The decorator should support a configurable maximum size, be thread-safe for concurrent access, and update recency on both reads and writes. Include a short usage example wrapping a dummy predict(input) function.

Sample Answer

Approach: use collections.OrderedDict to maintain recency (most-recent at end), protect access with threading.RLock to be safe for reentrancy, and implement a decorator factory that accepts max_size and optional key function. On each get or set, move the key to the end; when size exceeds max, evict oldest.

python

import threading
from collections import OrderedDict
from functools import wraps
import hashlib
import pickle

def default_key_func(args, kwargs):
    # deterministic hash of inputs (works for picklable inputs)
    data = (args, tuple(sorted(kwargs.items())))
    h = hashlib.sha256(pickle.dumps(data)).hexdigest()
    return h

def lru_cache_decorator(max_size=128, key_func=None):
    if key_func is None:
        key_func = default_key_func

    def decorator(func):
        cache = OrderedDict()
        lock = threading.RLock()

        @wraps(func)
        def wrapper(*args, **kwargs):
            key = key_func(args, kwargs)
            with lock:
                if key in cache:
                    # update recency on read
                    value = cache.pop(key)
                    cache[key] = value
                    return value

            # compute without holding the lock for whole call
            result = func(*args, **kwargs)

            with lock:
                # update recency on write (may overwrite)
                if key in cache:
                    cache.pop(key)
                cache[key] = result
                # evict oldest if needed
                while len(cache) > max_size:
                    cache.popitem(last=False)
            return result

        # optional introspection helpers
        wrapper._cache = cache
        wrapper._lock = lock
        return wrapper
    return decorator

Example usage:

python

@lru_cache_decorator(max_size=3)
def predict(x):
    # dummy expensive work
    return f"prediction_for_{x}"

Key points:- Thread-safe via RLock; compute happens outside lock to avoid blocking other threads.- Recency updated on both reads and writes by moving key to end.- Uses deterministic input hash (pickle + sha256); you can supply a custom key_func for non-picklable inputs.Time complexity: O(1) average for get/set (OrderedDict operations). Space: O(max_size).Edge cases: unpicklable args (provide custom key_func), long-running compute causing duplicate concurrent computations (could add in-progress sentinel or per-key locks if needed).

MediumTechnical

25 practiced

How would you design inference orchestration to tolerate partial failures such as tokenizer timeouts, feature-store slowdowns, or remote model-shard unavailability? Describe timeout and deadline policies, circuit-breakers, how to return partial results with metadata, and how to avoid cascading failures.

Sample Answer

Requirements & goals:- Low-latency, robust inference pipeline that tolerates partial failures (tokenizer, feature store, remote model shards).- Provide best-effort results with clear metadata and avoid cascading outages.

Design overview:- Pipeline stages: preprocessor/tokenizer → feature-store lookup → model shard routing → aggregation/postprocess → response.- Each stage has local timeout, overall request deadline, and fallbacks.

Timeouts & deadlines:- Per-stage timeouts tuned from SLAs (e.g., tokenizer 50ms, feature-store 100–200ms, remote shard 300ms). Use progressive deadlines so sum(stage_timeouts)+buffer ≤ end-to-end SLA.- Enforce an absolute request deadline (e.g., 1s). If deadline reached, cancel in-flight ops to avoid wasted resources.

Circuit breakers & retries:- Per-dependency circuit breakers (breaker state, error rate, cooldown). If feature-store slowdowns or shard errors exceed threshold, open circuit to short-circuit calls and use fallback path (cached features, model replica, degraded model).- Use retry with exponential backoff only for idempotent calls and within per-stage timeout budget; limit retries to avoid overload.

Returning partial results with metadata:- Return what’s available (e.g., model outputs for available shards + default or sampled logits for missing shards).- Include structured metadata: which stages succeeded/failed, latency per-stage, confidence/quality flags, whether fallbacks used, and partial=true.- Provide an API flag asking client to accept partial results or wait until full.

Avoid cascading failures:- Bulkhead isolation: per-tenant/route resource pools and rate limits so slow dependency doesn't starve other requests.- Backpressure: reject or queue low-priority requests when downstream is overloaded.- Degraded modes: serve cached responses, simpler model, or heuristic fallback when critical components fail.- Observability & automation: expose metrics, alerts, and automated breaker tuning; run chaos tests for timeout thresholds.

Example flow:- Tokenizer times out → use lightweight tokenizer fallback, mark metadata tokenization_fallback=true.- Feature-store slow → fall back to local cache or default features; circuit breaker opens after repeated timeouts.- Model-shard unavailable → route to alternative shard replica or return partial logits + metadata indicating missing shard IDs.

This approach prioritizes predictable SLAs, graceful degradation, and transparent client-facing metadata so consumers can make safe decisions on partial outputs.

MediumTechnical

22 practiced

Design a set of monitoring alerts and thresholds to detect model input schema mismatches and model performance drift (e.g., sudden distribution shifts, rising error rate). For each alert describe mitigation steps, who to notify, and the on-call runbook for triage and rollback.

Sample Answer

Below is a practical monitoring plan for detecting input schema mismatches and model performance drift, with mitigation steps, notification routing, and an on-call triage/rollback runbook.

1) Alert: Schema Violation — Critical- Trigger: >0.5% of requests in 5m fail schema validation (missing fields, wrong types, unexpected categorical values).- Notify: On-call ML engineer + Data Platform Slack channel; Pager for critical APIs.- Mitigation: - Short-term: Send failing examples to quarantine topic; route affected traffic to a safe fallback model/version or API that returns cached/default responses. - Long-term: Fix upstream data producer or update preprocessing/validation rules.- Runbook: 1. Confirm alert details and sample failing payloads from quarantine. 2. Check recent deployments or data pipeline changes (CI/CD, Kafka Connect). 3. If source issue: engage data owner; if model-side mapping error: revert latest model/preprocessor. 4. If impact high, rollback to last-known-good model and disable automated retraining until root cause fixed. 5. Postmortem and add new validation rules/tests.

2) Alert: Feature Distribution Shift — Warning → Critical- Trigger: K-S test p-value < 0.01 or population-level PSI > 0.2 on key numeric features over 1h (warning at PSI>0.1).- Notify: ML engineer + Product owner (email/slack).- Mitigation: - Short-term: Throttle/flag predictions for affected segments; tag data for retraining. - Long-term: Retrain model with recent data, add covariate shift correction (importance weighting) or feature transformation.- Runbook: 1. Inspect which features drifted and time window; correlate with deployments/events. 2. Validate upstream data schema and transformation code. 3. Run inference on replayed historical dataset to measure impact. 4. If performance unaffected, mark as monitored change; otherwise schedule urgent retrain or rollback.

3) Alert: Rising Error Rate / Performance Degradation — Critical- Trigger: Model metric (e.g., RMSE, top-1 accuracy, F1) degrades >5% absolute or >20% relative vs baseline over 1h, or SLA breach on downstream business metric.- Notify: Pager to ML on-call, Engineering manager, Product owner; create incident ticket.- Mitigation: - Short-term: Switch traffic to previous model version or degrade to simpler rule-based logic; increase monitoring granularity. - Long-term: Debug root cause, assemble retraining pipeline, possibly augment labels or feature set.- Runbook: 1. Confirm metric source; validate label freshness and correctness (delayed labels can cause false alarms). 2. Collect sample inputs and outputs, compare predictions vs recent labels. 3. Check for correlated schema or data drift alerts. 4. If verified, perform immediate rollback to last-good model and stop automated deployments. 5. Run targeted A/B tests with candidate fixes, then promote once validated.

4) Alert: Feature Missing Rate Spike — Warning- Trigger: Any required feature missing in >1% requests over 15m.- Notify: ML engineer + Data owner slack.- Mitigation: - Short-term: Impute missing values using median/default; route to fallback if imputation unacceptable. - Long-term: Fix upstream producer, add stricter contracts and unit tests.- Runbook: 1. Inspect telemetry to find affected clients/endpoints. 2. Apply temporary imputations if safe. 3. Notify and coordinate with data producer to deploy fix.

Operational best practices:- Maintain a runbook document with rollback commands, model registry links, dataset snapshots, and owner contacts.- Automate quarantining, alert enrichment (examples + diffs), and dashboards showing correlated system events (deployments, config changes).- Use canary and staged rollouts to reduce blast radius; require automated pre-deployment checks (schema, sample inference).- Post-incident: add synthetic tests and CI schema checks to prevent recurrence.

EasyTechnical

31 practiced

In Python, describe idiomatic exception-handling patterns for an AI inference microservice. For cases like model-load failure, corrupted input, and downstream timeouts, when should the service catch the exception locally and transform it (e.g., return 4xx/5xx) versus allowing it to propagate to an upper-level orchestrator or monitoring system?

Sample Answer

For an AI inference microservice, prefer explicit, minimal, and context-aware exception handling: catch and transform errors at the boundary where you can give a meaningful, actionable response; let exceptions propagate where higher-level orchestrators/monitoring can decide retries, failover, or alerts.

Patterns and guidance:- Validate inputs early and raise/return 4xx for client faults (bad schema, missing fields, corrupted payload).- Translate predictable runtime problems into clear HTTP codes: model not loaded -> 503 (service unavailable) or 500 depending on permanence; downstream timeout -> 504.- Use custom exceptions for domain clarity (e.g., ModelLoadError, InvalidInputError, BackendTimeout) so handlers can map to responses/metrics.- Let unexpected/unhandled exceptions bubble up to global middleware/ASGI framework so centralized logging, tracing, and error policies run.- Record structured logs, metrics, and error attributes (error type, model version, input id) at catch points.

Example:

python

class InvalidInputError(Exception): pass
class ModelLoadError(Exception): pass

async def predict(request):
    try:
        payload = await request.json()
        validate_schema(payload)  # raise InvalidInputError on bad input
        if not model.is_ready():
            raise ModelLoadError("model not loaded")
        return web.json_response(model.predict(payload))
    except InvalidInputError as e:
        # client error -> 400
        return web.json_response({"error": str(e)}, status=400)
    except ModelLoadError as e:
        # transient/unavailable -> 503; increment metric
        metrics.counter("model_load_fail").inc()
        return web.json_response({"error": "model unavailable"}, status=503)
    except asyncio.TimeoutError:
        # downstream or GPU stall -> 504
        return web.json_response({"error": "inference timeout"}, status=504)
    # let other exceptions propagate to global middleware for alerting/tracing

Best practices:- Centralize mapping of exception -> HTTP status in middleware.- Use retries/circuit-breakers for downstream calls, but avoid retrying client-caused errors.- Fail fast on corrupted inputs; degrade gracefully (cached response, read-only mode) on model load issues.- Ensure health endpoints reflect model readiness (so orchestrator can stop routing requests instead of relying solely on 503 responses).

HardSystem Design

43 practiced

Design a GPU cluster scheduler that defends against resource exhaustion from misbehaving jobs. Include per-user and per-team quotas, preemption policies, OOM detection and automated mitigation, container/cgroup isolation, graceful eviction with checkpointing, and fairness policies. Explain how to balance throughput, fairness, and responsiveness for interactive jobs.

Sample Answer

Requirements:- Functional: per-user & per-team GPU/CPU/memory quotas, OOM detection + mitigation, preemption, checkpoint+graceful eviction, container/cgroup isolation, fairness (short/long jobs), interactive responsiveness.- Non-functional: low-latency scheduling for interactive jobs (<500ms), high throughput for batch training, multi-tenant safety, scalable to thousands of GPUs.

High-level architecture:API Gateway → Scheduler + Quota Manager + Fairness Engine → Node Agents (container runtime, cgroups, GPU monitoring) → Distributed Storage (checkpoints), Metrics/Alerting, Preemption Controller

Core components:1. Quota Manager: stores per-user/team quotas with token-bucket-like allocation and soft/hard limits. Enforces admission: deny or queue if over hard limits; allow with throttling under soft limits.2. Scheduler: two-level hybrid (global resource pool + local bin-packing). Maintains queues: interactive (low latency), batch (throughput). Uses priority classes and aging to prevent starvation.3. Fairness Engine: implements Dominant Resource Fairness (DRF) with weights by team; enforces min-share guarantees and uses deficit-round-robin for interactive fairness.4. Preemption Controller: supports two policies — graceful preemption (checkpoint + SIGTERM grace window) and immediate reclaim for emergencies. Preemption decisions consider job priority, checkpoint cost, and estimated work lost.5. Node Agent: runs containers in cgroups + NVIDIA container toolkit; enforces memory/GPU isolation, monitors OOM, GPU ECC, power events. Reports metrics to scheduler.6. OOM Detector & Auto-mitigation: multi-tier detection — kernel OOM events, cgroup memory pressure, framework-level OOM logs. Mitigation: (a) reduce container memory via dynamic limits if supported, (b) throttle AI workers, (c) checkpoint & evict, (d) notify user with diagnostics.7. Checkpointing: integrate framework hooks (e.g., PyTorch checkpoint API) + lightweight snapshot (model weights + optimizer). Scheduler exposes a checkpoint API and stores to distributed object store; cost model used in preemption decisions.

Data flow:- Job submit includes user/team, priority, checkpoint capability, resource profile.- Quota Manager validates; Scheduler picks node via scoring (fit, data locality, preemption cost).- Node Agent enforces cgroups; metrics stream to Scheduler for adaptive decisions.

Balancing throughput, fairness, responsiveness:- Two queues: interactive jobs get small guaranteed share and fast path scheduling (reserve fast GPUs, preemptible low-priority batch). Batch uses backfill and gang-scheduling windows.- DRF weighted by team ensures long-term fairness; short-term responsiveness achieved by giving interactive jobs a small excess burst token bucket.- Preemption policy: prefer preempting jobs with cheap checkpoint cost and low team min-share impact. Use predictive models (job runtime and checkpoint time) to estimate eviction cost vs. user wait latency.- Throughput: enable elastic scaling (add ephemeral preemptible GPUs) and container packing; avoid overfragmentation via defragmenter that migrates queued jobs during low load.- Responsiveness vs throughput trade-off: tune interactive burst size and checkpoint frequency; higher checkpoint frequency reduces eviction pain but costs runtime. Expose policies per team (SLA tiers).

Failure and abuse handling:- Rate-limit job submissions per user; circuit-breaker to quarantine users with repeated OOMs or resource overuse.- Audit logs + cost allocation for accountability.

Trade-offs:- Strong isolation via cgroups and per-GPU MIG increases safety but may reduce packing efficiency.- Aggressive preemption improves responsiveness but lowers batch throughput—mitigated by cost-aware preemption and checkpointing.

Metrics to monitor:- Pod start latency, GPU utilization, preemption count, checkpoint time, fairness index (Jain/DRF deviation), OOM incidents.

This design defends against resource exhaustion while balancing throughput, fairness and interactivity through quotas, DRF, cost-aware preemption, and automated OOM mitigation integrated with checkpointing.

Unlock Full Question Bank

Get access to hundreds of Error Handling and Defensive Programming interview questions and detailed answers.

Join thousands of developers preparing for their dream job.