Error Handling and Code Quality Questions

Focuses on writing production quality code and scripts that are defensive, maintainable, and fail gracefully. Covers anticipating and handling failures such as exceptions, missing files, network errors, and process exit codes; using language specific constructs for error control for example try except blocks in Python or set minus e patterns in shell scripts; validating inputs; producing clear error messages and logs; and avoiding common pitfalls that lead to silent failures. Also includes code quality best practices such as readable naming and code structure, using standard libraries instead of reinventing functionality, writing testable code and unit tests, and designing for maintainability and observability.

HardSystem Design

80 practiced

Design an orchestration pattern for long-running data-preprocessing DAGs where downstream tasks depend on upstream outputs. The system should handle failures, retries, idempotent re-runs, and avoid duplicate work when tasks are restarted. Describe state management and task-level error handling.

Sample Answer

Requirements & constraints:- Long-running DAGs (hours+), downstream tasks depend on upstream outputs.- Must support retries, failure handling, idempotent re-runs, and avoid duplicate work.- Observable, scalable, easy operator recovery.

High-level architecture:- Orchestrator: Apache Airflow (orchestration + DAG definition).- Worker runtime: Kubernetes pods executing containerized tasks.- Durable state store: Relational DB (Postgres) or distributed key-value (DynamoDB) for task metadata + idempotency markers.- Artifact store: Object storage (S3/GCS) for outputs and atomic manifests.- Message bus (optional): Kafka for event-driven triggers and lineage.

Core pattern:1. Task contract: every task is a pure function of inputs → outputs + side-effect isolation. Each task writes a manifest file to S3 with checksum, schema, and status (PENDING→IN_PROGRESS→COMPLETED→FAILED) and an idempotency key derived from input IDs and code version.2. Atomic commit: worker writes output to a temp path, computes checksum, then performs atomic rename and writes manifest. The manifest write is conditional (compare-and-set) using DB txn or object-store conditional put to avoid races.3. State management: Airflow stores execution metadata; business-critical states and idempotency keys live in DB. Workers update both S3 manifest and DB state within retries.4. Retry semantics: On retry, task checks manifest and idempotency key: - If COMPLETED with matching checksum → skip (no-op). - If IN_PROGRESS older than timeout → claim using DB conditional update (lease) and resume or restart safe operation.5. Failure handling: - Exponential backoff retries with jitter; max attempts escalate to human alerting. - Partial failures: tasks must be able to resume from checkpoints (checkpoint files in S3) or re-run idempotently.6. Downstream dependency resolution: - Tasks depend on manifests/DB records, not on ephemeral signals. - Orchestrator triggers downstream only when upstream manifest is COMPLETED and checksum validated.7. Concurrency & deduplication: - Use DB unique constraints on idempotency key to prevent duplicate processing across concurrent restarts. - Use leader/lease (Redis or DB row) for singleton tasks.8. Observability & tooling: - Logs, metrics, and traces per task; alerting on repeated failures or stuck IN_PROGRESS beyond threshold. - A “repair” CLI to mark manifests as FAILED/RETRY/IGNORE for manual interventions.

Trade-offs:- Strong consistency (DB transactions/leases) adds complexity but prevents duplication; eventual-consistent object stores require manifest + checksum patterns.- Extra engineering for idempotency pays off for ML workflows (expensive ops like feature generation or model training).

This pattern ensures safe retries, idempotent re-runs, clear state ownership, and prevents duplicate work while remaining scalable and operable in production ML pipelines.

HardTechnical

101 practiced

Describe policies and implementation patterns to avoid logging secrets (API keys, tokens, PII) during exception handling. Include example approaches for sanitizing exception messages, secure logging libraries, and how to audit logs for accidental secret leakage.

Sample Answer

Policy summary:- Treat secrets (API keys, tokens, PII) as high-sensitivity data; default-deny inclusion in logs.- Define classification (public, internal, confidential, secret) and retention rules; require encryption-at-rest and RBAC for log access.- Enforce logging hygiene via CI checks, code reviews, and pre-commit hooks that scan for secret-like patterns.

Implementation patterns:1. Centralized structured logging- Use JSON logs with strict schemas (timestamp, service, level, event_id, context) and disallow free-text stack traces in top-level fields.- Context fields must be whitelisted; any dynamic values go through a sanitizer.

2. Sanitizer + safe-format API- Provide a library sanitize(value, type) that: - Masks tokens (keep prefix 4 chars + ****), redacts full SSNs/email local-part, or replaces with tokenized IDs. - Validates types (email, uuid, uri) and returns safe representation.- Require use of parameterized logging: logger.info("Inference failed for job=%s error=%s", job_id, sanitize(err_msg, "error")) instead of f-strings.

3. Exception handling patterns- Catch exceptions at boundary layers; log only error codes, event_id, and non-sensitive metadata.- Store full exception details in an encrypted secrets vault or secure artifact store accessible only to SRE/security for postmortem, linked by event_id in logs.

Secure libraries/tools:- Use vetted logging frameworks (structlog/python, logback/java) configured to apply sanitizers centrally.- Integrate secret scanners (git-secrets, truffleHog) and runtime log scanners (Datadog sensitive data masking, Splunk PII detection).- Enable log encryption and fine-grained access via cloud logging (Cloud Logging, AWS CloudWatch with KMS + IAM).

Auditing and detection:- Regularly run automated scans over historical logs to detect patterns (API keys, JWTs, credit-card regexes). Use regex + ML anomaly detection to surface unusual tokens.- Implement alerts when a match frequency or entropy indicates a leaked secret; rotate affected credentials immediately.- Maintain an incident playbook: identify leak, rotate secrets, delete or reindex logs (where allowed by compliance), notify stakeholders, and conduct root-cause review.

Example sanitizer (pseudocode):

python

def sanitize(value, kind):
    if kind == "api_key":
        return value[:4] + "****" if len(value)>8 else "****"
    if kind == "email":
        local, _, domain = value.partition("@")
        return local[0] + "***@" + domain
    return "REDACTED"

Operational controls:- Blue/green deployments for logging changes; unit tests enforcing no raw secret patterns; periodic red-team log reviews.- Compliance: document retention and masking for GDPR/PCI, provide data subject access procedures that exclude masked logs.

Why this matters for ML:- ML pipelines handle sensitive training data and credentials (data sources, feature stores). Preventing log leakage avoids data breaches and preserves model integrity (e.g., stolen API keys used to scrape training data). Implementing sanitizer libraries, structured logging, and continuous auditing balances observability with security.

HardTechnical

79 practiced

You observe increasing resident memory during model training until the process OOMs. Outline a systematic debugging checklist to find memory leaks in Python ML code and list concrete fixes (e.g., detach tensors, delete references, `torch.cuda.empty_cache()`, limiting DataLoader prefetch). Mention tools you'd use.

Sample Answer

Systematic checklist — fast path to reproduce, isolate, identify, and fix leaks.

1) Reproduce & isolate- Reproduce with a small deterministic script and fixed seed.- Run on CPU only to see if leak is Python-level vs CUDA-level (set CUDA_VISIBLE_DEVICES="" or move tensors to .cpu()).- Reduce batch size / num_workers to check resource sensitivity.

2) Monitor baseline metrics- GPU: nvidia-smi -l 2 or gpustat; inspect GPU memory growth over iterations.- Process: top/htop, ps aux --sort=-rss to watch resident memory.- Python: use tracemalloc, memory_profiler (@profile) to see allocation hotspots.

Example nvidia-smi watch:

bash

watch -n1 --nvidia-smi

3) Inspect PyTorch GPU allocations- Query runtime stats in loop:

python

import torch
print(torch.cuda.memory_allocated())
print(torch.cuda.memory_reserved())
print(torch.cuda.memory_summary())

- Use torch.cuda.reset_peak_memory_stats() between iterations to track per-iteration peaks.

4) Common leak sources & concrete fixes- Retained computation graph: - Cause: calling loss.backward() multiple times without zero_grad or accumulating tensors that require_grad. - Fix: optimizer.zero_grad()/model.zero_grad(); use with torch.no_grad() or detach() when storing outputs; avoid retain_graph unless needed. - Example: saved = output.detach().cpu()- Storing tensors in lists/dicts/logs: - Cause: appending outputs/metrics directly stores GPU tensors. - Fix: .detach().cpu().numpy() or store scalars only; del references and gc.collect().- Hooks and callbacks keeping refs: - Cause: forward/backward hooks that capture tensors. - Fix: remove hooks, ensure they don't store tensors or use weakrefs.- optimizer.state growth: - Cause: optimizer.state storing transient tensors (e.g., for large per-sample grads). - Fix: check optimizer.state_dict(); use gradient accumulation carefully; clear unused state if dynamically changing param set.- DataLoader worker leaks / pin_memory: - Cause: too many workers, persistent_workers True with buggy dataset, large prefetch_factor. - Fix: reduce num_workers, set pin_memory=False, lower prefetch_factor, set worker_init_fn to reset RNG and clear caches.- retain_graph True accidentally: - Fix: only set retain_graph=True when necessary; most loops should not use it.

5) Garbage collection & cache clearing- Explicit deletes and GC:

python

del tensor, large_list
import gc; gc.collect()
torch.cuda.empty_cache()

- Note: torch.cuda.empty_cache() frees cached memory to OS but does not free tensors still referenced.

6) Profiling tools (use in combination)- PyTorch profiler: torch.profiler (captures CPU/GPU memory and ops).- torch.utils.bottleneck for quick diagnostics.- tracemalloc for Python allocations.- memory_profiler (mprof) for line-by-line memory usage.- objgraph / Pympler / guppy for object retention graphs.- For GPU: Nsight Systems / Nsight Compute for deeper GPU traces.- For containers: docker stats, cgroups logs.

7) Steps to debug iteratively- Add per-iteration prints of torch.cuda.memory_allocated() and Python RSS.- Binary search: comment half of training loop / model layers to localize offending code.- Snapshot object counts: use gc.get_objects() + collections.Counter(type(o)) to detect growing classes.- Use tracemalloc snapshots and compare: filter top frames.

8) Mitigations & long-term fixes- Mixed precision (AMP) to reduce memory footprint.- Gradient checkpointing (torch.utils.checkpoint) to trade compute for memory.- Offload optimizer states or use ZeRO / DeepSpeed for large models.- Use data streaming and smaller batch sizes.- Add unit tests that run a few steps and assert memory usage stable.

9) Example troubleshooting snippet

python

# check allocations per step
for step, batch in enumerate(loader):
    loss = model(batch)
    loss.backward()
    print(step, torch.cuda.memory_allocated())
    optimizer.step(); optimizer.zero_grad()

Summary: start by reproducing and isolating CPU vs GPU; monitor with nvidia-smi, torch.cuda.* APIs, tracemalloc/memory_profiler; look for retained graphs, tensors stored in containers, DataLoader worker issues, hooks, and optimizer state growth; fix by detaching/moving to CPU, deleting refs + gc.collect, using torch.no_grad(), lowering prefetch/num_workers, torch.cuda.empty_cache() for cache, and consider AMP/checkpointing/ZeRO for large models. Use PyTorch profiler, tracemalloc, objgraph, Nsight to pinpoint root cause.

MediumTechnical

72 practiced

Implement a Python decorator `@retry(exceptions=(requests.exceptions.RequestException,), retries=3, backoff_factor=0.5)` that retries a decorated function on the given exceptions using exponential backoff with jitter. The decorated function may be synchronous and returns its result on success or raises the last exception after retries. Provide the function implementation body (no imports required).

Sample Answer

Approach: Create a parameterized decorator factory that accepts exceptions, retries, and backoff_factor; the wrapper will attempt the function, and on catching one of the target exceptions will sleep using exponential backoff multiplied by jitter (randomized factor in [0.5,1.5]) and retry up to retries times. Use functools.wraps to preserve metadata.

python

def retry(exceptions=(Exception,), retries=3, backoff_factor=0.5):
    from functools import wraps
    import time
    import random

    def decorator(func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            attempt = 0
            last_exc = None
            while attempt <= retries:
                try:
                    return func(*args, **kwargs)
                except exceptions as e:
                    last_exc = e
                    if attempt == retries:
                        break
                    # exponential backoff with jitter: base * (2**attempt) * jitter
                    base = backoff_factor * (2 ** attempt)
                    jitter = random.uniform(0.5, 1.5)
                    sleep_for = base * jitter
                    time.sleep(sleep_for)
                    attempt += 1
            # raise the last caught exception preserving stack if possible
            raise last_exc
        return wrapper
    return decorator

Key points:- Exponential backoff: backoff_factor * 2**attempt.- Jitter reduces thundering herd; here uniform jitter in [0.5,1.5].- retries parameter counts retries after the first attempt (attempts = retries+1).- Uses functools.wraps to keep function metadata.

Time/space:- Time: O(T) where T is cost of function calls including sleeps; overhead negligible.- Space: O(1).

Edge cases:- Ensure exceptions tuple contains the right exception types; if empty, decorator never retries.- If function raises non-matching exception, it propagates immediately.- If last_exc is None (shouldn't happen), wrapper will raise a generic RuntimeError — current implementation always sets last_exc when catching.

Alternatives:- Use full jitter (sleep = random.uniform(0, base)) or decorrelated jitter per AWS recommendations.- Add max_backoff cap or logging/metrics hooks to observe retries.

EasyTechnical

99 practiced

Explain Python's try/except/else/finally constructs and best practices for using them in production ML code. In your answer, cover: (1) when to catch broad exceptions (Exception) vs specific exception types, (2) how to preserve the original stack trace, (3) how to ensure resources (files, GPU contexts, DB connections) are cleaned up, and (4) an example scenario: loading a model file and performing a prediction where disk I/O or model-format errors may occur.

Sample Answer

Python's try/except/else/finally let you handle errors, run post-success logic, and guarantee cleanup.

- try: run code that may fail.- except: handle specific exceptions (or broad Exception).- else: runs if no exception occurred — good for code that should only run on success.- finally: always runs, used for guaranteed cleanup.

Best practices (ML production):

1) When to catch broad vs specific:- Prefer specific exceptions (FileNotFoundError, OSError, ValueError, torch.jit.Error) so you don't hide bugs.- Catch broad Exception only at outer boundaries (e.g., top-level request handling) to log/return a safe error; re-raise or fail fast in lower-level modules.

2) Preserve original stack trace:- Use logging.exception(...) to log with stack trace.- Re-raise with "raise" to preserve the original trace, or "raise from exc" to chain: raise MyError(...) from exc

3) Ensure resources are cleaned:- Prefer context managers (with open(...), torch.no_grad(), database connection pools) to auto-clean.- Use finally for resources without context managers (GPU allocations, manual handles).- For async or long-running jobs, ensure timeouts and cancellation handlers clean resources.

4) Example: load model and predict (handles disk I/O and model-format errors)

python

import logging
import torch

def load_and_predict(model_path, input_tensor, device='cpu'):
    try:
        # File read with context manager
        with open(model_path, 'rb') as f:
            state = f.read()  # could raise FileNotFoundError, OSError

        # Load model; torch.load may raise RuntimeError / pickle errors
        model = torch.jit.load(model_path, map_location=device)

    except FileNotFoundError as e:
        logging.exception("Model file not found")
        raise  # preserve stack trace
    except (OSError, RuntimeError, ValueError) as e:
        logging.exception("Failed to load model (I/O or format issue)")
        raise RuntimeError("Model load failed") from e  # chain original
    else:
        try:
            model.to(device)
            model.eval()
            with torch.no_grad():  # context ensures no grad state stays
                return model(input_tensor.to(device))
        finally:
            # explicit cleanup if needed (free GPU memory)
            del model
            torch.cuda.empty_cache()

Key takeaways:- Be specific with exceptions; log with stack traces; use context managers; use finally for last-resort cleanup; use broad catches only at boundaries and always log and (re-)raise appropriately.

Unlock Full Question Bank

Get access to hundreds of Error Handling and Code Quality interview questions and detailed answers.

Join thousands of developers preparing for their dream job.