Error Handling and Logging Questions

Covers systematic error management and observability practices for applications and services. Includes structured logging, error categorization and severity, returning appropriate error responses and HTTP status codes, try catch usage, graceful degradation and fallback strategies, retry logic and exponential backoff, circuit breaker patterns for external dependency failures, monitoring and alerting, and using logs and traces for debugging production issues. Emphasizes designing systems that surface actionable diagnostics while protecting user experience and system stability.

EasyTechnical

28 practiced

Describe the typical logging levels (debug, info, warn, error, critical/fatal). For each level, provide examples of events that should be logged at that level in a production microservice, and explain how an SRE team should map these levels to alerting and on-call priorities.

MediumTechnical

21 practiced

Write a pytest unit test (high-level pseudocode is fine) asserting that a web endpoint returns HTTP 422 for validation errors and that an error log line includes the tag "validation_failure" and the request_id. Describe how you'd structure the test setup and assertions.

Sample Answer

Approach: use pytest with a test client (Flask/FastAPI/starlette) and capture logs (caplog fixture or a logger mock). Send an invalid request that triggers validation, include a unique request_id header, then assert response status is 422 and that a log record contains both "validation_failure" and the same request_id.

python

def test_endpoint_returns_422_and_logs_validation_failure(client, caplog):
    # Arrange
    invalid_payload = {"bad": "data"}  # causes validation error
    request_id = "test-req-1234"
    headers = {"X-Request-ID": request_id, "Content-Type": "application/json"}

    # capture application logs at INFO/ERROR level
    caplog.set_level("ERROR", logger="myapp")  # or root logger name used by app

    # Act
    response = client.post("/items", json=invalid_payload, headers=headers)

    # Assert - HTTP status
    assert response.status_code == 422

    # Assert - log contains validation tag and request_id
    matching = [
        rec for rec in caplog.records
        if "validation_failure" in rec.getMessage() and request_id in rec.getMessage()
    ]
    assert len(matching) >= 1, "Expected at least one log with validation_failure and request_id"

    # Optional: check structured log fields if using JSON logs
    # for rec in caplog.records:
    #     data = json.loads(rec.getMessage())
    #     assert data.get("tag") == "validation_failure"
    #     assert data.get("request_id") == request_id

Key notes:- Use caplog for simple capture; for structured logs mock the logger to inspect emitted dict.- Ensure test injects request_id header or context so logs include it.- Edge cases: middleware might log at different level or use different logger name — adjust caplog.set_level and logger name.

EasyTechnical

19 practiced

What is a stack trace and when should an SRE team include or avoid including stack traces in logs or in API responses? Discuss privacy/security risks and how to sanitize or redact stack traces while maintaining usefulness for debugging.

HardSystem Design

22 practiced

Design a centralized logging system that can consistently ingest 1,000,000 structured log events per second, provide real-time alerting, and support hot retention of 30 days with searchable indices. Address ingestion architecture, buffering, sharding, indexing strategy, hot vs cold storage, cost-control techniques, and security/isolation between tenants.

Sample Answer

Requirements & constraints:- Ingest 1,000,000 structured events/sec (assume avg event 1 KB → ~1 GB/s, adjust if smaller)- Real-time alerting (sub-second to few-second)- Hot retention 30 days with searchable indices- Multi-tenant isolation, cost control, durability

High-level architecture:Ingesters (stateless collectors) → Distributed buffering (Kafka) → Stream processing (Flink/Kafka Streams) for alerts + enrichment → Indexers (OpenSearch/ES) for hot store → Cold storage (S3 + query layer) → Query/Alert API, UI

1) Ingestion & buffering- Use lightweight collectors (Fluentd/Vector) producing to a Kafka cluster sized for throughput (1 GB/s write, replication factor 3 → ~3 GB/s). Kafka provides backpressure, durability, and retention for replay.- Partitioning key: tenant_id + source to ensure per-tenant ordering and sharding.- Provision Kafka with NVMe SSDs, ample replicas, and ISR tuning; use topic partition count = enough to parallelize consumers (e.g., 2000 partitions → consumers scale horizontally).

2) Sharding & indexing strategy- Time-based indices per tenant or tenant-group: indexname = tenant_YYYY-MM-DD (or hourly for high-volume tenants).- Use rollover API: target index size ~30-50 GB. Primary shard sizing: aim for 20-50 GB per primary shard. For 30-day hot retention and 1 GB/s: ~2.6 TB/day → store primarily on an index cluster sized for SSDs.- Bulk index with batching (e.g., 5-15 MB batches) and tune refresh_interval (e.g., 30s) and translog settings to trade latency vs throughput.- Map strict schemas, ingest pipeline for typed fields to reduce mapping explosion and improve search perf.- Use replica count for read SLA; automate shard allocation and shard rebalancing policies.

3) Real-time alerting- Stream processor subscribes to Kafka, runs rules (stateless and windowed) and emits alerts to alerting engine (PagerDuty, webhook).- For rule evaluation against indexed historical data, use fast aggregated metrics stored in a TSDB (Prometheus/ClickHouse) updated by the stream job to avoid heavy ES queries.- Prioritize low-latency paths: basic alerting from stream processing; complex anomaly detection can run asynchronously.

4) Hot vs cold storage- Hot: OpenSearch cluster on SSDs for 30-day retention and search; enable ILM (index lifecycle): hot → warm → cold.- Cold: After 30 days (or as policy), snapshot indices to S3, use searchable snapshots or a separate analytical store (Parquet in S3 + Presto/Trino) for infrequent queries.- Optionally keep last 7-30 days on hot depending on SLA; move older to S3 to save cost.

5) Cost-control techniques- Tiered retention per tenant (SLA-based): e.g., free tier 7 days, paid 30/90 days.- Sampling and ingestion filters: early drop of debug-level or low-value logs; apply adaptive sampling per tenant.- Downsampling/rollups: aggregate high-cardinality fields into hourly metrics for long-term retention.- Index lifecycle management: shrink/forcemerge warm indices, use searchable snapshots for cold.- Rightsize shard count; use warm/cold nodes to replace expensive SSDs with cheaper disks for older indices.- Autoscaling policies for indexers and Kafka based on metrics.

6) Security & tenant isolation- Logical separation: tenant-specific indices/namespaces; index templates enforce mappings.- Access control: RBAC at API layer and OpenSearch fine-grained roles; OAuth/OpenID for users.- Network: ingesters in customer VPCs or over TLS; use private peering or VPN, mTLS to collectors.- Data protection: TLS in transit, SSE-KMS for S3, encryption at rest on clusters.- Resource isolation: throttle per-tenant ingestion (producer quotas in Kafka), circuit-breakers in indexers, per-tenant rate-limits to prevent noisy neighbor.- Audit logging and per-tenant billing metering.

Operational considerations & trade-offs- Capacity planning: target headroom (50–100%) for traffic spikes; use synthetic load tests.- Failure modes: Kafka retention prevents data loss; indexer nodes in multiple AZs, use rolling restarts and snapshotting.- Trade-offs: full 30-day hot retention on SSD is expensive—use tiering and tiers per customer SLA. Aggressive refresh_interval and larger bulk sizes increase throughput but add latency for search visibility.

Key numbers (example):- Kafka: ~3 GB/s write capacity (replication=3)- Indexer cluster: NVMe nodes sized for ~2.6 TB/day hot → plan ~80–120 TB usable (30 days) accounting for replicas and indices → choose node counts accordingly- Bulk size: 5–15 MB, refresh_interval: 30s–60s, primary shard target: 30–50 GB

SLOs & Observability- Define SLOs: ingest latency, alert latency, search latency, data durability.- Monitor lag (Kafka consumer lag), index throughput, node CPU/I/O, tail latencies.- Automate alerting on resource exhaustion and noisy tenants; implement runbooks and capacity autoscaling.

This design provides a resilient, scalable pipeline for 1M events/sec with real-time alerting, tenant isolation, and cost controls via tiering, sampling, and ILM.

HardTechnical

23 practiced

Design a distributed retry architecture for a chain of microservices where clients and intermediate services may retry requests. Ensure deduplication and idempotency of side effects using idempotency keys, outbox patterns, and request fingerprints. Discuss storage, key expiry, race conditions, and monitoring.

Sample Answer

Requirements & constraints:- Clients and intermediate services may retry; side effects must be deduplicated.- Low latency for common path, durable correctness for eventual delivery.- Handle concurrent retries, out-of-order delivery, and partial failures.

High-level design:- Each incoming request carries an Idempotency-Key (client-supplied or generated at edge) + a Request Fingerprint (hash of canonicalized method+path+body+relevant headers).- Per-service local idempotency store + transactional Outbox pattern. Services persist state and outbox atomically (same DB tx) then emit messages to broker (Kafka/SQS) for downstream calls.- Downstream services dedupe by Idempotency-Key + Request Fingerprint and maintain their own idempotency records.

Components & responsibilities:1. Edge / API Gateway- Ensure Idempotency-Key present (generate if absent) and attach trace id.- Optionally normalize and compute Request Fingerprint.

2. Service local storage- Primary durable store (RDBMS like Postgres with strong consistency or Cassandra for scale).- Idempotency table: key, fingerprint, status {IN_PROGRESS, COMMITTED, FAILED}, result payload (optional), created_at, last_seen.- Index on (key, fingerprint) + unique constraint to prevent duplicates.

3. Outbox- Outbox table in same DB transaction as updating business state and idempotency record. Worker publishes outbox messages to message broker and marks them SENT.

4. Cache/fast-path- Use a fast cache (Redis) to hold recent idempotency outcomes for low latency reads. Redis is a cache only; canonical source is DB.

Request flow:- On request: check cache for (key,fingerprint). If COMMITTED -> return cached response. If IN_PROGRESS -> either wait (short polling) or return 202 with retry-after. If absent -> attempt "insert IN_PROGRESS" via conditional insert/unique-constraint.- Perform business logic and write resulting state + outbox row in same DB tx.- On commit, update idempotency record -> COMMITTED and store response (or idempotent marker).- Publish outbox; once confirmed, optionally trim payload or mark final.

Concurrency & race conditions:- Use DB-level uniqueness (unique constraint on idempotency key + fingerprint) to serialise starters. Conditional insert (INSERT ... ON CONFLICT DO NOTHING RETURNING ...) tells winner vs losers.- For update races, use optimistic concurrency (version column) or atomic compare-and-set for Redis cache writes.- IN_PROGRESS semantics: include deadline/owner metadata so stalled IN_PROGRESS can be reclaimed after timeout by another worker. Reclaim only if safe: ensure previous worker crashed (use lease/heartbeat).- Avoid lost updates by ensuring business state changes and outbox write are atomic in same DB transaction.

Storage & expiry:- Store full records in durable DB; keep response payload up to a reasonable size (e.g., 64KB). For large responses, store pointer to blob store (S3) and a hash.- Apply TTL retention policy: short TTL for cache (minutes-hours), longer TTL for durable idempotency (days-weeks) depending on business needs and legal/regulatory constraints.- Background compaction job: remove COMMITTED entries older than retention, purge orphaned outbox rows.

Idempotency key hygiene:- Encourage clients to generate stable keys per user-intent (e.g., payment_id). Validate keys length/entropy and bind keys to authenticated principal to prevent cross-user reuse.

Monitoring, metrics, and SLOs:- Metrics: idempotency hits/misses, IN_PROGRESS count, reclaimed entries, duplicate requests rejected, outbox publish success/fail rate, time-to-commit, cache hit ratio.- Tracing: propagate trace/idempotency key across services for end-to-end correlation; capture spans for dedupe checks and outbox publish.- Alerts: spike in IN_PROGRESS, high reclaim rate, rising duplicate commits, outbox lag, broker publish failure, idempotency DB saturation.- Dashboards: per-service dedupe rate, average latency on first vs retry requests, storage growth.

Operational concerns:- Testing: chaos tests simulating retries, partial failures, duplicate deliveries.- Backwards compatibility: accept missing keys with gateway-generated keys; treat unknown fingerprint as unique.- Security: validate keys against auth; avoid leaking response payloads via cached idempotency results between tenants.- Cost: tune TTL to balance storage and user-experience (longer retention reduces accidental duplicates but raises storage).

Trade-offs:- Fast Redis-only dedupe is low-latency but may lose data on failover — use as cache with DB as ground truth.- Strong DB uniqueness provides correctness but can be a write-hotspot at extreme scale; partition keys (sharding), use consistent hashing, or use Cassandra with lightweight-transaction if needed.

This design ensures deduplication at each service boundary, durable side-effect ordering via transactional outbox, safe concurrent retries using DB uniqueness + IN_PROGRESS semantics, bounded storage with TTL/compaction, and operational visibility via tracing and targeted metrics.

Unlock Full Question Bank

Get access to hundreds of Error Handling and Logging interview questions and detailed answers.

Join thousands of developers preparing for their dream job.