Observability and Monitoring Architecture Questions

Designing and architecting end to end observability and monitoring systems that scale, remain reliable under load, and do not become single points of failure. Topics include deciding which telemetry to collect and why including metrics logs traces and events, instrumentation strategies, collection models such as push versus pull, high throughput telemetry ingestion and pipeline design, time series storage and compression, aggregation and partitioning strategies, metric cardinality and retention tradeoffs, distributed tracing propagation and sampling strategies, log aggregation and secure storage, selection of storage backends and time series databases, storage tiering and cost optimization, query and dashboard performance considerations, access control and multi tenancy, integration with deployment pipelines and tooling, and design patterns for self healing telemetry pipelines. Senior level assessments include designing scalable ingestion and aggregation architectures, storage tiering and query performance optimization, cost and operational tradeoffs, and organizational impacts of observability data.

EasyTechnical

38 practiced

Define SLI, SLO, SLA and error budget in the context of service reliability. Walk through how you would choose an SLO for an internal HTTP API used by multiple downstream teams. Include concrete metrics you would measure, baseline calculation approach, and how you'd set an initial error budget.

Sample Answer

SLI, SLO, SLA, Error Budget — definitions- SLI (Service Level Indicator): a measurable signal of user‑perceived reliability (e.g., request success rate, p99 latency).- SLO (Service Level Objective): a target on an SLI over a time window (e.g., 99.9% success over 30 days).- SLA (Service Level Agreement): a contractual promise to customers, often with penalties if missed (legal/business).- Error budget: 1 − SLO (the allowed fraction of bad or degraded behaviour in the window). It’s the tolerance you can spend on risk (deploys, experiments).

Choosing an SLO for an internal HTTP API (downstream teams)1. Clarify user impact & critical paths- Ask downstream teams: which endpoints are critical, which latencies cause failures, acceptable degradation windows.

2. Pick concrete SLIs (examples)- Availability/success SLI: ratio of HTTP requests with 2xx/3xx (or business-defined success) to total valid requests.- Latency SLI: p95 and p99 request latency for critical endpoints.- Error-rate SLI: percent of requests returning 5xx or timed out.- Freshness/ correctness SLI (if applicable): percent of responses passing schema/validation or cache staleness.

3. Baseline calculation approach- Gather ≥30 days of telemetry: total requests, status codes, latency histograms, by endpoint and client.- Compute per-endpoint SLIs and aggregate weighted by traffic or by business importance (weight critical endpoints higher).- Example: availability baseline = (sum successful requests) / (sum valid requests) over 30 days. Latency baseline = observed p95/p99 over same window.

4. Set initial SLOs- Use baseline + business tolerance. If baseline availability is 99.995% for core endpoints, consider starting slightly below (e.g., 99.99%) to allow improvement and operational breathing room. For non‑critical endpoints, 99.9% may suffice.- Combine metrics: e.g., SLO = availability ≥ 99.9% AND p95 latency < 200ms.

5. Set initial error budget- Error budget = 1 − SLO. For SLO 99.9% over 30 days → allowed downtime/errors ≈ 0.1% of requests → ~43.2 minutes of total error time in 30 days (or equivalent error volume).- Translate into actionable units: convert percent to minutes of outage or number of failed requests so teams can reason about budget consumption.

6. Operationalize and iterate- Implement monitoring, burn-rate alerts (e.g., when 25%, 50%, 100% of budget consumed), and governance (pause risky releases if burn rate too high).- Review after 1–3 months, adjust SLOs to align with actual user needs and engineering capacity. If SLO too easy/hard, recalibrate based on impact and cost.

Key trade-offs- Higher SLO → lower error budget → slower releases, higher cost. Lower SLO → faster pace but risk customer/internal frustration.- For internal APIs, balance developer productivity and downstream reliability; prioritize endpoints by customer impact and use weighted SLOs where appropriate.

EasyTechnical

59 practiced

Explain head-based sampling, tail-based sampling, and rate-limiting for distributed traces. For each method provide pros and cons and an example scenario where it is most appropriate (e.g., high-throughput services, troubleshooting rare errors). Mention implementation trade-offs such as complexity and backend load.

Sample Answer

Head-based sampling (client-side / front-door):- What: Decide to keep/drop a trace at the service that first receives the request (ingress or SDK).- Pros: Very low backend cost (fewer traces sent), simple to implement, immediate sampling decisions.- Cons: Loses context for downstream errors if a trace was dropped upstream; randomness can miss rare failures.- Best for: Very high-throughput services where reducing ingestion cost is primary (API gateways, public endpoints).- Trade-offs: Simple logic but requires consistent sampling policy rollout; can bias observability if upstream rejects important traces.

Tail-based sampling (collector / backend):- What: Collect full traces temporarily, then decide to keep traces based on observed attributes (errors, latency, particular spans).- Pros: Can retain rare/interesting traces (errors, anomalies), better for troubleshooting; more accurate selection.- Cons: Higher short-term backend load (store/inspect many traces), more complex infra (buffers, policies), higher cost.- Best for: Debugging rare errors, performance investigations where preserving error traces is critical.- Trade-offs: Need buffering/retention window, autoscaling collectors, and careful cost control to avoid overload.

Rate-limiting sampling (quota-based, reservoir):- What: Enforce a fixed max number of sampled traces per time unit (global or per-service), often combined with priority tiers.- Pros: Predictable cost and storage use; simple guarantees for budgets.- Cons: May drop important traces when quota exhausted; requires fair allocation across services to avoid skew.- Best for: Enforcing observability cost caps across many services; steady-state budgeting.- Trade-offs: Requires coordination (per-service quotas, priority tags), possible complexity when combining with tail-based rules (reservoirs for errors vs regular traffic).

Implementation notes for SREs:- Hybrid approaches are common: head-based for baseline, tail-based for error promotion, and rate limits to cap costs.- Monitor sampler effectiveness (error capture rate), backend queue depths, and sampling bias.- Keep sampling configuration manageable (central control plane), and ensure low-latency decision paths to avoid adding overhead.

HardTechnical

27 practiced

Explain how you would maintain trace continuity and correct parent-child relationships across heterogeneous languages and asynchronous boundaries (HTTP, gRPC, message queues). Describe how W3C Trace Context works, how to propagate context through queues, and how to handle retries and fork/merge trace patterns.

Sample Answer

Clarify goal: keep a single logical trace across services written in different languages and across sync (HTTP/gRPC) and async (message queues) boundaries so parent-child relationships remain correct for root cause and latency analysis.

W3C Trace Context (traceparent + tracestate)- traceparent: version-traceid-spanid-flags. traceid ties the whole trace; spanid is the current span; flags include sampled bit.- tracestate: vendor-specific key=value list for extra info / baggage.- All platforms should accept and emit these headers unchanged; libraries map them to native tracing spans.

Propagation across sync boundaries- For HTTP/gRPC: inject traceparent and tracestate into outgoing headers and extract on incoming requests to create a child span with parent spanid = incoming spanid.- Use official SDKs (OpenTelemetry) to normalize differences between languages.

Propagation through queues (async)- On enqueue: when producing a message, capture current span’s context and write traceparent/tracestate into message headers/attributes (e.g., Kafka headers, SQS MessageAttributes).- On dequeue: extract headers and start a new span whose parent is the original spanid — this preserves parent-child across time boundaries.- If the queue strips headers (legacy), include a lightweight correlation payload in the message body (JSON envelope) or store mapping in a consistent store (last resort).

Retries, forks, merges- Retries: treat each retry attempt as a new span with links to the original span (not as a child of the consumer span); annotate with retry count. Prefer link semantics (OpenTelemetry link) to indicate multiple attempts against same logical work while keeping timeline accurate. Ensure idempotency keys are propagated so retries don't double-act.- Forks (fan-out): when one request spawns N parallel work items, set the parent of each new span to the originating span (or use links plus children depending on semantics). Use a “producer” span that creates child spans for each consumer; include a deterministic child-span name including partition/fan-out index.- Merges (fan-in): when multiple upstream traces feed into one downstream operation, create a new span that links to each upstream span (links), because there’s no single parent. Mark metadata indicating multiple contributors.- Long-lived work: if background jobs live longer than the originating trace retention, include a persisted trace context and start a new trace root with a link to the original trace to avoid losing causality.

Operational practices- Enforce libraries and middleware that automatically inject/extract (OpenTelemetry collectors, HTTP/gRPC interceptors, message client wrappers).- Standardize header names and fallbacks; log traceid/spanid in structured logs; correlate traces with logs/metrics.- Monitor sampling: sample decisions should travel with traceparent; if sampling is adaptive, ensure downstream can override only via explicit policies.- Test: simulate cross-language flows, header loss, retries, and fan-in/out; validate parent-child links in a tracing UI.

Trade-offs- Links preserve causality without forcing single-parent model but add complexity in UI/analysis.- Storing context in message body increases payload but tolerates header-stripping systems.- Strict propagation everywhere increases overhead; balance sampling to control costs.

This approach keeps trace continuity and accurate parent-child relationships across heterogeneous, asynchronous systems while providing operational controls for retries, forks, and merges.

EasyTechnical

29 practiced

Coding (Python): Given this sample log line format: 2025-10-05T14:23:11Z INFO user=alice order_id=12345 action=checkout total=99.95 message="checkout completed", write a Python function parse_log(line: str) -> dict that returns a structured JSON-like dict with typed values (timestamp ISO string, level, keys as appropriate). Handle quoted values and numeric conversion. Provide a few example inputs and outputs in your answer.

Sample Answer

Approach: use a regex to capture the leading timestamp and level, then find all key=value pairs where value can be quoted (allowing spaces) or unquoted. Convert numeric-looking values to int/float; keep others as strings; return a dict with keys: timestamp (ISO string), level, plus parsed fields.

python

import re

KV_RE = re.compile(r'(\w+)=(".*?"|\S+)')
HEAD_RE = re.compile(r'^(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z)\s+([A-Z]+)\s*')

def _convert_value(s: str):
    # strip quotes if present
    if s.startswith('"') and s.endswith('"'):
        s = s[1:-1]
    # try int, then float, else keep string
    if re.fullmatch(r'-?\d+', s):
        return int(s)
    if re.fullmatch(r'-?\d+\.\d+', s):
        return float(s)
    if s.lower() in ('true', 'false'):
        return s.lower() == 'true'
    return s

def parse_log(line: str) -> dict:
    m = HEAD_RE.match(line)
    if not m:
        raise ValueError("line does not start with timestamp and level")
    ts, level = m.groups()
    rest = line[m.end():].strip()
    result = {'timestamp': ts, 'level': level}
    for k, v in KV_RE.findall(rest):
        result[k] = _convert_value(v)
    return result

Key points:- HEAD_RE extracts ISO timestamp and level reliably.- KV_RE finds key=value where value can be quoted to include spaces.- Conversion tries int→float→bool→string.

Time/space: O(n) over line length, O(k) extra space for fields.

Examples:- Input: 2025-10-05T14:23:11Z INFO user=alice order_id=12345 action=checkout total=99.95 message="checkout completed" Output: {'timestamp':'2025-10-05T14:23:11Z','level':'INFO','user':'alice','order_id':12345,'action':'checkout','total':99.95,'message':'checkout completed'}

- Input: 2025-11-01T00:00:00Z ERROR service=db error_code=500 retry=false message="connection failed" Output: {'timestamp':'2025-11-01T00:00:00Z','level':'ERROR','service':'db','error_code':500,'retry':False,'message':'connection failed'}

Edge cases:- Missing head -> raises ValueError- Unrecognized formats: adjust regex- Nested quotes or escaped quotes are not handled by this simple parser (would need a more advanced tokenizer).

HardTechnical

35 practiced

Design detectors to identify instrumentation bugs (for example a library accidentally emitting metric with request_id as a label, counters resetting unexpectedly, or a histogram with shifted bucket boundaries). Describe automated checks, alerting thresholds, and automated mitigations to prevent such bugs from causing system-wide cost or correctness issues.

Sample Answer

Approach: treat instrumentation bugs as a specific class of anomalies with deterministic checks (schema/metadata), statistical detectors (distributional shifts), and circuit-breaker mitigations. Combine static validation at build/deploy time with runtime monitoring and automated remediation.

Automated checks (build & deploy):- Schema/label linting: CI step that enforces allowed label names, types, and cardinality limits; fail builds that introduce new dynamic labels (e.g., request_id).- Static config tests: validate histogram bucket boundaries and counter monotonicity expectations in protobuf/Telemetry configs.

Runtime anomaly detectors:1. Cardinality explosion detector - Metric: distinct label values / minute per metric - Alert if growth rate > baseline*X (e.g., 5x) and absolute unique > threshold (e.g., 1k).2. Counter-reset detector - Check monotonic counters: alert if current < previous by > epsilon or if derivative spikes negative. - Use rolling windows to avoid transient restarts.3. Distribution/bucket shift detector - Compare current histogram bucket counts and quantiles vs 24h baseline using KS distance or KL divergence; alert when divergence > threshold.4. Unexpected zeroing or spike detector - Low-pass and high-pass filters (z-score) per metric to catch sudden drops/resets or unexplained spikes.

Alerting thresholds & policies:- Two-tier alerts: Pager for high-severity (system cost/correctness risk) — e.g., cardinality > 10k, counters reset across many hosts, KL divergence > critical; Ops-mailing list for medium severity; silent audit logs for low severity.- Dynamic thresholds: use rolling-percentile baselines (95th) per metric and per service to reduce false positives.- Alert enrichment: include sample offending label values, service, deploy revision, and last config change.

Automated mitigations:- Rate-limit ingestion: automatically throttle or sample offending metric labels at the collector when cardinality or ingestion cost spikes.- Label scrubbers: apply regex-based label drop rules on collector to remove unsafe labels (e.g., UUID-like values).- Rollback/Freeze: if detector correlates anomaly with recent deploys, automatically trigger a canary rollback or freeze further metric changes and notify owner.- Quarantine & reroute: divert high-volume bad metrics to a cheaper long-term storage with sampling while preserving originals for debugging.

Operational practices:- Runbooks per detector with playbook actions and runbook links in alerts.- Regular chaos tests: inject synthetic instrumentation faults in staging to validate detectors and mitigations.- Post-incident: add CI lint exceptions or extend schema formally when new labels are legitimate.

Expected outcomes & metrics:- Reduce cost spikes from bad cardinality by >80%- Detect >90% of counter resets within 5 minutes- Lower false-positive rate via dynamic baselining and CI schema enforcement

This layered approach (CI linting + statistical runtime detection + automated mitigations + human playbooks) minimizes cost and correctness impact while keeping developers in the loop.

Unlock Full Question Bank

Get access to hundreds of Observability and Monitoring Architecture interview questions and detailed answers.

Join thousands of developers preparing for their dream job.