Monitoring, Logging, and Observability Questions
Understanding running systems through their signals. Covers metrics, logs, and traces, instrumentation, dashboards, alerting design, and log analysis and correlation for debugging production. Emphasizes designing observability so problems are detectable and diagnosable before users are affected.
Design an automated system that reacts to error-budget burn: it should be able to roll back a feature flag, shift traffic away from a bad region, or freeze deploys entirely, depending on how fast the budget is burning. What safety checks and manual overrides would you build in, and how would you test it without risking a bad automated rollback in production?
Sample Answer
Direct answer: Build a burn-rate-driven policy engine, not a single "if error budget hits zero, do X" rule: it watches how fast the budget is burning, tiers its responses by severity and blast radius (a safe, automatic action for well-understood, reversible situations; a human-approved action for anything with wide blast radius), and every automated action must be independently reversible and testable in isolation before it's ever allowed to run against real traffic.
Structured elaboration
Two-tier action model
- Tier A (automated, safe): actions that are cheap to reverse and scoped narrowly, rolling back a single feature flag, shifting a small percentage of traffic away from a degraded region. These fire automatically once burn rate crosses a defined threshold.
- Tier B (automated proposal, human approval required): actions with wide blast radius, freezing all deploys org-wide, failing over an entire region. The system proposes the action with its reasoning and evidence, but a human clicks approve.
Safety checks before any automated action
- Verify the rollback target actually exists and was recently healthy (don't roll back to a flag state that was itself broken).
- Check a dependency graph so a feature-flag rollback or traffic shift doesn't break something else that depends on the current state.
- Rate-limit the automation itself: one automated action per service per cooldown window, so a flapping signal can't trigger a rollback loop.
- Every action requires a kill switch: one command that halts all in-flight and future automated actions immediately, checked before every state transition, not just at startup.
Testing without risking a bad automated rollback in production
- Run the exact policy engine against a staging environment with fault injection (inject latency, error rate) and confirm it fires the correct tier at the correct threshold, with no live traffic involved.
- Roll out the automation itself progressively: shadow mode first (it decides what it would do and logs the decision, but doesn't act), then Tier A actions only against a small percentage of traffic or a low-risk service, then expand.
- Chaos-test the automation's failure modes specifically: what happens if the actuator (feature-flag API, load balancer) is unreachable when the policy tries to act? The system should fail toward "alert a human," never toward "retry aggressively" or "silently do nothing."
Worked example (architecture)
flowchart TD
A["Metrics collector<br/>Prometheus / Datadog"] --> B["SLO evaluator<br/>burn-rate rules"]
B --> C{"Burn rate<br/>threshold?"}
C -->|below warning| D["No action, log burn rate"]
C -->|warning| E["Notify on-call + owner"]
C -->|critical, Tier A| F["Automated safe action<br/>flag rollback / traffic shift"]
C -->|critical, Tier B| G["Propose action,<br/>require human approval"]
F --> H["Actuator<br/>feature-flag API / LB / gateway"]
G -->|approved| H
H --> I["Verify: smoke test +<br/>post-action SLI check"]
I -->|recovered| J["Log to audit trail,<br/>close incident"]
I -->|not recovered| K["Escalate to Tier B<br/>/ page human"]
K --> G
This makes the tiering concrete: burn rate below a warning threshold does nothing but log, a warning-level burn notifies without acting, and only a critical burn rate triggers an action, split into Tier A (auto-execute) and Tier B (propose, wait for approval). Every path converges on a verification step, so the system checks its own work rather than assuming the action succeeded.
Trade-offs & pitfalls
- More automation reduces time-to-mitigate but increases blast radius if the policy engine itself has a bug; that's exactly why Tier A is scoped to narrow, reversible actions and Tier B requires a human for anything wider.
- A automation that fires too eagerly (low threshold, short window) risks flapping, an action, then a rollback of that action, then re-triggering. The cooldown and rate-limit above exist specifically to prevent this, and any burn-rate threshold should be validated against real historical burn-rate noise before being trusted to act automatically.
- Testing this system is unusually hard because the dangerous failure mode (a bad automated rollback under real production load) is exactly what you can't safely rehearse in production. Shadow mode plus staged rollout of the automation itself is the practical answer, accept slower confidence-building in exchange for never letting an untested policy touch live traffic.
- CI/CD gating integration (blocking new deploys while budget is critically low, on top of the runtime mitigations above) closes a related but separate gap: the runtime system reacts to what's already live, a deploy gate prevents making things worse while it's still burning.
During a cascading failure, dozens of related alerts can fire within minutes across services and regions. How would you make sure on-call sees a clear signal instead of a flood, without losing the context you need to find the actual root cause?
Sample Answer
Direct answer: Correlate alerts by shared root-cause signals (which service, which dependency, which region) within a short time window, collapse each correlated group down to one representative page that carries aggregated context from every alert in the group, and explicitly bypass suppression for anything that doesn't match an existing group, so a genuinely new, unrelated failure during the same cascade still pages immediately.
Structured elaboration
Correlation, not blind suppression
- Extract root-cause candidate keys from each alert as it arrives, service, dependency, cluster or region, and shared upstream components. Two alerts with matching keys within a short window (a few minutes) are treated as likely symptoms of the same underlying failure.
- A simple, explainable approach: fingerprint by explicit topology (this service calls that dependency, so an alert on the dependency and an alert on the caller are linked). A heavier approach: statistical or ML-based correlation across historical co-occurrence, useful when the failure graph is too large or dynamic to hand-maintain, but harder to explain and harder to validate, worth naming as an option, not a requirement for a correct answer here.
Choosing the representative alert
- Pick by broadest impact (affects the most downstream consumers), or highest severity, or earliest timestamp, whichever ordering the team has agreed correlates best with "closest to root cause" for their systems. This is the single alert that pages on-call.
- Every other alert in the group updates the representative's context (affected entities, timestamps, a running count) rather than generating its own separate page.
Preserving context for root-cause analysis
- The representative alert's payload should carry the aggregated list: which services/regions are affected, a handful of representative failing traces, and the timeline of when each grouped alert first fired, so responders get the full picture from one place instead of hunting across a dozen separate alert threads.
- Keep the raw, ungrouped alert events queryable even though they didn't individually page, they're the audit trail that lets someone later verify the grouping was correct, and undo it if it wasn't.
Not losing new, independent signals
- Anything that doesn't match an existing group's root-cause keys pages immediately and bypasses whatever suppression is active for the ongoing cascade. This is the guardrail against the biggest risk of alert-storm handling: a second, genuinely unrelated incident starting during the noisy window and getting silently absorbed into "yeah, everything's on fire already."
Worked example (architecture)
flowchart TD
A["Alert stream during cascade<br/>dozens of alerts across services"] --> B["Attach root-cause candidate keys<br/>service, dependency, region"]
B --> C["Correlate: group alerts<br/>sharing root-cause keys within window"]
C --> D["Select one representative<br/>alert per group<br/>(highest severity / earliest / broadest impact)"]
D --> E["Attach aggregated context<br/>to representative:<br/>affected entities, sample traces/logs"]
E --> F["Page on representative only"]
C --> G{"New alert matches<br/>no existing group?"}
G -->|yes| H["Treat as independent,<br/>page immediately, bypass suppression"]
G -->|no| C
Alerts sharing root-cause keys within the correlation window collapse into one group with one representative page carrying aggregated context; anything that doesn't match an existing group is treated as independent and pages immediately, which is what keeps a second, unrelated incident from disappearing into the noise of the first.
Trade-offs & pitfalls
- The evaluation question that matters for either correlation approach, heuristic (explicit topology rules) or statistical/ML, is precision and recall on grouping, not just alert-count reduction: precision measures how often a grouped alert genuinely shared the root cause; recall measures how often a truly related alert got correctly grouped instead of paging separately. A heuristic approach is more explainable and easier to validate against known topology, but stops working the moment the failure doesn't follow a mapped dependency; a statistical approach can catch novel correlation patterns but needs a labeled evaluation set and ongoing tuning to trust.
- Grouping on keys that are too broad (just "region=us-east") merges unrelated incidents that happen to share a region, hiding that there are two separate problems; too narrow, and it fails to group the actual cascade, defeating the purpose. Root-cause keys should reflect actual dependency structure, not just convenient labels.
- If the correlation window is too long, alerts from an already-resolved earlier incident can incorrectly merge with a new, unrelated one that starts shortly after; too short, and a slowly-cascading failure gets split into multiple ungrouped pages before the correlation catches up.
- The representative-selection rule needs to be deterministic and explainable, if on-call can't tell why alert X was picked as the representative over alert Y, they'll distrust the grouping the first time it picks the less-informative alert to lead with.
How would you set a naming and labeling convention for metrics across a multi-team organization, so ownership is discoverable and cardinality stays under control?
Sample Answer
Direct answer
A good metric naming and labeling convention is really an ownership and discoverability contract enforced through structure: names encode what's being measured and its unit so anyone can guess the meaning without documentation, and a small, mandatory set of labels makes every metric traceable back to an owning team, while everything else about label design is governed by an explicit cardinality budget rather than left to individual judgment.
The convention
Naming
- Pattern:
<namespace>_<subsystem>_<metric>_<unit>, snake_case, with a standard suffix that encodes type:_totalfor counters,_secondsor_bytesfor a base unit,_bucket/_sum/_countfor histogram components. - Names are nouns describing what's measured, not actions:
http_requests_total, nottrack_http_requests. - Treat a shipped metric name as a stable interface. If the semantics change, version it (
http_request_duration_seconds_v2) rather than silently redefining what an existing name means underneath dashboards and alerts that already depend on it.
Labeling for ownership and discoverability
- Mandatory labels on every metric:
service,team,environment. This is what makes "who owns this metric" answerable by a query, not a wiki page that goes stale. - A small, curated set of dimension labels beyond that (
region,status_code,method) that are explicitly allow-listed, not left open for anyone to add whatever seems useful in the moment. - No identifier-shaped labels, anything meant to be unique per request or per user, ever. That class of label belongs to logs and traces, not metrics.
Scaling the convention across teams without a central bottleneck
- Publish the convention as a small schema or lint rule, a CI check that validates new metric names and labels against the pattern and the mandatory-label list, rather than a document people are expected to remember, so it's enforced automatically at the point where it's cheapest to fix.
- Give teams a self-service allow-list process for adding a new dimension label to an existing metric family, with a lightweight review, so the convention doesn't become a bottleneck people route around.
- Use the
teamlabel to build an automatic ownership directory, which team owns which metrics, surfaced in the metrics catalog, so "who do I ask about this" is a query, not tribal knowledge.
Worked example
A recording rule that only works because the convention was followed consistently, and what breaks otherwise:
groups:
- name: service_latency
rules:
- record: service:request_latency_seconds:p95
expr: |
histogram_quantile(0.95,
sum(rate(http_request_duration_seconds_bucket[5m])) by (service, team, le)
)
This rule only produces a sane per-team latency rollup because every team's http_request_duration_seconds histogram uses the same metric name, the same bucket boundaries, and includes the mandatory service and team labels. If one team named theirs http_req_duration_ms instead, a different name and a different unit with no convention applied, this single rule can't include their data at all. They'd need a one-off query, which is exactly the discoverability cost the convention exists to prevent.
Trade-offs and pitfalls
- A convention that's too strict, a rigid required schema for every possible dimension, slows teams down and invites workarounds, like encoding extra information into the metric name itself to route around a label restriction, which is worse than the problem it was meant to solve. Keep the mandatory set small and make the allow-list process fast.
- A convention that's purely documented and not enforced by CI or lint decays within a quarter as new services get built by people who never read the doc. Enforcement has to be mechanical, not aspirational.
- Common wrong turn: retrofitting a naming convention onto an existing large fleet all at once. A big-bang rename breaks every dashboard and alert built on the old names simultaneously. Version and dual-ship, old and new names in parallel for a deprecation window, instead.
Have you managed your dashboards, alert rules, and metric definitions as code, checked into version control the same way as application code? What does that actually buy you, and what's harder about it compared to clicking around in a UI?
Sample Answer
Direct answer
Yes, and the honest answer to what it buys you is the same thing version control buys application code: review before it ships, a diff when something changes, and the ability to revert a bad alert threshold as fast as you'd revert a bad deploy. What's harder is that a GUI lets you make a quick, well-intentioned tweak in seconds, and observability-as-code trades that speed for review overhead, which is worth it for anything that pages someone, but can feel like friction for a genuinely trivial change.
Structured elaboration
What managing dashboards and alerts as code actually buys you
| Property | UI-managed | Managed as code |
|---|---|---|
| Reviewability | No diff, no approval step by default | PR review before a change to a paging threshold ships |
| Drift detection | Silent: a manual click can diverge from what anyone documented | Git is the source of truth; drift is detectable by diffing against deployed state |
| Rollback | Manual, relies on someone remembering the prior value | Revert the commit, same as reverting application code |
| Onboarding consistency | Every new service gets configured slightly differently by whoever set it up | A shared template/module ensures every service starts from the same baseline |
| CI validation | None | Linting (for example promtool for Prometheus rules) can catch a syntactically broken alert rule before it ever reaches production |
What's harder about it
The as-code tooling has its own learning curve (a YAML or JSONNET dialect on top of whatever the underlying platform's own query language already is), and some dashboard-heavy workflows, like eyeballing a panel while dragging its boundaries to get the layout right, are genuinely faster in a GUI. Not every vendor supports full-fidelity export/import either, so "everything as code" sometimes means committing to a subset of features the export format actually preserves.
Worked example
A concrete before/after: an engineer edits an alert rule's PromQL expression and, in the process, introduces an unbalanced parenthesis. In a GUI-only workflow, that error might not surface until the rule silently fails to evaluate in production, no clear symptom, just an alert that never fires again. In an as-code workflow, the same typo is caught by a CI lint step (promtool check rules) that parses the file and fails the pipeline before the change merges, so the broken rule never reaches production at all. The value isn't the specific tool, it's that a syntax error becomes a blocked PR instead of a silently dead alert discovered weeks later during an incident it should have caught.
Trade-offs and pitfalls
A passing lint check only proves the syntax is valid, not that the threshold or query logic is actually correct, so treat CI validation as catching one class of bug (the code doesn't parse) and pair it with an actual review of the logic, not a substitute for that review. Shared dashboard-as-code modules can also create merge conflicts when multiple teams edit the same file, which is a real cost that a GUI never has, and worth designing around (per-team files, clear ownership boundaries) rather than discovering the hard way. Finally, don't over-apply the discipline: a genuinely low-stakes internal debugging dashboard that nobody pages off of doesn't need the same review overhead as a paging alert rule, and treating every change identically just trains people to route around the process.
Walk me through instrumenting a request handler for an API endpoint. At each stage, say authentication, the business logic, and the database call, what would you measure, and which metric type would you use for it?
Sample Answer
Direct answer: Use a counter for anything that only goes up (total requests, total errors), a histogram for anything you need a distribution or percentile from (latency, per-stage duration), and a gauge for anything that moves up and down (in-flight request count). At each stage of the handler, auth, business logic, and the database call, you instrument the stage's duration as a histogram observation and any stage-specific failures as a counter increment, and you decide the request's overall status_code from what actually happened, not from having merely reached the end of the handler body.
Structured elaboration: metric per stage
| Stage | What to measure | Metric type | Why |
|---|---|---|---|
| Authentication | duration, and a counter of auth failures by reason (expired token, bad signature) | Histogram + Counter | Duration needs percentiles; failure reasons need discrete counts |
| Business logic | duration, counter of business-rule errors | Histogram + Counter | Same reasoning, scoped to this stage |
| Database call | duration, counter of DB errors, gauge of open connections | Histogram + Counter + Gauge | Connection pool exhaustion is a live, bidirectional signal, not a count |
| Whole request | total requests, total errors, overall latency, in-flight count | Counter + Counter + Histogram + Gauge | The user-facing SLI is built from these, not the per-stage numbers |
Label the metrics with route, method, and status_code (or error_type for the error counter), and keep the label set the same at every stage so slicing "auth failures for this route" is a straightforward query.
Worked example: instrumenting the handler (Python, Flask + prometheus_client)
from time import perf_counter
from flask import Flask, request, jsonify
from prometheus_client import Counter, Histogram, Gauge
app = Flask(__name__)
REQUESTS_TOTAL = Counter(
"http_requests_total", "Total requests", ["route", "method", "status_code"]
)
STAGE_LATENCY = Histogram(
"http_stage_latency_seconds", "Per-stage latency", ["route", "stage"]
)
STAGE_ERRORS = Counter(
"http_stage_errors_total", "Per-stage errors", ["route", "stage", "error_type"]
)
IN_FLIGHT = Gauge("http_in_flight_requests", "Requests currently being handled", ["route"])
# Stand-ins for a real auth check and a real database, so the example runs end to end.
FAKE_DB = {1: {"order_id": 1, "total_cents": 4200}}
def authenticate(req):
if not req.headers.get("Authorization"):
raise PermissionError("missing bearer token")
def run_business_logic(req, order_id):
return {"order_id": order_id}
def query_database(order_id):
return FAKE_DB.get(order_id)
@app.route("/orders/<int:order_id>", methods=["GET"])
def get_order(order_id):
route = "/orders/:id"
IN_FLIGHT.labels(route=route).inc()
status_code = "500"
try:
t0 = perf_counter()
try:
authenticate(request)
except Exception as e:
STAGE_ERRORS.labels(route=route, stage="auth", error_type=type(e).__name__).inc()
raise
finally:
STAGE_LATENCY.labels(route=route, stage="auth").observe(perf_counter() - t0)
t1 = perf_counter()
try:
result = run_business_logic(request, order_id)
except Exception as e:
STAGE_ERRORS.labels(route=route, stage="business_logic", error_type=type(e).__name__).inc()
raise
finally:
STAGE_LATENCY.labels(route=route, stage="business_logic").observe(perf_counter() - t1)
t2 = perf_counter()
try:
row = query_database(result["order_id"])
except Exception as e:
STAGE_ERRORS.labels(route=route, stage="db", error_type=type(e).__name__).inc()
raise
finally:
STAGE_LATENCY.labels(route=route, stage="db").observe(perf_counter() - t2)
if row is None:
status_code = "404"
return jsonify({"error": "not found"}), 404
status_code = "200"
return jsonify(row), 200
except Exception:
status_code = "500"
return jsonify({"error": "internal"}), 500
finally:
REQUESTS_TOTAL.labels(route=route, method="GET", status_code=status_code).inc()
IN_FLIGHT.labels(route=route).dec()
Verified: a GET with a valid auth header and an existing order returns 200 and increments status_code="200"; a GET with no auth header raises inside the auth stage, is caught by the outer handler, and correctly returns 500 with status_code="500"; a GET for an order ID that doesn't exist returns 404 with status_code="404", not a false 200.
Key points
- Every stage gets its own histogram observation, so a slow request can be attributed to auth vs business logic vs DB without touching a debugger.
- The stage error counter carries
error_type, not a raw exception message: exception messages are effectively unbounded strings and would blow up cardinality if used as a label. IN_FLIGHTis incremented before the try block and decremented infinally, so it stays correct even when the handler raises.status_codeis set once, right next to eachreturn, andREQUESTS_TOTALis only incremented in the outerfinally, after the response has actually been decided, not the moment the handler body finishes running. This matters because an earlier, naive version of this pattern (incrementingstatus_code="200"right beforereturn row, 200, with no check onrow) would silently record a 200 for a request that actually 404s or 500s downstream, if the response object itself turns out to be invalid; tying the label to the real outcome, evaluated infinally, is what keeps the metric honest.- The route label uses the path template (
/orders/:id), not the interpolated path, deliberately, so cardinality stays bounded by the number of distinct routes, not the number of distinct order IDs.
Complexity
- Each stage adds O(1) time overhead per request (a histogram observation and a counter increment are constant-time operations), so instrumentation cost scales linearly with the number of stages instrumented, not with request payload size.
- Space cost is proportional to the number of distinct label combinations actually observed (cardinality), which is why the route-template and error-type choices above matter more than the number of stages.
Edge cases
- A record that legitimately doesn't exist (a GET for an order ID nobody has) is a normal, expected outcome, not an exception: the handler checks for it explicitly and returns 404 with
status_code="404", rather than letting aNoneflow into a raw response and being miscounted as success. - An exception raised inside
finally(e.g., the DB stage's cleanup itself failing) can mask the original exception. Instrumentation code should never be able to throw; wrap the metric calls defensively or keep them to simple counter/histogram calls that can't fail. - Requests that never reach a stage (e.g., auth fails, so business logic and DB are never called) must not emit a stage-latency observation for stages they never entered; the code above achieves this because each stage's
observecall is scoped inside its own try/finally, not run unconditionally at the end. - Concurrent requests to the same route must not clobber the gauge, this is a library correctness question, not application code, and Prometheus client gauges are safe for concurrent inc/dec across threads.
Trade-offs & pitfalls
- If you also expose this app's own
/metricsendpoint on the public listener, it becomes an unauthenticated read of internal state (route names, error rates). Serve it on a separate internal port or behind network policy, and scrape it as a sidecar in Kubernetes so the metrics port never leaves the pod network. - Instrumenting every stage is more code to maintain; a smaller service might reasonably start with just the overall request histogram and one DB-call histogram, and add finer stages only once a real incident showed the coarse view wasn't enough.
- The subtlest bug in this whole pattern is deciding the status-code label from "the handler body finished without raising" instead of from the actual response. Anywhere a handler can return a response that doesn't map to a clean success (a
Noneresult, a partially-built object, a downstream call that returns an error payload with a 200 HTTP status), that gap has to be checked explicitly, or the metric silently lies about what the client actually received.
Unlock Full Question Bank
Get access to all Monitoring, Logging, and Observability interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.