Monitoring, Logging, and Observability Questions

Direct answer

Default to additive-only changes (new optional fields with sane defaults), never silently repurpose an existing field's name or meaning, and when the meaning genuinely has to change, introduce it as a new versioned field and dual-emit both the old and new during a defined deprecation window so every consumer (dashboards, alerts, downstream jobs) has time to migrate before the old one disappears.

Structured elaboration

Additive changes are the default and the cheap case
Adding a brand-new field with a sensible default (or simply absent, if consumers already tolerate unknown fields) is safe: existing dashboards and alerts that don't reference it are unaffected, and new tooling can start using it immediately. Most schema evolution should fit this case; if it doesn't, that's a signal the change is more than "add a field."

Never repurpose a field in place
Changing what an existing field means (e.g., a latency field that used to be measured in milliseconds and is now measured in microseconds, keeping the same name) is the most dangerous kind of change, because it fails silently: old dashboards keep running the same query and now show numbers that are wrong by a constant factor, with no error to alert anyone. A rename or unit change should always get a new field name (latency_ms retired in favor of latency_us, both emitted for a transition period), never an in-place redefinition.

Version the schema explicitly
Tag every emitted record with a schema_version. Consumers that need to branch on shape (a downstream parser, a strict dashboard query) can check the version rather than guessing from field presence. This also gives you a clean place to document exactly which version introduced which change.

Deprecation as a process, not an event

1. Announce the field's replacement and the planned sunset date.
2. Dual-emit: write both the old and new field for a fixed window.
3. Track actual usage of the old field (query logs, dashboard/alert definitions referencing it) to confirm consumers have migrated, not just assume they have.
4. Only stop emitting the old field once usage has genuinely dropped to zero (or the sunset date passes and remaining consumers have been explicitly notified they'll break).

Testing the transition
Contract tests (automated checks that a producer's output still satisfies what a known consumer expects) and shadow validation (running the old and new emission side by side and diffing the derived metrics they produce) catch the case where the "safe" additive change turns out to interact badly with an existing aggregation, before it reaches production dashboards.

Worked example

A service currently emits {"latency": 245, ...} where latency is milliseconds, and the team wants to switch to microsecond precision.

Wrong approach (in-place redefinition): change the emitter to write {"latency": 245000, ...} under the same field name. A dashboard panel computing avg(latency) over the last hour now silently reports a number 1000x larger with zero errors or warnings; anyone glancing at the dashboard sees "avg latency: 245000ms" and either panics or, worse, doesn't notice because the panel has no sanity bound configured.

Correct approach: add latency_us alongside the existing latency field, dual-emit both for a stated transition window (e.g., until every dashboard query referencing latency has been rewritten to use latency_us, confirmed by grepping the dashboard/alert config repository for the old field name), then drop latency only after that grep returns zero references.

The key diagnostic in this example: the failure mode is not "the pipeline throws an error," it's "the pipeline keeps running and produces a wrong number that looks plausible." That's why additive-with-a-new-name is the default, not an optional extra step.

Trade-offs & pitfalls

• Dual-emitting indefinitely accumulates cost and confusion; every deprecation needs an explicit sunset date, not an open-ended "eventually."
• Tracking actual field usage (rather than assuming consumers migrated because you announced it) is the step most teams skip, and it's exactly the step that prevents a surprise outage when the old field is finally dropped.
• Additive changes still need CI-enforced schema compatibility checks (backward/forward compatibility validation), because "just add a field" can still break a strict consumer that rejects unknown fields.
• A silent semantic change is strictly worse than a loud break: a query that errors gets noticed and fixed; a query that keeps returning a plausible-looking wrong number can go unnoticed for months.

Direct answer

An SLI is the measured signal (e.g. the percentage of requests that were fast and successful), an SLO is the internal target for that signal over a time window (e.g. 99.9% of requests succeed under 300ms over a rolling 30 days), and an SLA is the external, often contractual promise built on top of an SLO, usually with margin, and real consequences (service credits, penalties) if it's breached. In short: the SLI measures, the SLO targets, the SLA promises with a business wrapper attached.

Defining each one concretely, for an e-commerce checkout service

Explained for a non-technical stakeholder: the SLO is the bar the engineering team holds itself to internally so problems get caught and fixed before they become customer-visible; the SLA is the, usually more forgiving, bar the business is willing to be held to externally, on paper, with money attached if it's missed. The gap between the two is deliberate headroom, not sloppiness.

Worked example: computing the error budget

The error budget is the amount of allowed failure baked into the SLO. It's what lets a team ship changes at all instead of freezing forever.

For the 99.9% SLO above:

Over the 30-day measurement window (30 days equals 2,592,000 seconds):

So this service is allowed about 43 minutes of budget-consuming failure (however that failure is defined: downtime, over-300ms responses, and so on) across a 30-day window before the SLO itself is breached. If checkout handles, say, 1,000,000 requests over that same window, the equivalent request-based budget is:

Both framings, time-based and request-based, describe the same budget; which one is more useful depends on whether the SLI is availability-style (was the service up) or ratio-style (what fraction of requests succeeded).

Trade-offs and pitfalls

• Confusing SLO and SLA in conversation causes real problems: teams sometimes design their alerting and release-gating around the SLA (the looser, contractual number) instead of the SLO (the tighter, internal number), which means by the time anyone notices, the team is already close to breaching the external promise with no margin left to react.
• An SLO with no error-budget policy attached is just a number on a dashboard; the value comes from what happens when the budget is nearly exhausted (freeze risky launches, redirect engineering time to reliability work), not from the target itself.
• Picking an SLI that doesn't reflect real user experience (e.g. "server process is running" instead of "requests succeed within an acceptable time") gives a green dashboard while users are still unhappy. The SLI has to be as close to what the user actually experiences as the team can measure.

Direct answer

Observability rests on three complementary signal types: metrics, logs, and traces. Metrics tell you something is wrong and roughly how bad; logs tell you what specifically happened in a given event; traces tell you where in a multi-service request the time or failure occurred. None of the three alone gives a complete picture: a strong incident response usually starts with one pillar to detect and scope the problem, then pivots to another to find root cause.

The three pillars

Instrumentation example, one flow

For a checkout endpoint, an on-call engineer might instrument it like this: a checkout_requests_total counter metric with labels {status, payment_provider}, plus a checkout_latency_seconds histogram with the same labels for percentiles; a structured log line at the point of failure with fields {request_id, trace_id, error_type, payment_provider}; and a trace with spans named checkout.validate, checkout.charge, checkout.persist, each carrying the same trace_id that appears in the log line. The shared trace_id and request_id are what let you jump from "the metric moved" to "here is the specific failing request" to "here is the exact log line explaining why."

Trade-offs and pitfalls

• Treating one pillar as sufficient is the most common mistake: teams that only have logs end up searching blind during an incident because they have no aggregated signal telling them where to look first; teams that only have metrics can detect a problem but can't explain it.
• High-cardinality labels (like an unbounded user_id on a metric) turn cheap metrics into an expensive, slow-to-query mess; that data belongs in logs or traces instead.
• Trace sampling is a real trade-off: full sampling captures every rare failure but is expensive at scale; low sampling rates are cheap but can miss the exact failing request you need. Tail-based sampling (keep traces for slow or error requests) is a common middle ground.
• Retention windows differ by pillar in practice (metrics are cheap to keep for months, verbose logs and full traces are usually much more expensive to retain), which shapes how far back a postmortem can actually look.

Direct answer

Unstructured logging is free-form text intended for a human to read line by line; structured logging emits each entry as a consistent, machine-parsable record (typically JSON) with named fields, so a log-query tool can filter, aggregate, and correlate across millions of lines the way you'd query a database. Log levels (DEBUG, INFO, WARN, ERROR) are an orthogonal concept from structure: they control which entries you even keep or surface, so during an incident an on-call engineer isn't wading through routine noise to find the handful of lines that actually explain what broke.

Structured versus unstructured

When to log at each level, and why it matters during an incident

Worked example: a structured log line for a failed request

{
  "timestamp": "2026-07-17T15:04:05.123Z",
  "level": "ERROR",
  "service": "payments-api",
  "request_id": "req-8f21",
  "trace_id": "trace-a93c",
  "status_code": 502,
  "duration_ms": 247,
  "error_type": "UpstreamTimeout",
  "message": "upstream charge provider timed out"
}

During an incident, an on-call engineer can now do something like find all ERROR entries with error_type: UpstreamTimeout in the last 15 minutes, grouped by service, instead of searching for the word "timeout" across every service's free-text logs and hoping the wording matches. The shared trace_id also lets them jump directly from this log line to the distributed trace for the same request.

Trade-offs and pitfalls

• Logging everything at INFO "just in case" defeats the purpose of levels: if INFO volume is as high as DEBUG would be, the on-call engineer is back to searching through noise. Levels only help if they're used with discipline.
• Structured logging without a shared, enforced schema across services becomes almost as unqueryable as unstructured text, just in JSON clothing; a status_code field that's a string in one service and an integer in another breaks cross-service queries.
• Sensitive fields (user identifiers, tokens, payment details) need to be redacted or hashed at the point of logging, not cleaned up later; once something sensitive is in a log aggregator, deleting it retroactively is unreliable.
• DEBUG-level logging left on in production is a common, avoidable cost problem: log ingestion is usually billed by volume, and DEBUG noise can dominate that cost without adding proportional value.

Design a fixed schema (not a free-for-all), enforce it through a shared logging library rather than convention, control verbosity by sampling low-value log levels on hot paths while always keeping errors, and roll it out by dual-emitting the old and new formats side by side until every dashboard and script that greps the old format has migrated.

Framework

What a log entry should capture. At minimum, every structured log line needs:

• timestamp (ISO 8601, UTC), level, service, env
• correlation_id / trace_id (and span_id if tracing is wired up), so a log line can be tied back to the request it came from
• message (still free text, but now one field among many, not the whole line)
• Context relevant to what's being logged: route, status, duration_ms for a request-handling log; error.type / error.message for a failure

Sensitive fields (user identifiers, emails, anything PII) should never be logged raw. Either omit them, or hash/redact them at the point of emission, enforced by the shared logging library so it isn't up to each call site to remember.

Keep field values bounded, not just field names fixed. A stable schema still has a cardinality trap: a field like route should be a small set of route templates (/orders/{id}), not the raw URL with the real ID interpolated in, and free-text fields like stack traces should be capped in length rather than allowed to grow unbounded. Uncontrolled cardinality is what makes a downstream index expensive, so the schema review has to look at value shape, not just field names.

Why this pays off beyond "logs are searchable now." Once every service emits the same fixed fields, three concrete things get easier that were hard with free-text logs: alert rules can key off a stable field (error.type = "TimeoutError") instead of a regex over a message whose wording changes every release; a post-incident timeline can be reconstructed by filtering and sorting on correlation_id and timestamp across services instead of manually eyeballing each service's raw output in turn; and a legacy service that's risky to touch can be instrumented with this schema before any behavioral refactor, giving you a baseline of real production behavior (error rates, latency, call patterns) to diff the refactor against, turning "did the refactor change behavior" from a guess into something you can actually check.

Verbosity on high-traffic endpoints. Logs and metrics are not interchangeable: a hot endpoint doesn't need a log line per request just to know it's up, that's what a request-count metric is for. Reserve full logging for what metrics can't tell you:

• Always log ERROR and WARN at 100%, since those are rare and high-value.
• Sample INFO-level access logs on high-traffic endpoints (a small, fixed percentage), and keep the sample rate configurable per endpoint rather than global.
• Keep DEBUG out of production by default, gated behind a feature flag or short-lived toggle for active investigations.

Rollout without breaking existing tooling. The riskiest part of this kind of change is not the schema, it's cutting over a service whose logs some existing dashboard or on-call script already greps. A safe sequence:

1. Introduce the shared logging library and have it emit the new structured fields alongside the existing free-text message, so old tooling that parses the raw message keeps working unchanged.
2. Stand up new dashboards/queries against the structured fields in parallel with the old ones, and validate they agree.
3. Once consumers (dashboards, alert rules, on-call runbooks) have migrated to the structured fields, deprecate the old free-text-only path.
4. Roll out service by service, starting with a low-traffic one, not as a single cutover across the whole fleet.

flowchart LR
  A[Service code] --> B[Shared logging library]
  B --> C{Migration phase}
  C -->|during rollout| D["Emit: old free-text message + new structured fields"]
  C -->|after rollout| E[Emit: structured fields only]
  D --> F[Log shipper]
  E --> F
  F --> G["Old dashboards (parse message field)"]
  F --> H["New dashboards (query structured fields)"]

Worked example

Take one high-traffic endpoint doing 5,000 requests/sec. Suppose the team sets a per-endpoint logging budget of 200 INFO-level access-log events/sec for it, to keep total ingestion cost bounded. The required sample rate is:

Meanwhile, if this endpoint's baseline error rate is 0.3%, that's $0.003 \times 5{,}000 = 15$ error events/sec, which stays comfortably inside the budget even at 100% logging, so ERROR/WARN can stay unsampled while INFO gets sampled down to 4%. This is the concrete reasoning a strong candidate walks through: log level and sample rate aren't picked in the abstract, they come out of (a) an ingestion budget you've set and (b) the request/error volume you actually have.

Trade-offs and pitfalls

• Sampling INFO logs means a specific request that had no error and wasn't in the sampled 4% has no log trail. That's an acceptable trade for routine traffic, but pair it with always-log-on-error and, ideally, trace-based sampling that always keeps traces for anything flagged interesting (slow, retried, or touching a specific customer), so the rare interesting case isn't lost to the sample.
• A schema that isn't enforced by a shared library drifts fast: one team adds userId, another adds user_id, and six months later nothing joins cleanly. Put schema validation in the library, not in a style guide.
• Rolling out too fast (cutting the whole fleet to the new format in one release) is the single biggest risk here: any on-call script, saved dashboard, or SIEM rule (a Security Information and Event Management rule: a saved detection query that watches log data for suspicious patterns) that still parses the old free-text line breaks silently until someone notices during an incident, which is the worst possible time to notice. The dual-emit window exists specifically to avoid that.
• Over-redacting can be its own failure mode: hashing or dropping a field that turns out to be needed for debugging (say, an order ID that isn't actually PII) makes incidents harder to resolve. Decide field by field, don't blanket-redact everything that looks user-related.