Monitoring, Logging, and Observability Questions

Direct answer

Alert fatigue is what happens when on-call engineers get so many low-value, noisy, or duplicate pages that they start treating all alerts as probably-not-real, including the ones that matter. It's a trust problem as much as a technical one: once someone has been paged repeatedly in a night for something that turned out to be nothing, the next page, which might be the real incident, gets a slower, more skeptical response.

How I'd prevent it on a team I'm leading

• Deduplication and grouping: alerts that share a root cause (same service, same error type) should collapse into a single incident with a count, not fire a separate page per occurrence. This is usually a config change in the alerting tool (fingerprinting by service and error signature) rather than a code change.
• Severity tuning tied to required response time: not every alert deserves a page. A three-tier split (page now, notify during business hours, dashboard-only) forces every new alert to justify why it needs to interrupt someone's sleep.
• Actionable-by-default policy: no new paging alert ships without a linked runbook and a clear "what to check first." An alert with no next step is a dashboard panel that accidentally has a pager attached.
• Automated remediation for known, safe, repeatable fixes: if the same alert reliably resolves by restarting a stuck worker or clearing a queue, and that action is safe and idempotent, automate it and only page if the automated fix fails.
• A regular noise review: periodically look at which alerts fired most often and whether they led to real action; alerts that never lead to action get tuned or removed, not left running indefinitely out of habit.

Worked example

Suppose a team's on-call rotation is getting paged for "queue depth over 100" on a background job processor, firing several times a week, always self-resolving within a few minutes without anyone doing anything. Applying the framework above: first, check whether these spikes line up with a predictable traffic pattern (a nightly batch job, say) and if so, either raise the threshold above that expected peak or add a time-of-day exception. Second, if the queue really can back up unpredictably but always self-resolves within a known window without intervention, the alert should require a longer sustain window (e.g. "queue depth over 100 for 15 minutes") so it only fires when it isn't going to resolve on its own. Third, if manual intervention when it does page is always the same action (scale up worker count), that's a strong automated-remediation candidate: auto-scale on the same threshold, and only page if depth is still elevated after the auto-scale has had time to take effect.

Trade-offs and pitfalls

• Automated remediation without an audit trail or human confirmation for higher-severity cases can turn a noisy-alert problem into a silent-failure problem: the system "fixes" itself repeatedly while masking a root cause that's getting worse.
• Tuning thresholds purely to reduce page volume, without checking against real past incidents, risks quietly increasing false negatives; the goal is signal-to-noise, not just fewer pages.
• Alert fatigue prevention is not a one-time project. It needs an ongoing review cadence, because new alerts get added faster than old noisy ones get cleaned up if nobody owns the process.

Direct answer: Use a static threshold when the acceptable value is genuinely fixed and well understood (a hard SLO limit, a resource ceiling). Use a baseline or trend-based threshold when the metric has predictable seasonality (daily or weekly traffic patterns) and you care about deviation from "normal for this time," not deviation from a fixed number. Reach for anomaly detection only when the failure signature is multi-metric or too irregular for either simpler approach to capture, and treat it as the heaviest, most maintenance-intensive option of the three, not the default.

Structured elaboration

Worked example: the same metric (CPU utilization across a fleet), three ways

• Static: "alert if CPU > 85% for 5 minutes." Works well for catching a clear resource ceiling being breached, but a fleet that always runs at 80% during Monday-morning batch jobs will either false-positive weekly or force the threshold so high it misses a real starvation event on a quieter day.
• Baseline/trend: "alert if CPU is more than 25% above the same hour-of-week's rolling average." This correctly treats Monday's expected batch-job spike as normal and doesn't fire, but a fleet-wide CPU starvation caused by a bad deploy that happens to coincide with the batch window can get absorbed into "well, Mondays are always high" and go unflagged.
• Anomaly detection: a model watching CPU jointly with request queue depth and GC pause time can catch a starvation event that presents as a subtle joint shift across all three metrics well before any single one crosses an obvious line, at the cost of being the hardest of the three to explain to an on-call engineer at 3am ("the model flagged it" is a much weaker debugging starting point than "CPU crossed 85%").

Trade-offs & pitfalls

• Picking anomaly detection by default because it sounds more sophisticated is a common mistake: it's the most expensive to build, tune, and explain, and a static or baseline threshold solves the large majority of real alerting needs with far less operational overhead.
• A static threshold set once and never revisited becomes wrong as the system's normal load shifts (more users, more traffic), it needs periodic review, not a fire-and-forget config.
• A baseline model needs an explicit exception mechanism for known, one-time events (a planned traffic spike, a migration), otherwise every legitimate change to "normal" trains the model to treat pathological behavior as expected, which is exactly how a baseline silently drifts into missing real regressions.
• Heavier ML-based detectors, seasonal decomposition, isolation forests, autoencoders, are worth knowing exist as options for genuinely hard multi-metric cases, but they trade explainability and maintenance burden for sensitivity: every one of them needs periodic retraining and a human who can debug why it fired (or didn't) when it matters. Most interview-relevant alerting design doesn't need to go there, and naming that trade-off explicitly is usually more valuable than trying to design the algorithm itself.

Direct answer: Once the error budget is gone, the team's default posture flips: feature releases to that service pause, and the team's priority shifts to reliability work until the budget recovers. This isn't a vague cultural norm, it's a policy with a named decision-maker (typically the service owner or on-call lead, with an SRE or eng-manager escalation path if the freeze is contested) and a concrete exit condition, not just "be careful."

Structured elaboration

Who decides

• The service owner (or the on-call lead in the moment) declares the freeze the moment the budget hits zero, this should be close to automatic given a burn-rate alert, not a discussion. What's actually debated is whether to grant an exception for a specific change.
• Exceptions (an urgent security patch, a fix for the very issue that burned the budget) go through a lightweight approval, usually the service owner plus one more senior engineer or the EM, logged so there's a record of why the freeze was breached.

What changes about how you ship

• Non-essential feature work to the affected service stops. Bug fixes and reliability work continue and are, in fact, prioritized.
• Anything that does ship goes through a tighter gate: smaller blast radius (canary a smaller percentage of traffic, or a longer soak time before full rollout), and a mandatory rollback plan reviewed before merge, not written after something breaks.
• On-call attention increases, more eyes on dashboards, lower threshold for calling something an incident rather than watching it.

Getting back to a healthy budget

• Root-cause the consumption first: was it one bad deploy, a sustained degraded state, or a dependency's fault? The remediation differs (revert a change vs fix a dependency vs re-architect a retry storm).
• Budget doesn't get "restored," it recovers naturally as the rolling measurement window slides past the bad period, assuming no further burn. This is why the team's job during the freeze is to stop burning more, not to find a way to reset the counter.
• Track burn rate daily during the freeze (not just the raw percentage remaining), a service that's still actively burning needs different action than one that stabilized right after the incident and is just waiting for the window to roll forward.
• Exit condition: the freeze lifts when the budget crosses back above whatever threshold triggered it (commonly the point where slow-burn alerting would re-arm, e.g., back above 50% remaining) or after a fixed re-review date, whichever the team's policy specifies, so "when it feels better" is never the actual criterion.

Worked example

Take a payments API with a 30-day rolling correctness SLO of 99.9%, serving 5,000,000 requests over that window.

Monthly error budget:

A dependency degrades for 3 days early in the month, during which the service logs 4,600 failed requests:

That crosses the freeze threshold immediately. Only 400 failed requests of budget remain for the other 27 days of the window. At the service's steady-state failure rate before the incident, roughly 50 failed requests per day under normal load, even zero new releases would burn through that remaining budget in:

With only 8 days of steady-state headroom left against a 27-day remainder, adding any release risk on top of normal operation isn't safe, which is the concrete reason the freeze holds until the bad 3-day stretch rolls out of the 30-day window and the budget recovers, not until someone judges the incident "handled."

Trade-offs & pitfalls

• A freeze with no named decision-maker turns into either nobody enforcing it (defeats the point) or everybody enforcing it inconsistently (a hard "no" for a trivial change, a shrug for something risky, based on who's in the room). Naming the owner up front avoids both failure modes.
• Freezing feature work but not tightening the release gate on what still ships (bug fixes, reliability work) misses the point, those changes can burn more budget just as easily as a feature would.
• A policy that never has a real exception path gets quietly ignored the first time there's a genuinely urgent fix that needs to ship; a policy with a too-easy exception path gets used as a workaround every time. The approval-plus-log step above is the balance: fast enough not to block a real emergency, visible enough that it isn't free.
• Some teams propose their own numeric release-gating thresholds here (for example, "block deploys once daily burn rate implies budget exhaustion within N days") rather than a binary "budget is zero, freeze" rule, that's a reasonable variant worth walking through if asked, since a binary rule reacts only after the fact while a burn-rate threshold reacts to the trend.

Direct answer

An SLI is the measured signal (e.g. the percentage of requests that were fast and successful), an SLO is the internal target for that signal over a time window (e.g. 99.9% of requests succeed under 300ms over a rolling 30 days), and an SLA is the external, often contractual promise built on top of an SLO, usually with margin, and real consequences (service credits, penalties) if it's breached. In short: the SLI measures, the SLO targets, the SLA promises with a business wrapper attached.

Defining each one concretely, for an e-commerce checkout service

Explained for a non-technical stakeholder: the SLO is the bar the engineering team holds itself to internally so problems get caught and fixed before they become customer-visible; the SLA is the, usually more forgiving, bar the business is willing to be held to externally, on paper, with money attached if it's missed. The gap between the two is deliberate headroom, not sloppiness.

Worked example: computing the error budget

The error budget is the amount of allowed failure baked into the SLO. It's what lets a team ship changes at all instead of freezing forever.

For the 99.9% SLO above:

Over the 30-day measurement window (30 days equals 2,592,000 seconds):

So this service is allowed about 43 minutes of budget-consuming failure (however that failure is defined: downtime, over-300ms responses, and so on) across a 30-day window before the SLO itself is breached. If checkout handles, say, 1,000,000 requests over that same window, the equivalent request-based budget is:

Both framings, time-based and request-based, describe the same budget; which one is more useful depends on whether the SLI is availability-style (was the service up) or ratio-style (what fraction of requests succeeded).

Trade-offs and pitfalls

• Confusing SLO and SLA in conversation causes real problems: teams sometimes design their alerting and release-gating around the SLA (the looser, contractual number) instead of the SLO (the tighter, internal number), which means by the time anyone notices, the team is already close to breaching the external promise with no margin left to react.
• An SLO with no error-budget policy attached is just a number on a dashboard; the value comes from what happens when the budget is nearly exhausted (freeze risky launches, redirect engineering time to reliability work), not from the target itself.
• Picking an SLI that doesn't reflect real user experience (e.g. "server process is running" instead of "requests succeed within an acceptable time") gives a green dashboard while users are still unhappy. The SLI has to be as close to what the user actually experiences as the team can measure.

Direct answer

Walk the object graph depth-first, track which objects are currently ancestors of the node you're visiting (not just "have I ever seen this object"), replace any object still on that ancestor chain with a circular-reference marker, and redact any key whose name matches a configured pattern before it's included in the output. The distinction between "ancestor" and "ever seen" matters: a naive "seen before" check falsely flags a shared, non-cyclic reference (the same sub-object reached twice, but not a real cycle) as circular.

Structured elaboration

Approach

• Recurse through the value. For primitives and null, return them (with light handling for values JSON can't represent, like BigInt).
• Maintain a Set of "ancestors": objects currently on the path from the root down to the node being visited. Add an object when entering it, remove it when leaving it (after finishing its children).
• If the object being visited is already in the ancestor set, it's a genuine cycle: emit a marker instead of recursing.
• For each key, check it against a list of redaction patterns (regexes or plain strings); if it matches, replace the value with a redaction marker without recursing into it (so a redacted object's own nested secrets don't leak either).
• Serialize the resulting sanitized structure with JSON.stringify.

function createSanitizer(redactPatterns = []) {
  const redactors = redactPatterns.map((p) => (p instanceof RegExp ? p : new RegExp(p, 'i')));
  const isRedactedKey = (key) => redactors.some((rx) => rx.test(key));

  function sanitize(root) {
    const ancestors = new Set(); // path-based, not "ever visited"

    function walk(value) {
      if (value === null || typeof value !== 'object') {
        if (typeof value === 'bigint') return `[BigInt:${value.toString()}]`;
        if (typeof value === 'function' || typeof value === 'symbol') return undefined;
        return value;
      }
      if (ancestors.has(value)) return '[Circular]';
      ancestors.add(value);
      let result;
      try {
        if (Array.isArray(value)) {
          result = value.map((v) => walk(v));
        } else {
          result = {};
          for (const [k, v] of Object.entries(value)) {
            if (isRedactedKey(k)) { result[k] = '[REDACTED]'; continue; }
            const walked = walk(v);
            if (walked !== undefined) result[k] = walked;
          }
        }
      } finally {
        ancestors.delete(value); // leaving this node: it's no longer an ancestor
      }
      return result;
    }

    return JSON.stringify(walk(root));
  }
  return sanitize;
}

Worked example

Running this exact code (Node.js, no external dependencies) against three cases:

const sanitize = createSanitizer(['token', 'password', 'ssn', '^creditCard$']);

// 1. A true circular reference
const a = { user: 'alice', token: 'abc123' };
a.self = a;
console.log(sanitize(a));

// 2. A shared but non-cyclic reference (same object reached via two paths)
const shared = { region: 'us-east-1' };
const b = { primary: shared, backup: shared, password: 'hunter2' };
console.log(sanitize(b));

// 3. Nested redaction inside an array
const c = { users: [{ name: 'bob', creditCard: '4111111111111111' }], ssn: '123-45-6789' };
console.log(sanitize(c));

Actual output:

{"user":"alice","token":"[REDACTED]","self":"[Circular]"}
{"primary":{"region":"us-east-1"},"backup":{"region":"us-east-1"},"password":"[REDACTED]"}
{"users":[{"name":"bob","creditCard":"[REDACTED]"}],"ssn":"[REDACTED]"}

Case 2 is the one that separates a correct implementation from a common near-miss: shared is referenced twice (as primary and backup), but it's never its own ancestor, so it correctly serializes in full both times instead of collapsing the second occurrence to [Circular]. An implementation that tracks "every object ever seen" (a WeakSet populated once and never cleared) would get this case wrong and silently drop real, non-redundant data from the log.

Key points

• Ancestor tracking (add on enter, remove on exit) distinguishes a real cycle from a shared DAG (directed acyclic graph: a node reachable by more than one path, but not by itself) reference.
• Redaction happens before recursion into the value, so a redacted key's children (which might themselves contain sensitive nested data) never get walked or leaked piecemeal.
• Function and symbol values are dropped rather than serialized (JSON has no representation for them); BigInt is stringified explicitly since JSON.stringify throws on raw BigInt.

Complexity

• Time: O(n) where n is the number of reachable properties/elements, since each is visited exactly once (a true cycle short-circuits back to the marker rather than re-walking).
• Space: O(n) for the sanitized clone, plus O(d) for the ancestor set where d is the maximum depth of the object graph (not n, since ancestors are removed on exit).

Edge cases

• A shared (non-cyclic) reference appearing multiple times: must serialize in full each time, not collapse (shown in the worked example above).
• Very deep, non-cyclic graphs can still overflow the call stack; a depth limit with a truncation marker is a reasonable production safeguard even though it isn't shown here.
• Keys matching a redaction pattern by name but holding non-sensitive values (e.g., a field literally named "password_policy" matching a "password" pattern) will still get redacted; patterns need to be specific enough to avoid over-redacting fields you actually want in logs.
• Array elements go through the same ancestor/redaction logic as object values, including nested circular arrays (an array containing itself).

Trade-offs & pitfalls

• Regex-based key matching is simple but can both over-redact (matching an unrelated field whose name happens to contain "token") and under-redact (a sensitive field named something the pattern list didn't anticipate); pairing it with a small allowlist review during code review catches drift over time.
• Doing this synchronously on every log call adds CPU work on the hot path proportional to payload size; for very high-throughput services, consider sampling deep objects or moving sanitization to an async logging pipeline stage instead of inline in the request path.
• Centralizing the redaction pattern list (rather than letting each service define its own) is what actually prevents a sensitive field from leaking through one service that forgot to add it to its local list.
• Redacting at the source (in this utility, before the log line is even constructed) is safer than redacting downstream at ingestion, since it means the unredacted value never leaves the process or crosses the network in the first place.