Monitoring, Logging, and Observability Questions

Direct answer

Walk the object graph depth-first, track which objects are currently ancestors of the node you're visiting (not just "have I ever seen this object"), replace any object still on that ancestor chain with a circular-reference marker, and redact any key whose name matches a configured pattern before it's included in the output. The distinction between "ancestor" and "ever seen" matters: a naive "seen before" check falsely flags a shared, non-cyclic reference (the same sub-object reached twice, but not a real cycle) as circular.

Structured elaboration

Approach

• Recurse through the value. For primitives and null, return them (with light handling for values JSON can't represent, like BigInt).
• Maintain a Set of "ancestors": objects currently on the path from the root down to the node being visited. Add an object when entering it, remove it when leaving it (after finishing its children).
• If the object being visited is already in the ancestor set, it's a genuine cycle: emit a marker instead of recursing.
• For each key, check it against a list of redaction patterns (regexes or plain strings); if it matches, replace the value with a redaction marker without recursing into it (so a redacted object's own nested secrets don't leak either).
• Serialize the resulting sanitized structure with JSON.stringify.

function createSanitizer(redactPatterns = []) {
  const redactors = redactPatterns.map((p) => (p instanceof RegExp ? p : new RegExp(p, 'i')));
  const isRedactedKey = (key) => redactors.some((rx) => rx.test(key));

  function sanitize(root) {
    const ancestors = new Set(); // path-based, not "ever visited"

    function walk(value) {
      if (value === null || typeof value !== 'object') {
        if (typeof value === 'bigint') return `[BigInt:${value.toString()}]`;
        if (typeof value === 'function' || typeof value === 'symbol') return undefined;
        return value;
      }
      if (ancestors.has(value)) return '[Circular]';
      ancestors.add(value);
      let result;
      try {
        if (Array.isArray(value)) {
          result = value.map((v) => walk(v));
        } else {
          result = {};
          for (const [k, v] of Object.entries(value)) {
            if (isRedactedKey(k)) { result[k] = '[REDACTED]'; continue; }
            const walked = walk(v);
            if (walked !== undefined) result[k] = walked;
          }
        }
      } finally {
        ancestors.delete(value); // leaving this node: it's no longer an ancestor
      }
      return result;
    }

    return JSON.stringify(walk(root));
  }
  return sanitize;
}

Worked example

Running this exact code (Node.js, no external dependencies) against three cases:

const sanitize = createSanitizer(['token', 'password', 'ssn', '^creditCard$']);

// 1. A true circular reference
const a = { user: 'alice', token: 'abc123' };
a.self = a;
console.log(sanitize(a));

// 2. A shared but non-cyclic reference (same object reached via two paths)
const shared = { region: 'us-east-1' };
const b = { primary: shared, backup: shared, password: 'hunter2' };
console.log(sanitize(b));

// 3. Nested redaction inside an array
const c = { users: [{ name: 'bob', creditCard: '4111111111111111' }], ssn: '123-45-6789' };
console.log(sanitize(c));

Actual output:

{"user":"alice","token":"[REDACTED]","self":"[Circular]"}
{"primary":{"region":"us-east-1"},"backup":{"region":"us-east-1"},"password":"[REDACTED]"}
{"users":[{"name":"bob","creditCard":"[REDACTED]"}],"ssn":"[REDACTED]"}

Case 2 is the one that separates a correct implementation from a common near-miss: shared is referenced twice (as primary and backup), but it's never its own ancestor, so it correctly serializes in full both times instead of collapsing the second occurrence to [Circular]. An implementation that tracks "every object ever seen" (a WeakSet populated once and never cleared) would get this case wrong and silently drop real, non-redundant data from the log.

Key points

• Ancestor tracking (add on enter, remove on exit) distinguishes a real cycle from a shared DAG (directed acyclic graph: a node reachable by more than one path, but not by itself) reference.
• Redaction happens before recursion into the value, so a redacted key's children (which might themselves contain sensitive nested data) never get walked or leaked piecemeal.
• Function and symbol values are dropped rather than serialized (JSON has no representation for them); BigInt is stringified explicitly since JSON.stringify throws on raw BigInt.

Complexity

• Time: O(n) where n is the number of reachable properties/elements, since each is visited exactly once (a true cycle short-circuits back to the marker rather than re-walking).
• Space: O(n) for the sanitized clone, plus O(d) for the ancestor set where d is the maximum depth of the object graph (not n, since ancestors are removed on exit).

Edge cases

• A shared (non-cyclic) reference appearing multiple times: must serialize in full each time, not collapse (shown in the worked example above).
• Very deep, non-cyclic graphs can still overflow the call stack; a depth limit with a truncation marker is a reasonable production safeguard even though it isn't shown here.
• Keys matching a redaction pattern by name but holding non-sensitive values (e.g., a field literally named "password_policy" matching a "password" pattern) will still get redacted; patterns need to be specific enough to avoid over-redacting fields you actually want in logs.
• Array elements go through the same ancestor/redaction logic as object values, including nested circular arrays (an array containing itself).

Trade-offs & pitfalls

• Regex-based key matching is simple but can both over-redact (matching an unrelated field whose name happens to contain "token") and under-redact (a sensitive field named something the pattern list didn't anticipate); pairing it with a small allowlist review during code review catches drift over time.
• Doing this synchronously on every log call adds CPU work on the hot path proportional to payload size; for very high-throughput services, consider sampling deep objects or moving sanitization to an async logging pipeline stage instead of inline in the request path.
• Centralizing the redaction pattern list (rather than letting each service define its own) is what actually prevents a sensitive field from leaking through one service that forgot to add it to its local list.
• Redacting at the source (in this utility, before the log line is even constructed) is safer than redacting downstream at ingestion, since it means the unredacted value never leaves the process or crosses the network in the first place.

Direct answer

Observability rests on three complementary signal types: metrics, logs, and traces. Metrics tell you something is wrong and roughly how bad; logs tell you what specifically happened in a given event; traces tell you where in a multi-service request the time or failure occurred. None of the three alone gives a complete picture: a strong incident response usually starts with one pillar to detect and scope the problem, then pivots to another to find root cause.

The three pillars

Instrumentation example, one flow

For a checkout endpoint, an on-call engineer might instrument it like this: a checkout_requests_total counter metric with labels {status, payment_provider}, plus a checkout_latency_seconds histogram with the same labels for percentiles; a structured log line at the point of failure with fields {request_id, trace_id, error_type, payment_provider}; and a trace with spans named checkout.validate, checkout.charge, checkout.persist, each carrying the same trace_id that appears in the log line. The shared trace_id and request_id are what let you jump from "the metric moved" to "here is the specific failing request" to "here is the exact log line explaining why."

Trade-offs and pitfalls

• Treating one pillar as sufficient is the most common mistake: teams that only have logs end up searching blind during an incident because they have no aggregated signal telling them where to look first; teams that only have metrics can detect a problem but can't explain it.
• High-cardinality labels (like an unbounded user_id on a metric) turn cheap metrics into an expensive, slow-to-query mess; that data belongs in logs or traces instead.
• Trace sampling is a real trade-off: full sampling captures every rare failure but is expensive at scale; low sampling rates are cheap but can miss the exact failing request you need. Tail-based sampling (keep traces for slow or error requests) is a common middle ground.
• Retention windows differ by pillar in practice (metrics are cheap to keep for months, verbose logs and full traces are usually much more expensive to retain), which shapes how far back a postmortem can actually look.

Direct answer

I'd start with metrics, because they answer the cheapest, most important question first: is this an isolated blip or a real, ongoing problem, and is it affecting one endpoint or many? Metrics take seconds to check and tell you where to point traces and logs next, whereas jumping straight into logs on a single failed request risks spending several minutes reading detail for something that turns out to be a one-off retry that already resolved itself.

Prioritized first steps

1. Metrics first: check the error rate and latency (p95/p99) for the timing-out endpoint over the last 15 to 30 minutes. This shows whether it's a sustained pattern worth investigating now, or an isolated spike worth a note but not an emergency, and whether it's scoped to one endpoint or spread across the service.
2. Traces next, if the metric confirms a real pattern: pull a trace for a recent timed-out request to see which span in the call chain is actually slow or hanging. This shows where the time is going: our own code, a specific downstream dependency, or a shared resource like a connection pool.
3. Logs last, once I know where to look: with the specific span and a request ID in hand, pull the logs for that exact request to get the precise error, a timeout waiting on what exactly, instead of searching broadly through every log line mentioning "timeout."

Decision flow

flowchart TD
    A[Request times out] --> B[Check metrics first]
    B --> C{Isolated to one endpoint or widespread?}
    C -->|One endpoint| D[Pull traces for that endpoint]
    C -->|Widespread| E[Check shared dependency: DB, network, downstream API]
    D --> F[Find the slow span]
    E --> F
    F --> G[Pull logs for that span's request ID]
    G --> H[Read the error detail to confirm root cause]

If forced to pick only one signal

Under real time pressure, say a spike in slow logins and only one tool available before the next status update is due, I'd still pick metrics, not traces or logs, because metrics answer the triage question fastest: is this getting worse, better, or staying flat, and how many users are affected right now. That's what's needed to decide how urgently to escalate, even before the root cause is known. Traces or logs alone can explain one request beautifully while leaving me blind to whether the problem is spreading, which is the wrong trade-off in the first few minutes of an incident.

Trade-offs and pitfalls

• Starting in logs is tempting because it feels like "real investigation," but without a metric to scope the problem first, it's easy to spend the first several minutes reading detail about a request that isn't representative of what's actually happening.
• Traces are only useful if sampling actually captured the failing request; on a low-sample-rate service, the trace for the specific timeout you care about may simply not exist, in which case logs (which usually aren't sampled the same way) become the fallback.
• This ordering (metrics, then traces, then logs) is a good default, not a law: if metrics are known to be unreliable for this specific service, a common gap right after a migration, before dashboards catch up, it's reasonable to start with whichever signal is actually trustworthy.

Direct answer: Use a counter for anything that only goes up (total requests, total errors), a histogram for anything you need a distribution or percentile from (latency, per-stage duration), and a gauge for anything that moves up and down (in-flight request count). At each stage of the handler, auth, business logic, and the database call, you instrument the stage's duration as a histogram observation and any stage-specific failures as a counter increment, and you decide the request's overall status_code from what actually happened, not from having merely reached the end of the handler body.

Structured elaboration: metric per stage

Label the metrics with route, method, and status_code (or error_type for the error counter), and keep the label set the same at every stage so slicing "auth failures for this route" is a straightforward query.

Worked example: instrumenting the handler (Python, Flask + prometheus_client)

from time import perf_counter
from flask import Flask, request, jsonify
from prometheus_client import Counter, Histogram, Gauge

app = Flask(__name__)

REQUESTS_TOTAL = Counter(
    "http_requests_total", "Total requests", ["route", "method", "status_code"]
)
STAGE_LATENCY = Histogram(
    "http_stage_latency_seconds", "Per-stage latency", ["route", "stage"]
)
STAGE_ERRORS = Counter(
    "http_stage_errors_total", "Per-stage errors", ["route", "stage", "error_type"]
)
IN_FLIGHT = Gauge("http_in_flight_requests", "Requests currently being handled", ["route"])

# Stand-ins for a real auth check and a real database, so the example runs end to end.
FAKE_DB = {1: {"order_id": 1, "total_cents": 4200}}


def authenticate(req):
    if not req.headers.get("Authorization"):
        raise PermissionError("missing bearer token")


def run_business_logic(req, order_id):
    return {"order_id": order_id}


def query_database(order_id):
    return FAKE_DB.get(order_id)


@app.route("/orders/<int:order_id>", methods=["GET"])
def get_order(order_id):
    route = "/orders/:id"
    IN_FLIGHT.labels(route=route).inc()
    status_code = "500"
    try:
        t0 = perf_counter()
        try:
            authenticate(request)
        except Exception as e:
            STAGE_ERRORS.labels(route=route, stage="auth", error_type=type(e).__name__).inc()
            raise
        finally:
            STAGE_LATENCY.labels(route=route, stage="auth").observe(perf_counter() - t0)

        t1 = perf_counter()
        try:
            result = run_business_logic(request, order_id)
        except Exception as e:
            STAGE_ERRORS.labels(route=route, stage="business_logic", error_type=type(e).__name__).inc()
            raise
        finally:
            STAGE_LATENCY.labels(route=route, stage="business_logic").observe(perf_counter() - t1)

        t2 = perf_counter()
        try:
            row = query_database(result["order_id"])
        except Exception as e:
            STAGE_ERRORS.labels(route=route, stage="db", error_type=type(e).__name__).inc()
            raise
        finally:
            STAGE_LATENCY.labels(route=route, stage="db").observe(perf_counter() - t2)

        if row is None:
            status_code = "404"
            return jsonify({"error": "not found"}), 404

        status_code = "200"
        return jsonify(row), 200
    except Exception:
        status_code = "500"
        return jsonify({"error": "internal"}), 500
    finally:
        REQUESTS_TOTAL.labels(route=route, method="GET", status_code=status_code).inc()
        IN_FLIGHT.labels(route=route).dec()

Verified: a GET with a valid auth header and an existing order returns 200 and increments status_code="200"; a GET with no auth header raises inside the auth stage, is caught by the outer handler, and correctly returns 500 with status_code="500"; a GET for an order ID that doesn't exist returns 404 with status_code="404", not a false 200.

Key points

• Every stage gets its own histogram observation, so a slow request can be attributed to auth vs business logic vs DB without touching a debugger.
• The stage error counter carries error_type, not a raw exception message: exception messages are effectively unbounded strings and would blow up cardinality if used as a label.
• IN_FLIGHT is incremented before the try block and decremented in finally, so it stays correct even when the handler raises.
• status_code is set once, right next to each return, and REQUESTS_TOTAL is only incremented in the outer finally, after the response has actually been decided, not the moment the handler body finishes running. This matters because an earlier, naive version of this pattern (incrementing status_code="200" right before return row, 200, with no check on row) would silently record a 200 for a request that actually 404s or 500s downstream, if the response object itself turns out to be invalid; tying the label to the real outcome, evaluated in finally, is what keeps the metric honest.
• The route label uses the path template (/orders/:id), not the interpolated path, deliberately, so cardinality stays bounded by the number of distinct routes, not the number of distinct order IDs.

Complexity

• Each stage adds O(1) time overhead per request (a histogram observation and a counter increment are constant-time operations), so instrumentation cost scales linearly with the number of stages instrumented, not with request payload size.
• Space cost is proportional to the number of distinct label combinations actually observed (cardinality), which is why the route-template and error-type choices above matter more than the number of stages.

Edge cases

• A record that legitimately doesn't exist (a GET for an order ID nobody has) is a normal, expected outcome, not an exception: the handler checks for it explicitly and returns 404 with status_code="404", rather than letting a None flow into a raw response and being miscounted as success.
• An exception raised inside finally (e.g., the DB stage's cleanup itself failing) can mask the original exception. Instrumentation code should never be able to throw; wrap the metric calls defensively or keep them to simple counter/histogram calls that can't fail.
• Requests that never reach a stage (e.g., auth fails, so business logic and DB are never called) must not emit a stage-latency observation for stages they never entered; the code above achieves this because each stage's observe call is scoped inside its own try/finally, not run unconditionally at the end.
• Concurrent requests to the same route must not clobber the gauge, this is a library correctness question, not application code, and Prometheus client gauges are safe for concurrent inc/dec across threads.

Trade-offs & pitfalls

• If you also expose this app's own /metrics endpoint on the public listener, it becomes an unauthenticated read of internal state (route names, error rates). Serve it on a separate internal port or behind network policy, and scrape it as a sidecar in Kubernetes so the metrics port never leaves the pod network.
• Instrumenting every stage is more code to maintain; a smaller service might reasonably start with just the overall request histogram and one DB-call histogram, and add finer stages only once a real incident showed the coarse view wasn't enough.
• The subtlest bug in this whole pattern is deciding the status-code label from "the handler body finished without raising" instead of from the actual response. Anywhere a handler can return a response that doesn't map to a clean success (a None result, a partially-built object, a downstream call that returns an error payload with a 200 HTTP status), that gap has to be checked explicitly, or the metric silently lies about what the client actually received.

Direct answer

A counter only goes up (or resets to zero on a process restart) and is for counting events, like total requests or errors. A gauge holds a point-in-time value that can go up or down, like current queue depth. A histogram and a summary both capture a distribution of observed values, like request latency, so you can compute percentiles, but they differ in where that computation happens: a histogram lets the backend compute percentiles at query time from raw bucket counts, while a summary computes them client-side and ships the already-calculated quantile.

The four types side by side

Aggregation semantics that matter for dashboards versus alerts

• For dashboards: histograms let one query produce fleet-wide p50/p95/p99 by summing buckets across every instance, which is what you want for an aggregate latency panel.
• For alerts: counters (via rate()) are what you alert on for error-rate thresholds, gauges are what you alert on for instantaneous saturation thresholds like queue depth above N, and histogram-derived percentiles are what you alert on for latency SLOs.
• Summaries are the odd one out for fleet-wide alerting, because averaging five instances' p99s is not the fleet's real p99. A single instance handling an unlucky slice of traffic gets diluted by the others and hides inside the average.

Worked example

For a fleet of $n$ instances each exposing a histogram with identical bucket boundaries, the fleet-wide count in bucket $le$ is additive:

and the fleet-wide quantile is computed by interpolating within the merged buckets $B_{le}$, not by averaging each instance's own quantile. This is exactly why histograms (raw counts, additive) are the right choice for fleet-wide latency, and why summaries (already-computed quantiles, not additive) are not: summation is associative, a pre-computed quantile is not.

Trade-offs and pitfalls

• Using a gauge for something that's really cumulative (like a running error count tracked as a gauge that resets on deploy) loses the ability to compute an accurate rate across restarts. Use a counter and let rate() handle resets.
• Choosing a summary because it's simpler and skipping the bucket-tuning work of a histogram is a common shortcut that quietly breaks fleet-wide percentile dashboards later, once the service scales past one instance.
• Histogram accuracy is bounded by bucket granularity: more buckets means better percentile accuracy but higher cardinality and storage cost per series.