Monitoring, Logging, and Observability Questions

Direct answer: Build a burn-rate-driven policy engine, not a single "if error budget hits zero, do X" rule: it watches how fast the budget is burning, tiers its responses by severity and blast radius (a safe, automatic action for well-understood, reversible situations; a human-approved action for anything with wide blast radius), and every automated action must be independently reversible and testable in isolation before it's ever allowed to run against real traffic.

Structured elaboration

Two-tier action model

• Tier A (automated, safe): actions that are cheap to reverse and scoped narrowly, rolling back a single feature flag, shifting a small percentage of traffic away from a degraded region. These fire automatically once burn rate crosses a defined threshold.
• Tier B (automated proposal, human approval required): actions with wide blast radius, freezing all deploys org-wide, failing over an entire region. The system proposes the action with its reasoning and evidence, but a human clicks approve.

Safety checks before any automated action

• Verify the rollback target actually exists and was recently healthy (don't roll back to a flag state that was itself broken).
• Check a dependency graph so a feature-flag rollback or traffic shift doesn't break something else that depends on the current state.
• Rate-limit the automation itself: one automated action per service per cooldown window, so a flapping signal can't trigger a rollback loop.
• Every action requires a kill switch: one command that halts all in-flight and future automated actions immediately, checked before every state transition, not just at startup.

Testing without risking a bad automated rollback in production

• Run the exact policy engine against a staging environment with fault injection (inject latency, error rate) and confirm it fires the correct tier at the correct threshold, with no live traffic involved.
• Roll out the automation itself progressively: shadow mode first (it decides what it would do and logs the decision, but doesn't act), then Tier A actions only against a small percentage of traffic or a low-risk service, then expand.
• Chaos-test the automation's failure modes specifically: what happens if the actuator (feature-flag API, load balancer) is unreachable when the policy tries to act? The system should fail toward "alert a human," never toward "retry aggressively" or "silently do nothing."

Worked example (architecture)

flowchart TD
    A["Metrics collector<br/>Prometheus / Datadog"] --> B["SLO evaluator<br/>burn-rate rules"]
    B --> C{"Burn rate<br/>threshold?"}
    C -->|below warning| D["No action, log burn rate"]
    C -->|warning| E["Notify on-call + owner"]
    C -->|critical, Tier A| F["Automated safe action<br/>flag rollback / traffic shift"]
    C -->|critical, Tier B| G["Propose action,<br/>require human approval"]
    F --> H["Actuator<br/>feature-flag API / LB / gateway"]
    G -->|approved| H
    H --> I["Verify: smoke test +<br/>post-action SLI check"]
    I -->|recovered| J["Log to audit trail,<br/>close incident"]
    I -->|not recovered| K["Escalate to Tier B<br/>/ page human"]
    K --> G

This makes the tiering concrete: burn rate below a warning threshold does nothing but log, a warning-level burn notifies without acting, and only a critical burn rate triggers an action, split into Tier A (auto-execute) and Tier B (propose, wait for approval). Every path converges on a verification step, so the system checks its own work rather than assuming the action succeeded.

Trade-offs & pitfalls

• More automation reduces time-to-mitigate but increases blast radius if the policy engine itself has a bug; that's exactly why Tier A is scoped to narrow, reversible actions and Tier B requires a human for anything wider.
• A automation that fires too eagerly (low threshold, short window) risks flapping, an action, then a rollback of that action, then re-triggering. The cooldown and rate-limit above exist specifically to prevent this, and any burn-rate threshold should be validated against real historical burn-rate noise before being trusted to act automatically.
• Testing this system is unusually hard because the dangerous failure mode (a bad automated rollback under real production load) is exactly what you can't safely rehearse in production. Shadow mode plus staged rollout of the automation itself is the practical answer, accept slower confidence-building in exchange for never letting an untested policy touch live traffic.
• CI/CD gating integration (blocking new deploys while budget is critically low, on top of the runtime mitigations above) closes a related but separate gap: the runtime system reacts to what's already live, a deploy gate prevents making things worse while it's still burning.

Direct answer

I'd start from user experience outward: request success rate, latency (p95/p99), and traffic or throughput are the three signals that most directly reflect whether real users are having a good or bad time right now, so those are what I'd page on if I could only pick a handful. Everything else (queue depth, resource usage, thread-pool saturation, dependency health) is valuable for diagnosis and capacity planning, but it's typically a leading indicator or a root-cause detail rather than something a first responder needs to be paged on directly.

What I'd monitor, and how I'd prioritize

This is the familiar "four golden signals" framing (latency, traffic, errors, saturation), narrowed here to the three that most directly track user-visible harm, with saturation kept as a fast diagnostic step rather than a primary page.

Worked example: deciding what pages versus what doesn't, for a checkout service

If I could only page on a handful of signals for this service, I'd page on error rate crossing a sustained threshold (checkout failing for real users), p95/p99 latency crossing a threshold (checkout technically succeeding but painfully slow), and a sudden traffic drop (which often means something upstream, like a CDN or DNS issue, broke before requests even reach us, and a pure error-rate alert would miss it because there's no request to error on).

I'd deliberately not page on CPU or memory alone: high CPU that isn't yet causing elevated latency or errors is useful to know about, and worth watching as a leading indicator for a proactive look, but paging on it directly tends to produce alerts that fire before there's any actual user impact, which is exactly the kind of noisy, not-yet-actionable signal that trains people to ignore pages. The right response to rising CPU with no user-facing symptom yet is usually "look at this during business hours and consider scaling," not "wake someone up."

Trade-offs and pitfalls

• Paging on too many signals defeats the purpose of prioritizing at all; if everything can page, the team is back to full alert fatigue with extra steps. The "handful" constraint in the question is doing real work: it forces a genuine choice about what's essential.
• Traffic-drop alerts need a sensible baseline that accounts for real traffic patterns (day of week, time of day, known low-traffic periods); a naive fixed threshold will false-positive constantly on legitimate quiet periods.
• Saturation metrics are tempting to page on because they feel proactive, but a resource that's near its limit and staying stable isn't yet a user-facing problem. The discipline is to treat saturation as a diagnostic and capacity-planning signal, escalating to a page only once it actually starts producing errors or latency.

Direct answer

Split this into two tracks. In the next few days, go after the two named drivers directly with reversible, low-risk levers: tag hygiene to cap cardinality (the number of distinct label/tag value combinations a metric or log field produces, since every unique combination becomes its own stored time series or index entry), and adaptive sampling that keeps close to full fidelity for errors and anomalies while cutting the sampling rate hard for routine traffic. Over the following weeks to months, replace flat sampling with a real tail-based pipeline and separate high-cardinality debug context from the aggregate metrics store, so cost scales with genuinely interesting traffic instead of total traffic.

Structured elaboration

Diagnose the drivers first

High-cardinality tags (raw user IDs, session IDs, request IDs used as labels) multiply the number of unique time series or indexed log fields, and cost in most telemetry backends scales with that cardinality, not just with event volume. Full trace sampling multiplies volume directly: capturing 100% of traces costs roughly proportional to however many times more spans that is than whatever sampling rate you had before.

Next few days: quick, reversible levers

• Tag hygiene / cardinality caps. Identify the highest-cardinality fields and stop using raw free-form values as indexed labels; hash or drop them, or move them into unindexed log body content instead.
• Adaptive sampling as a stopgap. Keep close to 100% sampling for traces tied to errors or already-firing alerts, and drop the sampling rate hard for routine, healthy traffic. This is the single biggest lever because it directly reverses the "full trace sampling" driver while explicitly protecting the traces you'd need during a P0.
• Shorten retention on the highest-cardinality dimensions rather than cutting them entirely, so recent incidents keep full fidelity and only old, rarely-queried data ages out sooner.

Bigger architectural changes (weeks to months)

• True tail-based sampling pipeline: buffer full trace data briefly, decide what to keep after seeing whether the trace was anomalous (error, high latency, matches an active alert), not before.
• Separate the debug stream from the metrics store: high-cardinality request-level context lives in a short-retention, low-cost store; aggregated metrics and dashboards run on a high-throughput store that never carries per-request cardinality.
• Schema and cardinality linting in CI so a new high-cardinality label can't ship without review, which is what let costs triple in the first place.
• Multi-region Prometheus federation is a concrete example of the kind of durable architectural change worth pursuing: each region runs its own low-cardinality local Prometheus for fast local queries and alerting, and a global layer only pulls pre-aggregated, low-cardinality rollups for cross-region dashboards, instead of every region shipping raw high-cardinality series to one central store.

graph LR
  L1[Region A local Prometheus] --> G[Global federated query layer]
  L2[Region B local Prometheus] --> G
  L3[Region C local Prometheus] --> G
  G --> D[Cross region dashboards]

• Multi-tenant isolation is the complication that makes this harder at scale: without per-team or per-service cardinality quotas, one noisy team can blow through the whole organization's telemetry budget again even after you've fixed the current spike, so the architectural fix needs enforcement, not just cleanup.

What not to sacrifice

Whatever you cut, keep near-full fidelity specifically for traces linked to errors, active alerts, or recent deploys. That is what "investigate a major incident quickly" actually depends on; the routine, healthy-path traffic is where nearly all of the safe savings live.

Worked example

Here is a fully worked numeric illustration with pinned assumptions (not a real account's actual numbers, since none were given, but a concrete, checkable model of how the two named drivers combine and how the near-term fix reduces them). Model per-second ingest cost as sampling rate times request rate times average indexed bytes per span, holding request volume constant at 50,000 requests/sec:

$C_0$ is the baseline before the growth: 50% trace sampling, 1.0 KB of indexed bytes per span. $C_1$ is today: sampling went to 100% (a 2x factor) and indexed bytes per span grew to 1.5 KB because of added high-cardinality tags (a 1.5x factor), and 2 times 1.5 is exactly 3, reproducing the "tripled" in the question from the two named drivers.

Now apply the near-term fix: keep 100% sampling only for the roughly 2% of traffic that's already flagged as an error or anomaly, drop routine traffic sampling to 20%, and revert the indexed-bytes-per-span back to 1.0 KB via tag hygiene:

That's an 85.6% cut from today's cost, using only the two quick levers, and it lands below the original pre-growth baseline ($10.8$ vs $25$ MB/s) while still capturing every error and anomalous trace at full fidelity. The exact percentages depend on the assumed error-traffic fraction and sampling rates, but the structure of the calculation, and the fact that error/anomaly traffic is what stays at full fidelity, is the part that generalizes.

Trade-offs and pitfalls

Aggressive sampling on "routine" traffic means you lose fidelity on the silent tail: patterns that are unusual but don't yet trip an error or alert (a slow memory creep, a rare-but-legitimate code path) become harder to investigate retroactively, because they were never flagged as interesting at capture time. Cardinality caps can silently drop a dimension someone actually needed for a future investigation if they're applied without review, so pair caps with a lightweight approval process rather than a blanket ban. Retention cuts trade long-tail trend analysis and slow-burn postmortems for savings, which is usually the right trade for cost but should be a stated decision, not an accident.

Direct answer

Yes, and the honest answer to what it buys you is the same thing version control buys application code: review before it ships, a diff when something changes, and the ability to revert a bad alert threshold as fast as you'd revert a bad deploy. What's harder is that a GUI lets you make a quick, well-intentioned tweak in seconds, and observability-as-code trades that speed for review overhead, which is worth it for anything that pages someone, but can feel like friction for a genuinely trivial change.

Structured elaboration

What managing dashboards and alerts as code actually buys you

What's harder about it

The as-code tooling has its own learning curve (a YAML or JSONNET dialect on top of whatever the underlying platform's own query language already is), and some dashboard-heavy workflows, like eyeballing a panel while dragging its boundaries to get the layout right, are genuinely faster in a GUI. Not every vendor supports full-fidelity export/import either, so "everything as code" sometimes means committing to a subset of features the export format actually preserves.

Worked example

A concrete before/after: an engineer edits an alert rule's PromQL expression and, in the process, introduces an unbalanced parenthesis. In a GUI-only workflow, that error might not surface until the rule silently fails to evaluate in production, no clear symptom, just an alert that never fires again. In an as-code workflow, the same typo is caught by a CI lint step (promtool check rules) that parses the file and fails the pipeline before the change merges, so the broken rule never reaches production at all. The value isn't the specific tool, it's that a syntax error becomes a blocked PR instead of a silently dead alert discovered weeks later during an incident it should have caught.

Trade-offs and pitfalls

A passing lint check only proves the syntax is valid, not that the threshold or query logic is actually correct, so treat CI validation as catching one class of bug (the code doesn't parse) and pair it with an actual review of the logic, not a substitute for that review. Shared dashboard-as-code modules can also create merge conflicts when multiple teams edit the same file, which is a real cost that a GUI never has, and worth designing around (per-team files, clear ownership boundaries) rather than discovering the hard way. Finally, don't over-apply the discipline: a genuinely low-stakes internal debugging dashboard that nobody pages off of doesn't need the same review overhead as a paging alert rule, and treating every change identically just trains people to route around the process.

Direct answer

OpenTelemetry (OTel) is a vendor-neutral standard and set of libraries for generating and exporting the three core telemetry signals, traces, metrics, and logs, so applications instrument once and can send data to any compatible backend instead of being locked into one vendor's proprietary agent. Auto-instrumentation gives you traces and metrics for common frameworks with no code changes; manual instrumentation is what you add for anything specific to your application's own logic.

The main pieces and what each is responsible for

• SDK: the language-specific library that generates telemetry, propagates trace context across service calls, applies sampling decisions, and batches data before sending it.
• Instrumentation: the code that actually produces spans and metrics from a specific library or framework, either automatic (a framework-level hook that wraps HTTP clients, database drivers, and so on with zero application code changes) or manual (calling the API directly to create a span or record a metric around your own business logic).
• Collector: a standalone process sitting between your services and your backends. It receives telemetry through receivers, transforms it through processors (batching, filtering, redacting sensitive attributes), and forwards it through exporters to one or more backends. Using a Collector is optional but common, since it lets you change or add backends, apply central sampling and filtering, and buffer against a backend outage without touching application code.

flowchart LR
    A[Application code] --> B[OTel SDK plus instrumentation]
    B --> C[OTLP exporter]
    C --> D[OTel Collector]
    D --> E[Receivers]
    E --> F[Processors such as batch, filter, redact]
    F --> G[Exporters]
    G --> H[Metrics backend]
    G --> I[Trace backend]
    G --> J[Log backend]

Auto-instrumentation versus manual instrumentation

• Auto-instrumentation wraps known libraries (web frameworks, HTTP clients, database drivers) automatically, typically by installing an instrumentation package and running the app through a small launcher. You get spans for every HTTP request and database query with no code changes.
• Manual instrumentation is calling the OTel API directly: wrapping a specific function or business operation in a span, adding a custom attribute, or recording a domain-specific metric, for anything auto-instrumentation can't see because it's specific to your application logic.
• Most real services use both: auto-instrumentation for the framework and library boundary, which gets useful traces immediately, with manual instrumentation layered on top for the business-logic detail that actually helps debug a specific problem.

What you actually configure when turning auto-instrumentation on

• Service identity: a service.name resource attribute so telemetry is attributable in the backend.
• Exporter target: where telemetry goes, typically an OTLP endpoint pointing at a local Collector rather than directly at a vendor backend, to get the buffering and redaction benefits described above.
• Sampling: a sampling rate or strategy so you're not shipping and paying to store a trace for every single request at high traffic volumes.
• Batching and retry settings: how much telemetry to buffer before sending, and how to behave if the export target is temporarily unreachable, so a backend blip doesn't back-pressure the application itself.
• Attribute filtering: dropping or redacting specific attributes, like anything that might carry PII, before export, usually configured in the Collector rather than per-service.

Worked example

Enabling auto-instrumentation for a Python web service, and what each configuration choice is actually protecting against:

pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install
OTEL_SERVICE_NAME=checkout-api \
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317 \
OTEL_TRACES_SAMPLER=parentbased_traceidratio \
OTEL_TRACES_SAMPLER_ARG=0.1 \
opentelemetry-instrument python app.py

This installs the distro, a curated bundle of common instrumentations, points the exporter at a local Collector rather than a vendor endpoint directly (so a Collector-side outage doesn't require redeploying the app to change destinations), and samples 10% of traces, a starting point for a moderate-traffic service, tuned down further if trace volume or cost becomes a problem, or up temporarily while debugging a specific incident.

Operational concerns for the Collector

Because the Collector becomes real production infrastructure once you rely on it, not just a config file, it needs the same operational care as any other service: its own scaling (a single instance can become a throughput bottleneck or a single point of failure, so run it as a horizontally scaled deployment or a per-node sidecar), monitoring for its own latency and queue depth (a slow or backed-up Collector adds latency to the telemetry pipeline and, if buffers overflow, silently drops data), and capacity planning for its own memory (batching and any Collector-side processing holds data in memory before export).

Trade-offs and pitfalls

• Sampling 100% of traces to be safe is rarely sustainable past a small service. The real trade-off is trace completeness versus storage and query cost, and most teams land on high (or no) sampling for errors and slow requests, low sampling for routine successful requests.
• Running the Collector as a single centralized instance for convenience creates exactly the single point of failure it's meant to help avoid. Treat it as a real service in the architecture, not an afterthought config file.
• Auto-instrumentation gives fast breadth but shallow depth. Don't stop there for the service that's actually causing pain, add manual spans and attributes for the business logic that needs debugging.
• Common wrong turn: exporting telemetry directly from every service straight to the vendor backend because it's simpler, which works fine at small scale but means every future change, adding redaction, switching vendors, adding a second backend, requires touching every service instead of one Collector config.