Observability and Monitoring Architecture Questions

Direct Answer

The OpenTelemetry Collector is a pipeline with three stages: a receiver takes telemetry in (from an app via OTLP, or by scraping an endpoint), a processor chain transforms it in flight (batching, filtering, enriching), and an exporter sends it out to one or more backends. Deploy it as a per-host agent when you want telemetry to leave the host already batched and lightly processed; deploy it as a central gateway when you want heavier processing (like sampling decisions that need to see traffic from many sources at once) done in one place instead of duplicated on every host.

Structured Elaboration

Pipeline diagram

flowchart LR
  A["Instrumented App"] -->|"OTLP"| R["Receiver"]
  R --> P["Processor Chain: batch, filter, enrich"]
  P --> E["Exporter"]
  E --> B1[("Backend A")]
  E --> B2[("Backend B")]

Receiver
Accepts telemetry in a given format and protocol, most commonly the OTLP receiver (gRPC or HTTP) for data pushed from an instrumented application, or a Prometheus-style receiver that scrapes metrics endpoints on a schedule. A single collector can run multiple receivers at once.

Processor
Sits between receiver and exporter, transforming data without changing where it came from or where it's going. Common processors: batch (groups items to amortize per-request overhead, see the worked example), memory_limiter (sheds load before the process runs out of memory), attributes (adds, renames, or redacts fields), and sampling processors (drop a fraction of data by a configured rule).

Exporter
Sends processed telemetry to one or more backends in whatever protocol that backend expects (Prometheus remote-write, OTLP to a vendor, a file, and so on). A pipeline can fan out to multiple exporters from the same processed stream.

Agent versus gateway
An agent runs on or near every host (often as a DaemonSet), doing initial receipt and light processing close to the source. A gateway is a smaller number of centrally deployed instances that agents forward to, doing heavier fleet-wide processing (tail-based trace sampling needs to see many sources at once to decide, which an agent working alone cannot do) before the final export.

Worked Example

Consider sending spans without the batch processor, one OTLP request per span, versus with it, spans grouped into batches of 500 before a single request.

Assume each request carries a fixed protocol/framing overhead of roughly 1,024 bytes (headers, envelope) and each span payload is roughly 500 bytes (a stated design input, not a measured wire capture).

Without batching, one span per request:

Roughly two-thirds of every byte sent is framing overhead, not span data.

With batching, 500 spans per request:

The batch processor takes the overhead fraction from about 67% down to about 0.4%, which is the concrete reason batching sits in almost every collector pipeline by default rather than being an optional tuning step.

Trade-offs and Pitfalls

Running the collector as an agent everywhere is simple to reason about but limits what any single instance can see: it cannot make a decision (like tail-based trace sampling) that depends on data arriving at a different host. That kind of decision needs a gateway tier the agents forward to.

Too large a batch size trades throughput efficiency for latency: a batch that only flushes every few seconds under low traffic delays every piece of telemetry inside it by that long, even though the framing-overhead math above still favors big batches under high traffic. Most collector configurations flush on whichever comes first, a size threshold or a time threshold, exactly to avoid this under low load.

A processor chain's order matters and is easy to get wrong: a memory_limiter should run early (to shed load before spending CPU on later processors), and a batch processor should generally run late, right before the exporter, so it only batches data that already survived filtering or sampling decisions rather than wasting batch capacity on data about to be dropped.

Direct answer

Use three correlation identifiers with distinct scopes: job_id for the logical run, task_instance_id for a specific task within it, and attempt_id for a specific retry of that task, and put them in the right telemetry signal for their cardinality. Metrics get labeled only by low-cardinality dimensions like task_name (a fixed, small set), while job_id/task_instance_id/attempt_id live on trace spans and structured logs, which are built to handle high-cardinality identifiers. A durable idempotency ledger keyed on job_id + task_name makes retries and partial failures attribute correctly instead of double-counting or getting lost.

Identifier design

• job_id: one per logical run of the pipeline (a specific execution of "yesterday's ETL," not the pipeline definition itself).
• task_instance_id: one per task within that run, stable across retries of the same task.
• attempt_id: increments on each retry of a task_instance_id.
• All three propagate together through logs, trace spans, and lineage events, so any of the three signals can be joined against the others for a given execution.

Why the label placement matters: cardinality

Cardinality here means the number of distinct label-value combinations a metric can produce; each distinct combination is a separate time series the metrics backend has to store and index. A metric labeled by something that's unique per job run (like task_instance_id) creates a brand-new time series on every single run, forever, which is exactly the failure mode metrics backends are not built for.

Metrics: aggregate counters/histograms keyed only by task_name (a fixed set, e.g. 50 distinct task names), plus a small number of other genuinely low-cardinality dimensions like environment or region. Never job_id, task_instance_id, or attempt_id as a metric label.

Traces: one root span per job_id, child spans per task attempt, with job_id, task_instance_id, attempt_id as span attributes (not labels that create new series). Trace backends are designed for exactly this kind of high-cardinality attribute.

Logs: structured JSON logs carrying the same identifiers, for full-detail debugging of a specific run.

Lineage: events keyed by asset_id, carrying job_id as a join key back to the run.

Idempotency and retry attribution

• A durable status store records terminal outcomes keyed by job_id + task_name (not attempt_id), written via compare-and-swap. A retried attempt that races with an already-succeeded prior attempt loses the CAS and cannot double-apply its effect.
• Job-level SLIs (success rate, throughput) are computed from terminal status only, not from every attempt. A task that fails twice and succeeds on the third attempt counts once, as a success, in the job-level SLI; the retry history is visible in traces/logs for debugging, but doesn't pollute the aggregate metric.
• Partial failures (some tasks in a job succeed, others don't) are recorded per-asset_id in the lineage stream, so a downstream consumer can tell exactly which outputs are trustworthy even if the overall job is marked failed.

Worked example

Assume $N = 10{,}000$ job runs/day, each with $T = 50$ tasks.

If a metric were (incorrectly) labeled by task_instance_id:

Against a typical metrics-backend active-series budget (illustrative figure of 1,000,000 active series for a shared cluster), this single pipeline alone would consume roughly half the entire budget every single day, and it never stops growing since task_instance_id values never repeat.

Labeled correctly by task_name instead:

The correct label placement isn't a minor optimization, it's the difference between a metrics backend that stays healthy indefinitely and one that degrades daily as more job runs accumulate.

flowchart LR
  JobRun[Job Run: job_id] --> Task[Task Attempt: task_name + attempt_id]
  Task --> IdemStore[Idempotency Store: CAS by job_id + task_name]
  Task --> Metrics[Metrics: labeled by task_name only]
  Task --> Trace[Trace Span: job_id + task_instance_id attrs]
  Task --> Lineage[Lineage Event: asset_id + job_id]
  IdemStore --> Finalizer[Job Finalizer: terminal SLI]
  Metrics --> Finalizer
  Lineage --> Catalog[Data Catalog]

Trade-offs and pitfalls

• The most common mistake in batch-job telemetry is putting a per-run identifier directly on a metric label because it's convenient for a one-off debugging query; it's cheap the first time and catastrophic at scale, exactly because cardinality growth is invisible until the backend is already struggling.
• Attempt-level metrics (per-attempt counters) are still useful for detecting retry storms or flaky tasks, but they should be aggregated at task_name granularity (a counter incremented per attempt, not a new series per attempt), keeping the diagnostic value without the cardinality cost.
• Idempotency keyed on job_id + task_name assumes tasks within a job have stable, unique names; a pipeline that dynamically generates task names (e.g. embedding a partition key in the name) reintroduces the same cardinality and idempotency problems this design was built to avoid, so dynamic task naming needs its own bounded-cardinality scheme.
• Computing SLIs from terminal status only is correct for "did the job ultimately succeed," but it can mask a job that's technically succeeding while burning through an unhealthy number of retries; that's why attempt-count needs its own aggregated signal (task-level, not job-level) even though it isn't the primary SLI.

Direct answer

Frame this as a non-inferiority experiment (a test designed to show a new approach isn't meaningfully worse than the current one, rather than trying to prove it's better), not a simple A/B test: the new sampling strategy only ships if it's statistically shown to not meaningfully hurt error-trace recall, while independently delivering the promised cost reduction. Randomize traffic 50/50 at a stable unit (request or session, sticky for the duration), size the experiment around the rarer, riskier metric (recall of error traces, since errors are a small fraction of traffic), and use a one-sided non-inferiority test on recall alongside a standard comparison on cost, with the sample-size math driven by whichever metric needs more data.

What to measure

• Primary safety metric: recall of error traces, the fraction of true error-bearing requests whose trace was actually sampled. This is the one that can silently regress and cause real incident-investigation blind spots, so it's the non-inferiority gate.
• Primary efficiency metric: cost reduction (bytes stored or ingested per unit traffic), the thing the new strategy is promising.
• Secondary: false-negative rate (1 minus recall, reported for interpretability), precision of sampled error traces, and downstream alerting rate (to catch a strategy that technically samples errors but too late to be useful).

Traffic split and safety

50/50 randomized split, sticky per request/session to avoid the same logical flow bouncing between arms. Roll out to a small canary slice first (as in the sampling control-plane design) before the full 50/50 split, and define an explicit stopping rule: if observed recall drops noticeably below the non-inferiority margin at any interim check, stop immediately rather than waiting for the full planned duration.

Sample size: why recall drives the experiment duration, not cost

Cost is a continuous metric with typically low relative variance around its mean, so it needs comparatively few observations to detect a large (50%) effect. Recall is a proportion computed only over the rare subset of traffic that's actually an error trace, so it needs many more total observations to gather enough error-trace examples. The recall calculation is the binding constraint.

Non-inferiority sample size for a proportion, with baseline recall $p_1 = 0.95$, a tolerated absolute margin $\delta = 0.03$ (willing to accept up to a 3-point recall drop), one-sided $\alpha = 0.05$ ($z_\alpha = 1.645$), and 80% power (statistical power: the probability of correctly detecting a real effect when one truly exists) ($z_\beta \approx 0.84$, the z-score corresponding to that power level):

Using $p_2 \approx p_1$ for the planning-stage variance estimate (standard practice: if the treatment truly isn't worse, its recall should be close to baseline):

Round up: $n_{per\ arm} = 652$ error traces needed in each arm.

Converting to total traffic and duration

Assume the production error rate (fraction of traces that are true error-bearing) is $0.4\%$ ($0.004$):

Assume the feature area under test generates $40{,}000$ eligible traces/day total, split $20{,}000$/arm/day:

The statistical floor is about 9 days, but a real rollout should run at least one full extra weekly cycle beyond that to cover weekday/weekend traffic pattern differences that a single 9-day window (which starts mid-week) wouldn't cleanly capture. Rounding up to 14 days (two full weekly cycles) is the practical duration, not because more samples are needed, but because non-stationarity in traffic mix, not statistical power, is the real constraint once the count floor is already cleared in under two weeks.

Statistical test and decision rule

• Recall: one-sided non-inferiority test on the difference in proportions. Compute the one-sided 95% confidence interval for $p_{treatment} - p_{control}$; declare non-inferior if its lower bound exceeds $-\delta = -0.03$.
• Cost: two-sample comparison (t-test, or a nonparametric test like Mann-Whitney, which compares two groups by ranking values instead of assuming a specific distribution shape, if the cost distribution is heavy-tailed) on relative cost reduction, reported with a 95% CI.
• Decision rule: ship only if both hold: the recall non-inferiority test passes, and the observed cost reduction is at least 45% (a margin under the 50% target) with a 95% CI lower bound above 40%. Either condition failing blocks the rollout; a border-line result triggers extending the observation window rather than a coin-flip decision.

Trade-offs and pitfalls

• Sizing the experiment around cost instead of recall is the most common mistake here: cost differences are usually large and easy to detect quickly, which tempts a team to call the experiment "done" long before there's enough error-trace data to trust the recall result.
• Using $p_2 \approx p_1$ for the sample-size calculation is a planning-stage approximation, not a guarantee; if the true treatment recall is meaningfully worse than assumed, the actual achieved power will be lower than the targeted 80%, which is a reason to keep monitoring recall through the full run rather than only checking it once at the end.
• A $0.4\%$ error rate is an assumption plugged into the traces-per-arm conversion; if the real error rate is lower, the same statistical floor requires proportionally more total traffic and a longer run, so this number should be pulled from real production data before finalizing the experiment plan, not assumed.
• Extending the run to two weekly cycles helps with weekday/weekend seasonality but doesn't protect against a one-off event during the window (a major incident, a deploy freeze) skewing the error-trace mix; a senior reviewer would still sanity-check the observed error-rate distribution during the run against historical baselines before trusting the result.

Direct answer

Fingerprint every deployed model as a composite hash (weights, container digest, git commit, dataset snapshot ID, config) and register that fingerprint once, immutably, in a provenance registry at deploy time. At inference, attach only a small reference, a model_version_token, plus an input checksum and timestamp to each trace span and log line, never the full metadata block. Reproducing an investigation means resolving the token through the registry to get the full fingerprint, then re-materializing the dataset snapshot and re-running inference in the pinned environment.

What to fingerprint and store

• Artifact fingerprint: SHA-256 of the serialized model weights/graph.
• Build fingerprint: container image digest, dependency lockfile hash, framework/runtime versions.
• Source fingerprint: git commit hash, repo path, and (if deploying from an uncommitted patch, which should be discouraged but happens) a patch ID.
• Data fingerprint: dataset snapshot ID from the feature store or data lake, including the sampling seed used to build it, so the exact training data is reconstructable, not just referenced by a mutable name.
• Config: hyperparameters, preprocessing pipeline spec, and the model's declared input/output schema.
• All of this is composed into one immutable registry record, and that record's content-addressed ID is what gets minted as the short model_version_token attached to inference traffic.

Attaching provenance without bloating every span

The naive approach, inlining the full metadata block (dataset ID, commit hash, config hash, environment digest) into every span, scales the span-storage cost with inference volume times metadata size, and that metadata is redundant across every inference from the same deployed model version. Instead, attach only {model_version_token, input_checksum, timestamp} to each span/log; the token is a lookup key into the registry, not a copy of the registry entry. The registry itself grows only with the number of distinct model versions deployed, not the number of inferences.

Worked example

Assume the serving system handles $500$ inferences/sec.

Token-based reference approach: model_version_token (36-char UUID) + input_checksum (64-char SHA-256 hex) + timestamp (20-char ISO8601) $\approx 120$ bytes of values, plus an assumed 80 bytes of attribute-key overhead, for $200$ bytes/span:

Full-inline approach (assume a full provenance JSON block, all fingerprint fields plus a short eval-metrics summary, is $\approx 1{,}200$ bytes/span):

Token-based indirection cuts the per-inference span overhead by 6x compared to inlining full provenance on every span, which is the concrete answer to "without bloating every span."

Registry cost, for comparison: assume the org deploys roughly 50 distinct model versions/month across all models, and each registry record (including eval report references) is $\approx 5$ KB:

The registry itself is negligible; essentially all of the storage cost lives in the per-inference reference overhead (8.64 GB/day, roughly 259 GB/month at 30 days), confirming that token size is what to optimize, not registry design.

flowchart LR
  Artifact[Model Artifact] --> Fingerprint[Fingerprint Service]
  Fingerprint --> Registry[Immutable Provenance Registry]
  Registry --> Deploy[Deploy: mint model_version_token]
  Deploy --> Middleware[Inference Middleware]
  Middleware --> Span[Trace Span + token + checksum]
  Middleware --> Log[Structured Log + token]
  Span --> Backend[Trace Backend]
  Investigator[Investigator] --> Registry
  Registry --> Reproduce[Reproducibility Sandbox]

Retention policy

The per-inference reference (token, checksum, timestamp) rides along with the rest of the trace's attributes under the same operational retention/tiering policy as other span data, since on its own it's small and disposable once a trace ages out. The registry record it points to is treated differently: kept for a dedicated model-risk audit retention window (commonly multi-year, driven by regulatory or internal audit policy rather than operational usefulness), independent of how long the raw trace survives, since an investigation may start long after the originating trace has aged out of hot or warm storage. As long as the token and its registry record both still exist, the model side of an investigation is reproducible even if the raw trace itself is gone; only the specific input that triggered the trace would then need to come from wherever the input checksum can still be resolved (e.g. a retained, compliance-tagged input log), which is a separate retention decision from the model provenance itself.

Trade-offs and pitfalls

• Fingerprinting from an uncommitted patch (a common shortcut during rapid iteration) breaks reproducibility outright; the source fingerprint has to make an unclean deploy state visible and discouraged, not silently allowed.
• A mutable "current model version" pointer instead of a content-addressed registry ID would make historical provenance ambiguous the moment a version gets updated in place; immutability at the registry level is what makes a token minted six months ago still resolve to exactly the model that was actually running then.
• Storing only a token and checksum means reproducing an investigation depends on the dataset snapshot still being materializable from the feature store; if the feature store's own retention doesn't match the model-provenance retention window, "reproducible" investigations quietly stop being reproducible past that point, and that mismatch needs to be an explicit, reviewed policy decision, not an accident.
• The 6x span-overhead reduction from token-based indirection is worth it at this inference volume, but for a much lower-traffic model (a few inferences/minute) the full-inline approach's extra bytes barely matter, and the added indirection (a registry lookup on every investigation) may not be worth the complexity; this is a volume-dependent design choice, not a universal rule.