Data Pipeline Monitoring and Observability Questions

Direct answer

A pipeline job reporting success while producing incomplete or corrupt output, zero rows, a truncated file, is a "silent failure": the orchestrator's exit code says everything is fine because the process didn't crash, but the actual deliverable is wrong. Catching it requires monitoring the OUTPUT'S content and shape, not just the process's exit status.

Structured elaboration

• Detector 1, output row-count sanity check: compare the current run's row count against a trailing baseline (same weekday's median over the last N weeks). A run producing zero rows, or a fraction far below the baseline, fails this check even though the process exited 0.
• Detector 2, file/artifact existence and non-triviality check: after a job claims completion, verify the expected output file exists AND exceeds a minimal size threshold (a truncated file often lands at a suspiciously small byte count, well under a typical run's size).
• Detector 3, schema/shape assertion on the output: assert the output has the expected columns and that a sample of critical fields (a primary key, a required amount field) is non-null above some threshold, catching the case where the job ran but wrote garbage or an empty schema.
• Query/metric example: SELECT COUNT(*) FROM output_table WHERE partition_date = CURRENT_RUN_DATE compared against SELECT PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY row_count) FROM historical_run_counts WHERE day_of_week = EXTRACT(DOW FROM CURRENT_RUN_DATE); if the current count is under roughly 70% of the historical median for that weekday, flag it.

Worked example

Concretely: a job's orchestrator marks the DAG (directed acyclic graph) run "success" because the Spark job exited 0. But an upstream input file was accidentally empty (a source system's export job failed silently upstream of THIS pipeline), so the transform read zero input rows, wrote zero output rows, and exited cleanly because "process zero rows successfully" is not an error condition to the Spark job itself. The row-count sanity check, run automatically as a post-job step, compares today's output row count (0) to the trailing 4-week median for this weekday (roughly 2.1 million), computes that 0 is far below any reasonable tolerance band, and fires a page even though the orchestrator's own success/failure signal never triggered anything.

Trade-offs and pitfalls

The immediate automated action for a caught silent failure should be to QUARANTINE the output (mark the partition as not-ready-for-consumption, or block the downstream dependency from firing) rather than merely alerting, because a page that arrives after downstream consumers have already read the bad partition doesn't prevent the damage. The pitfall in implementing this check itself is setting the tolerance band too tight, treating any day below the exact historical median as anomalous, which produces constant false alarms on legitimately slow business days; a workable middle ground is a wide band (for example 50 to 150% of trailing median) combined with an absolute floor (row count under some small number is always suspicious regardless of the historical baseline).

Direct answer

A metric naming and tagging convention for a shared data platform needs a small, consistent grammar (a predictable name structure plus a bounded set of standard label keys) applied across every team, so that dashboards, alerts, and aggregation queries work the same way regardless of which team's pipeline emitted the metric, and so a new metric doesn't accidentally blow up cardinality by using an unbounded label.

Structured elaboration

• Naming pattern: <domain>_<component>_<measurement>_<unit>, for example pipeline_stage_duration_seconds (domain=pipeline, component=stage, measurement=duration, unit=seconds), which is both self-documenting and lets tooling programmatically parse the unit from the name, a widely-adopted convention (similar to Prometheus's own naming guidance).
• Standard label keys: a small, FIXED set of labels usable across metrics, for example pipeline, stage, team, environment, each with a bounded set of legal values (pipeline names come from a registry, not arbitrary strings), rather than allowing each team to invent their own ad hoc label keys, which fragments dashboards and makes cross-team aggregation impossible.
• Concrete examples:
  • Pipeline-stage timer: pipeline_stage_duration_seconds{pipeline="orders_etl", stage="transform", team="data_platform"}
  • Per-topic throughput: pipeline_topic_throughput_events_total{pipeline="orders_etl", topic="orders_raw", team="data_platform"}
  • Data-quality check metric: pipeline_dq_check_result{pipeline="orders_etl", check_name="null_rate_amount", team="data_platform", status="pass"}
• Preventing cardinality pitfalls: enforce that label VALUES come from a bounded, known set (pipeline names, stage names, team names, all drawn from a registry) and explicitly BAN unbounded values (a raw user id, a request id, a raw timestamp) as label values, ideally enforced by a lint or a metric-registration gate at emission time, not just documented as a guideline teams may or may not follow.

Worked example

Concretely, before this convention, three different teams independently named similar metrics etl_time, transform_duration_ms, and stage_latency_seconds, each with different label schemes (one used job, another used pipeline_name, a third used no team label at all), making it impossible to build one shared "platform-wide pipeline health" dashboard without per-metric special-casing. After adopting the shared convention, all three become pipeline_stage_duration_seconds with consistent pipeline/stage/team labels, and a single sum by (team) (rate(pipeline_stage_duration_seconds_sum[5m])) / sum by (team) (rate(pipeline_stage_duration_seconds_count[5m])) query now computes average stage duration per team across the ENTIRE platform, something that would have required three separate hand-written queries before.

Trade-offs and pitfalls

The naming convention's real payoff is exactly this cross-team aggregation capability, which only works if adoption is consistent, a convention that's documented but not enforced tends to drift as new teams onboard and either don't know about it or interpret it loosely. The pitfall to actively guard against is a label key that SEEMS bounded today but isn't guaranteed to stay that way, "environment" seems safely bounded (dev/staging/prod) until someone starts spinning up per-feature-branch ephemeral environments with unique names, so the enforcement mechanism (a registry-backed gate, not just a style guide) needs to catch drift as new, unexpected label values appear, not just at initial rollout.

Direct answer

OpenLineage/Marquez, DataHub, and Apache Atlas differ most on maturity and operational weight: OpenLineage/Marquez is a lightweight, standards-based approach focused specifically on lineage capture with a simple metadata model, DataHub is a fuller-featured metadata platform (lineage plus catalog, discovery, and governance) with a more complex deployment footprint, and Apache Atlas is the most operationally heavy, historically tied closely to the Hadoop ecosystem, with a rich but more rigid metadata/classification model.

Structured elaboration

• Metadata and lineage model: OpenLineage defines a standard, portable EVENT format (job started, job completed, with input/output datasets) that many tools can emit natively, and Marquez is the reference implementation storing and serving that data as a graph; DataHub has a broader entity model covering datasets, dashboards, pipelines, and people, with lineage as one of several first-class relationship types; Atlas has a rich, extensible TYPE SYSTEM for classification and governance tagging, with lineage captured as part of that broader classification model, which is powerful but requires more upfront modeling effort.
• Ease of instrumentation: OpenLineage's growing ecosystem of native integrations (Spark, Airflow, dbt) means many jobs can emit lineage with configuration alone, no custom code; DataHub similarly has a metadata-ingestion framework with many connectors, though standing up the metadata GRAPH and getting teams to actively use its catalog/discovery features (not just lineage) is a bigger adoption lift; Atlas typically requires more deliberate integration work and is most natural if you're already deep in a Hadoop/Hive-centric stack.
• Operational maturity/scalability: Marquez (OpenLineage's reference server) is lightweight and easy to run for lineage alone, but you may outgrow its feature set if you want a full catalog/discovery experience later; DataHub is more operationally involved to run (more moving parts: a metadata service, search index, graph store) but delivers a fuller platform in one system; Atlas has the longest operational track record in large, established Hadoop-ecosystem deployments but is a heavier system to stand up and maintain than either of the other two, and is less actively evolving relative to OpenLineage's momentum as an emerging cross-tool standard.

Worked example

For a mid-size, fast-growing analytics company, OpenLineage plus Marquez is generally the better starting recommendation: the growing native-integration ecosystem (Spark, Airflow, dbt all have OpenLineage support) means lineage capture can be stood up incrementally, tool by tool, without a large upfront platform investment, and the lighter operational footprint matches a growing company's constrained platform-engineering headcount. If, later, the organization also wants full data-catalog and discovery features (search, business glossary, access-request workflows) beyond pure lineage, migrating to or layering DataHub becomes the natural next step, and OpenLineage's standard event format is portable enough that lineage captured for Marquez isn't fully wasted work if that migration happens, since DataHub also consumes OpenLineage-format events natively.

Trade-offs and pitfalls

The recommendation hinges specifically on "fast-growing, mid-size" in the prompt, a large, already-Hadoop-centric enterprise with existing Atlas investment and dedicated platform staff might reasonably stick with Atlas rather than migrating; the "right" choice is genuinely dependent on existing stack and team size, not a universal ranking. The pitfall in this kind of comparison is treating "which tool" as the main decision when the harder, more consequential choice is often GRANULARITY (table-level versus column-level lineage) and CAPTURE METHOD (automatic instrumentation versus manual annotation), decisions that matter more for whether the lineage graph is actually trustworthy than which specific tool implements the graph.

Direct answer

A service level indicator (SLI) is the actual measured metric, for example "the fraction of daily reports available by 07:00." A service level objective (SLO) is the internal target for that indicator, for example "99% of days over a rolling 30-day window." A service level agreement (SLA) is the external, often contractual, commitment, usually set looser than the SLO to leave error-budget margin, for example "reports available by 08:00 on 95% of days, with a defined remedy if missed."

Structured elaboration

For a daily reporting pipeline that must be ready by 07:00 with completeness at or above 99.5%:

• SLI: two indicators, "minutes past 07:00 that the report was actually available" and "percentage of expected rows present at publish time."
• SLO: "99% of days, the report is available by 07:00 with completeness ≥ 99.5%," measured over a rolling 30-day window so a single bad day does not permanently break the target.
• SLA: what you promise the business, typically looser, for example "report available by 08:00 on at least 95% of days," with an escalation or credit if breached repeatedly.

Worked example

Concretely, you would instrument a job-completion timestamp event and a row-count-vs-expected-baseline comparison at publish time. Each day produces one data point: (published_at, completeness_pct). Over a rolling 30-day window, count the days where published_at is at or before 07:00 AND completeness_pct is at least 99.5. If 29 of the last 30 days met both criteria, compliance is 29/30, about 96.7%, which is BELOW the 99% target: a single bad day in a 30-day window is already an SLO violation worth investigating, since 99% of 30 days rounds up to needing all 30 to pass. That is a useful, slightly counterintuitive fact about a 99% target measured over a short window: it has essentially zero tolerance for even one bad day at that window length. Widening the look-back to a rolling 90-day window changes the picture: 89 good days out of 90 is 89/90, about 98.9%, still below 99% but closer, while 90/90 is the only way to clear 99% outright at that window length too, illustrating why teams often pick a window long enough (or a target loose enough) that the SLO can absorb an occasional bad day without every single miss becoming a declared violation.

Trade-offs and pitfalls

The common mistake is treating the SLO and SLA as the same number. If your internal SLO and external SLA are identical, you have zero error budget: any single miss is simultaneously an internal target failure and a customer-facing breach, which pushes teams toward reactive firefighting instead of using the budget deliberately (for example, deferring a risky migration until the budget has recovered). Another pitfall is defining the SLI too loosely, "the pipeline ran," instead of tying it to what the consumer actually cares about, "the data was both on time and complete," since a pipeline can run successfully and still produce an incomplete or stale report.

Direct answer

A CI-friendly test framework for your OWN monitoring, alerting, and runbooks, not the underlying pipeline, needs to inject a KNOWN synthetic condition (a fabricated metric anomaly or a canary pipeline run engineered to fail in a specific way), confirm the expected alert actually fires with the expected content, and confirm the linked runbook's steps are actually executable, all without ever paging a real, on-call human during the test itself.

Structured elaboration

• Synthetic data or metric injection: rather than waiting for a real production anomaly to test whether your alerting fires correctly, inject a synthetic data point or metric value engineered to cross a known alert threshold in an isolated, clearly-labeled test namespace or environment, so it's testable on a schedule (in CI, on every change to the alerting rules themselves) rather than only discovered to be broken during a real incident.
• Canary pipelines: a small, dedicated pipeline instance specifically built to be triggered into known failure modes on demand (a canary that can be told to fail its schema check, or to report a row count of zero) purely for the purpose of testing the monitoring and alerting wired to it, isolated from any real production pipeline.
• Verifying the expected alert fires: after injecting the synthetic condition, query the alerting system's own state (not wait for an actual page) to confirm the expected alert transitioned to a firing state, with the expected labels and content, this can run fully automated in a CI pipeline without ever routing to a real notification channel.
• Verifying runbook execution without paging real engineers: route the TEST alert to a sandboxed or dry-run notification channel (a dedicated test Slack channel, or a notification-system dry-run mode that logs what WOULD have been sent without actually sending it) rather than the real on-call rotation, and, for a runbook with any automated remediation steps, execute those steps against the SAME isolated canary/synthetic environment to confirm they behave as documented, again never touching real production infrastructure.

Worked example

Concretely, a CI job runs nightly: it triggers the canary pipeline into its "schema mismatch" failure mode, waits a bounded time, then queries the alerting system's API to confirm a SchemaDriftDetected alert is now in a firing state with the canary pipeline's identifier attached; if the alert doesn't fire within the expected window, the CI job itself fails loudly, flagging that the alert rule (or its wiring to this specific pipeline) is broken, caught by an automated test rather than discovered the next time this failure mode happens for real and nobody gets paged. The test also confirms the alert's routing metadata points to the correct dry-run test channel (not the real on-call rotation) before considering the test pipeline "safe to run repeatedly and automatically," an explicit safety check on the test infrastructure itself, not just the thing being tested.

Trade-offs and pitfalls

Explicitly verifying the test alert routes to a DRY-RUN or sandboxed channel, as its own checked precondition before the test suite is allowed to run automatically and repeatedly, is a critical safety property, without it, a bug in the test's own routing configuration could accidentally page a real on-call engineer every night for a synthetic, non-real condition, which would be a self-inflicted noise problem exactly the opposite of this framework's purpose. The pitfall in canary-based testing generally is keeping the canary pipeline's configuration in sync with the REAL pipeline's monitoring configuration as the real one evolves, a canary that tests against an outdated version of the alerting rules gives false confidence that current production alerting works, when it's actually validating a stale configuration that no longer matches reality.