Meta Staff Data Engineer Interview Preparation Guide

Direct answer

Monitoring tells you whether a known, predefined signal has crossed a threshold you already decided mattered, for example "did the job exit non-zero" or "did latency exceed 500ms." Observability is the broader capability to ask NEW questions of a system you did not anticipate, by combining logs, metrics, traces, and lineage so you can diagnose a failure mode nobody wrote an alert for. For a data pipeline specifically, monitoring answers "is something wrong," observability answers "what exactly is wrong and why."

Structured elaboration

• Monitoring is a fixed, curated set of dashboards and alerts: job success/failure, row counts, latency percentiles. It is necessary but closed-world: it only catches the failure modes someone predicted.
• Observability is the raw material (structured logs, labeled metrics, distributed traces, and lineage metadata) plus the ability to correlate them ad hoc. It matters most for the failure modes nobody wrote a check for.
• In a multi-stage pipeline (ingestion, transformation, materialization), the three signals map differently at each stage:
  • Logs carry the specific error text and row-level context (a schema-validation exception naming the offending field).
  • Metrics carry aggregate trend information (row count per partition, transformation duration per stage).
  • Traces, or lineage in a batch context, carry causal ordering: which upstream partition fed which downstream table, and in what order stages ran.

Worked example

A concrete scenario where observability catches what monitoring misses: a nightly ETL job finishes on time and reports success (monitoring is green: job succeeded, latency was normal). Two days later an analyst notices a dashboard's revenue figure is 8% low. Monitoring has nothing more to say, the job "succeeded." Observability lets you reconstruct what happened: the transformation stage's log shows a schema-validation warning (not an error) for a subset of rows where a currency field arrived as a string instead of a decimal; the metric for "rows silently coerced to null" spiked for that run but had no alert attached because nobody had anticipated that failure mode; and lineage lets you confirm exactly which downstream tables and dashboards consumed that run's output, so you know the blast radius without guessing. None of that reconstruction is possible from the pass/fail monitoring signal alone.

Trade-offs and pitfalls

Observability has a real cost: structured logging, trace/lineage instrumentation, and label cardinality all consume engineering time and storage budget. A common pitfall is over-investing in monitoring dashboards (more thresholds, more alerts) as a substitute for observability, which produces alert fatigue without actually improving diagnosability of novel failures. The two are complementary, not substitutes: monitoring should catch the failure modes you already know about cheaply, and observability should be reserved for the deep-dive when monitoring says "something" but not "what."

Observability also tells you whether you can TRUST a dataset right now (is it fresh, complete, and correctly shaped). That is a different concern from data governance, which controls WHO can use a dataset and for how long it may be retained (access policies, retention schedules, data contracts). A pipeline can be perfectly observable and still be governed poorly, or vice versa.

A CTE (WITH clause) is not necessarily materialized once and reused; several engines are free to re-evaluate a CTE at every place it is referenced (inlining it like a subquery), which can silently multiply the cost of an expensive CTE by the number of times downstream code references it, even though the query reads as if it only computed that logic once.

The rewrite

When a CTE is expensive and referenced multiple times, and the engine does not materialize it, wrapping the same logic in a single derived subquery (or an engine-specific materialization hint, where available) forces it to actually compute once:

-- fixture: 50,000 orders, 500 users, 20% refunded
CREATE TABLE orders (order_id INT, user_id INT, amount DECIMAL(10,2), status VARCHAR);
INSERT INTO orders
  SELECT i, i % 500, (i % 97) + 1.0, CASE WHEN i % 5 = 0 THEN 'refunded' ELSE 'completed' END
  FROM range(50000) t(i);

-- risk: 'completed' may be re-evaluated once per downstream reference on some engines
WITH completed AS (SELECT * FROM orders WHERE status = 'completed'),
     per_user AS (SELECT user_id, sum(amount) AS total FROM completed GROUP BY user_id)
SELECT count(*) AS n_users, sum(total) AS grand_total FROM per_user;

-- rewrite: collapse into a single pass so there is only one place the filter can be evaluated
SELECT count(*) AS n_users, sum(total) AS grand_total FROM (
  SELECT user_id, sum(amount) AS total FROM orders WHERE status = 'completed' GROUP BY user_id
) t;

Executed against a 50,000-row synthetic orders table, both forms returned the identical result (400 distinct users, a grand total of 1,959,082.00), confirming the rewrite changes nothing about correctness, only the number of times the underlying filter and scan potentially execute.

Which engines inline versus materialize

Behavior genuinely differs by engine and version, and the warehouse engines this pattern matters most for are exactly the ones worth naming. BigQuery and Snowflake do not materialize a non-recursive CTE by default, however many times it is referenced: each reference is independently re-evaluated, so a CTE referenced three times is computed (and on BigQuery, scanned and billed) three separate times unless you materialize it yourself into a temporary table or a materialized view. PostgreSQL (12 and later) defaults the other way for the multi-reference case this question is about: a CTE referenced exactly once is inlined like a subquery by default, but a CTE referenced more than once is still materialized automatically by default, and Postgres additionally lets you override either default explicitly with the MATERIALIZED or NOT MATERIALIZED keywords. Because this is genuinely engine- and version-dependent, do not assume a specific engine's current behavior transfers to a different engine or a future version of the same one without checking that engine's own documentation or query plan for the specific case.

Trade-offs and pitfalls

Collapsing everything into nested subqueries for safety sacrifices the readability CTEs exist to provide, and is unnecessary when a CTE is referenced only once, or when the engine you are on is known to materialize expensive CTEs correctly. The pragmatic approach is to write CTEs for readability by default, and only rewrite the specific one that both is expensive and is referenced more than once, once you have confirmed with that engine's EXPLAIN plan that it is genuinely being re-evaluated rather than assumed.

Direct answer

Deploy a read replica in every user-facing region and route reads to the nearest one for latency, while anchoring writes to a single-writer-per-shard model: each account or entity has one home region that owns writes for it (sharded by key, not globally centralized), and a write originating from any other region gets forwarded to that entity's home region. Reads stay fast everywhere because they never leave the local region; writes pay a forwarding cost only when they originate somewhere other than the entity's home region, which for most workloads is the minority case.

Architecture

flowchart TD
    CLIENT[Clients worldwide] --> RA[Region A read replica]
    CLIENT --> RB[Region B read replica]
    CLIENT --> RC[Region C read replica]
    RA --> FWD[Write forwarder]
    RB --> FWD
    RC --> FWD
    FWD --> LEADER[Anchor write leader, sharded by key range]
    LEADER --> CDC[CDC stream]
    CDC --> RA
    CDC --> RB
    CDC --> RC

• Read routing: DNS-based or edge-proxy latency routing sends each client to its nearest region; that region serves reads from its local replica, optionally backed by a local edge cache for hot keys to cut load further.
• Write routing: a lightweight write-forwarder in each region inspects the target entity's shard key, determines which region owns it, and forwards the write there if it isn't local; the write commits in its home region and the forwarder returns the result (or an idempotency-tracked async acknowledgment) to the originating client.
• Replication: the write's home region streams committed changes via change-data-capture (CDC) to every read replica, asynchronously; replicas apply changes in the order the CDC stream delivers them, tracking a per-record last-writer timestamp so replay order and true causal order stay consistent.

Consistency model

This design is eventually consistent for reads: a read served from Region B immediately after a write committed in Region A's home shard may not reflect that write yet, bounded by CDC replication lag rather than by any hard guarantee. That's an explicit, necessary trade for the latency goal, since making every read wait for a cross-region round-trip to confirm it has the absolute latest value would defeat the entire point of routing reads to the nearest region. Where a specific read genuinely needs to see its own very-recent write (a user immediately viewing an item they just created), the standard fix is read-your-own-writes: route that specific read to the entity's home region (or to a replica known to have caught up past a specific CDC watermark) instead of the nearest replica, rather than weakening the consistency model for every read to satisfy the rare case.

Write conflicts are structurally rare by design, because each entity has exactly one home region and therefore exactly one writer at any time; there's no multi-master merge problem to solve because there's no multi-master. The forwarding hop is the cost of ruling that problem out entirely rather than solving it after the fact.

Worked example: tracing one write end to end

Pin a concrete case: user_id=482's home region is us-east (that's where its shard's writer lives), but the client happens to be connected to the nearest edge in eu-west. The eu-west write-forwarder inspects the shard key for user_id=482, sees it belongs to us-east, and forwards the write there; assume a cross-region round trip of about 90ms for that forward, so the write is accepted and committed in us-east roughly 95ms after the client sent it (90ms network plus a small local processing cost). From there, the CDC stream carries that committed change out to every read replica asynchronously; assume that hop adds about 300ms before eu-west's local replica has applied it (its own network hop plus normal stream batching, separate from the synchronous forward that carried the write there). So the total time from write-acceptance to eu-west's replica reflecting the change is about $95\text{ms} + 300\text{ms} = 395\text{ms}$, call it roughly 400ms. A read served from eu-west's local replica 1 second (1,000ms) after the write was accepted already reflects it, comfortably past the ~400ms it takes to land; a read served only 50ms after acceptance would not yet reflect it, since 50ms is well inside that ~400ms propagation window, which is exactly the read-your-own-writes gap the design accepts and works around by routing that specific kind of read to the home region instead.

Trade-offs & pitfalls

The biggest latency cost this design accepts is on writes that originate far from an entity's home region: a user in Region C writing to an entity whose home shard is in Region A pays a full cross-region round trip for that write, even though every other user's reads and most other writes stay fast. If write locality doesn't naturally match user geography (an entity created in one region gets written to mostly by users somewhere else over time), this cost compounds instead of amortizing away, which is worth checking against real traffic patterns before committing to a static shard-to-region mapping. Replication lag is the other pitfall: CDC-based replication is asynchronous by nature, so a region that falls behind (network partition, replica overload) serves increasingly stale reads without necessarily surfacing an error, which is why lag needs to be an actively monitored metric with alerting, not just an assumed-small property of the pipeline. Finally, resist the temptation to solve the "occasional write from any region" requirement with full multi-master writes accepted locally everywhere; that reintroduces exactly the conflict-resolution complexity (concurrent writes to the same entity from two regions, needing merge logic or last-write-wins with its own correctness risks) that single-writer-per-shard was specifically chosen to avoid, in exchange for a write-latency win that the stated 90-percent-read workload doesn't actually need.

Direct answer
Proving blameless postmortems causally reduce recurrence, rather than assuming it, requires comparing incident outcomes between groups that did and did not receive the full blameless-postmortem treatment, while controlling for the fact that incidents are relatively rare, which makes a small, underpowered comparison unreliable.

Structured elaboration

• Define the primary metric precisely. Incident recurrence rate for the same or closely related failure category within a defined window (say, six months) after a postmortem, and mean-time-to-recovery for any recurrence that does happen, are both reasonable primary outcomes.
• Form comparison groups given low incident frequency. A staggered rollout across teams (some teams adopt full blameless postmortems now, others adopt a few months later) gives you a natural comparison without denying anyone the practice indefinitely, and it's more feasible than a strict randomized controlled trial in most organizations. Alternatively, compare incident classes that received a full postmortem against similar-severity incident classes from before the practice was adopted, using the organization's own history as the comparison.
• Account for low frequency directly. Because a single team's incident count is small, aggregate across many teams or many incident categories to get enough statistical power, and be honest that with genuinely rare, high-severity incidents, you may only be able to speak confidently about a proxy (like recurrence of the underlying vulnerability class in code review or testing) rather than recurrence of an actual outage.
• Control for confounders explicitly. Teams that adopt blameless postmortems early are often also the teams already investing more broadly in reliability practices, so any observed improvement could be due to that general investment rather than the postmortem practice specifically; a staggered rollout across otherwise-similar teams helps isolate this, and tracking a secondary metric less directly tied to postmortems (like general code quality trends) as a check helps rule out a confound affecting everything at once.
• Report the honest limitation. Even a well-designed study in this space will likely have wide confidence intervals given how rare severe incidents genuinely are; report that uncertainty rather than overstating confidence in a clean causal result.

Worked example
An organization with 40 teams rolls out mandatory blameless postmortems to half the teams (chosen to be broadly similar in size and incident history) starting this quarter, with the other half adopting the practice three months later. Primary metric: recurrence rate of a related incident category within six months of any postmortem-eligible incident. After the study window, teams in the early-adoption group show a lower recurrence rate than the later-adoption group during the period before the second group adopted the practice, and the gap narrows once the second group also adopts it, which is more convincing evidence of a causal effect than a simple before-and-after comparison on a single group would have been, since it rules out a general org-wide trend as the sole explanation.

Trade-offs and pitfalls
The most common mistake is treating a simple before-and-after comparison on one group as proof of causation, when it's equally consistent with unrelated organizational improvements happening over the same period. A second is understating how much statistical power genuinely rare, severe incidents cost you, and presenting a result with far more confidence than the small sample size actually supports.

Direct answer

A circuit breaker wraps a downstream call (an API request from a data pipeline) with a state machine that tracks that dependency's recent health and stops calling it entirely once it looks broken, rather than letting every caller keep retrying into a known-failing dependency. It has three states: CLOSED (normal, calls pass through), OPEN (the dependency is considered failing, calls are rejected immediately without even attempting the network call), and HALF-OPEN (a cooldown has elapsed, a small number of probe calls are allowed through to test recovery). This protects both the caller (no more time wasted waiting on doomed calls) and the struggling dependency (no continued load from callers who cannot succeed anyway).

Structured elaboration

Failure threshold. The circuit opens once a configured fraction of recent calls fail (e.g., more than 50% of the last 20 calls, or more than N consecutive failures), not on the first single failure, since a single transient blip should not trip a breaker meant to catch SUSTAINED trouble; the exact threshold trades false-positive risk (opening on normal, isolated hiccups) against false-negative risk (staying closed too long into a genuine outage, still sending traffic that will fail).

Cooldown window. Once open, the circuit stays open for a fixed cooldown period before allowing any probe calls, giving the struggling dependency time to recover WITHOUT continued load from this caller during that window. Too short a cooldown re-opens the circuit into a still-broken dependency repeatedly (thrashing); too long delays recovery detection once the dependency IS actually healthy again.

Interaction with retry/backoff. The circuit breaker and retry-with-backoff operate at different granularities and are complementary, not redundant: backoff governs how AGGRESSIVELY a single caller retries an individual failed call, while the circuit breaker governs whether to attempt the call AT ALL, given the dependency's recent aggregate health. A well-designed system checks the circuit breaker state FIRST (fail fast if open, skip backoff entirely) and only applies backoff-and-retry logic for calls that proceed because the circuit is closed or half-open.

Interaction with backpressure. When the circuit is open, calls that would have gone to the failing dependency are rejected immediately rather than queued indefinitely; this is itself a form of backpressure, signaling upstream (the pipeline stage feeding this call) to either buffer (write to a durable queue), drop, or reroute, rather than accumulating unbounded in-flight work waiting on a dependency that will not respond in time anyway.

Worked example

A pipeline calls a downstream enrichment API at 2,000 requests/sec. The dependency begins failing at 80% of requests (a partial but severe degradation). With a threshold of "open if more than 50% of the last 20 calls failed": within roughly $20 / 2{,}000 = 0.01$ seconds of the degradation beginning (the time to accumulate 20 calls at this rate), the failure ratio crosses 50%, and the circuit opens. From that point, roughly:

worth of requests per second of cooldown are rejected immediately (fast, cheap rejections) instead of each attempting a doomed network call and waiting for its own timeout; at a 30-second cooldown, this is 60,000 requests that would otherwise have each paid a network round-trip's worth of latency waiting to fail, now failing in microseconds instead. After the cooldown, a small number of half-open probe calls (not all 2,000/sec resuming at once) test whether the dependency has recovered; if they succeed, the circuit closes and full traffic resumes; if they still fail, the circuit reopens for another cooldown period.

Trade-offs and pitfalls

• Common mistake: opening on the first failure. This makes the breaker indistinguishable from "stop on any error," far too aggressive for a dependency that has occasional, normal transient blips; the threshold should reflect SUSTAINED degradation, not any single failure.
• Common mistake: resuming full traffic immediately after cooldown instead of a small half-open probe. Sending all 2,000 requests/sec back at once the instant cooldown ends risks immediately re-triggering the same overload/failure condition if the dependency has not FULLY recovered, undoing the cooldown's benefit in the first probe cycle.
• A circuit breaker without any coordination with backpressure just moves the problem, not solves it, per the worked example: rejected calls still need somewhere to go (buffer, drop, or reroute), a design decision separate from the circuit breaker itself.
• Per-dependency circuit state, not a single global breaker, is essential once a pipeline calls MULTIPLE downstream dependencies; a struggling dependency should not trip a breaker that also blocks calls to healthy, unrelated dependencies.

Role: On-device analytics reduces raw telemetry, preserves privacy, and enables low-latency personalization and local quality metrics without centralized raw data transfer.
Signals best processed on-device:

• Sensitive personal signals (listening history, keystrokes, health metrics): aggregate locally and send only anonymized or differential-private summaries.
• Low-latency personalization features (recommendation embeddings, caching decisions): compute locally to reduce latency and bandwidth.
• Quality telemetry tied to UX (local crash logs, sensor diagnostics): pre-processed on-device to filter noise and redact PII before upload.
  Signals better centralized:
• Cross-user aggregates (global popularity trends, training data for models), heavy-weight model training, and cross-device deduplication requiring many users' data.
  Why: on-device processing minimizes privacy risk and bandwidth, reduces structural latency, and enables personalization without exposing raw data. Central servers are necessary for global learning, model consolidation, and business analytics that require population-level views.

Selection criteria — I'll evaluate each provider against these weighted factors:

• Managed service availability (25%): native data services (managed Spark/EMR, serverless SQL, streaming, object storage, data warehouse) and maturity of data ecosystem.
• Total cost of ownership (20%): compute, storage, networking, license, ops, staff productivity; include reserved/spot pricing and autoscaling benefits.
• Egress and network behavior (15%): inter-region and cross-cloud egress costs, VPC peering, private connectivity (Direct Connect/ExpressRoute/Interconnect), latency for regional consumers.
• Regional coverage & resiliency (10%): presence in required regions, multi-AZ/region DR options, compliance with data residency.
• Vendor lock-in risk (10%): ease of migrating workloads (open-source compatibility, containerization, standard formats like Parquet), managed service portability.
• Compliance & security (10%): certifications (SOC2, ISO, HIPAA, GDPR), KMS, IAM granularity, logging/audit capabilities.
• Existing team expertise (10%): familiarity with provider SDKs, CLI, managed services — impacts ramp time.

PoC plan to evaluate 2–3 providers (e.g., AWS, GCP, Azure):

1. Scope & success metrics (day 0): ingest 1 TB/day, run a nightly 2 TB ETL (Spark), serve queries with sub-2s latency, cost target X.
2. Minimal reference architecture (3–4 days per provider): set up secure VPC, object store, managed Spark or Dataproc/EMR/Synapse, warehouse (Redshift/BigQuery/Synapse), and streaming (Kinesis/PubSub/EventHub).
3. Implement test workloads (1 week): ingest synthetic data, run ETL, run 10 representative BI queries, measure throughput, latency, failure modes.
4. Measure & record: performance, provisioning time, operational effort, monitoring maturity, security setup time, egress behavior with cross-region transfer.
5. Cost modeling: run the workload for 7 days, capture bill, extrapolate TCO for 12/36 months with different growth scenarios.
6. Risk assessment: evaluate lock-in vectors, exportability of data, tooling compatibility.
7. Demo & recommendation (2 days): present findings with quantitative scorecard, recommend provider(s) plus migration/mitigation plan.

This approach balances technical validation, cost realism, and team ramp considerations to make a defensible choice.

Situation: Designing storage strategies for a petabyte-scale Delta Lake containing event-level telemetry used by analysts and ML teams.

Partition keys:

• Choose coarse-grained, high-cardinality-safe keys. Primary: ingestion_date (yyyy-MM-dd or yyyy-MM) to scope most time-based queries and retention. Secondary: event_type or topic when a small number (~10–50) of stable categories heavily drive queries.
• Avoid high-cardinality user_id/session_id as partition keys — causes many small files and metadata explosion.
• Use multi-level partitioning: yyyy/MM for month-based backfill and yyyy/MM/dd for recent hot data (or dynamic partitioning where tail is daily, older data monthly).

Clustering (Z-order / data skipping):

• Use Z-order on columns frequently filtered together (e.g., user_id hashed bucket, event_type, timestamp) to colocate related rows within files and improve data skipping.
• Consider Spark bucketing or hash partitioning for join keys used by ML (user_id) combined with Z-order on timestamp.

File sizes:

• Target Parquet file size ~256–512 MB (read-optimized) for HDFS/S3; 128–256 MB if many small queries. This balances IO throughput and parallelism; avoids too many small files.

Compaction cadence:

• Nearline compaction for recent data: run small-file compaction hourly or every few hours to merge micro-batches into target file size (streaming ingestion).
• Weekly full compaction on older partitions (monthly partitions) and re-Z-ordering for cold data.
• Trigger compaction based on thresholds: number of files in partition > N (e.g., >100), average file size < target/2.

Measuring impact:

• Track query latency (P99/P50) and scan bytes per query before/after.
• Monitor number of files per partition, average file size, and Delta table transaction log size.
• Measure job runtimes for common ETL and ML training workloads and cluster CPU utilization and shuffle/read throughput.
• Use cost metrics (S3 egress, EMR/Azure Databricks compute hours) and SLA adherence for data freshness.
• A/B test: run queries against compacted vs non-compacted snapshots and compare read bytes, query time, and success rates.

Trade-offs:

• More aggressive compaction reduces read overhead but increases write/compute cost and latency for freshness.
• Z-order improves selective reads but is expensive at rewrite; schedule during low-usage windows.

This strategy balances query performance, metadata scalability, and operational cost while keeping fresh data queryable and ML pipelines efficient.

Direct answer

A written decision record needs enough context, alternatives, and reasoning that someone with none of the meeting-room context can reconstruct why the choice was made, not just what was chosen. For a decision I expect to be reversed, I keep the record deliberately lighter and add one thing most templates skip: an explicit trigger for when to revisit it, so it doesn't quietly calcify into a permanent decision nobody re-examines.

The sections I insist on

• Context: the problem and constraints as they were understood at the time, written for someone who wasn't there. This is the section people skip and the one that makes the record useless six months later without it.
• Decision: one clear, unambiguous statement of what was chosen.
• Alternatives considered, with why they were rejected: not just a list of options, but the specific reason each one lost, so a future reader doesn't have to independently re-litigate an option that was already ruled out for a reason still worth knowing.
• Consequences and known trade-offs: what got worse to get this benefit, stated honestly.
• Owner and reviewers: who's accountable for the decision and who signed off, since "who do I ask" is usually the first question a future reader has.
• Status and review date: whether it's active, superseded, or up for reconsideration, and when.

What changes for a decision I expect to be reversed

I keep it short (a couple hundred words, not a multi-page document) because a heavy record for a decision I already expect to revisit is wasted effort and discourages anyone from actually updating it later. I set an explicit review trigger up front, either a date or a concrete condition ("revisit once traffic on the new path exceeds X" or "revisit at next quarter's planning"), rather than leaving "temporary" as an implicit, unenforced label. And I mark the status plainly as provisional in the document itself, so a future reader doesn't mistake a stopgap for a considered, permanent architecture choice, which is exactly how temporary decisions quietly outlive their justification.

Making sure the record actually gets used

Writing the document isn't the same as the decision being understood. I circulate it to the teams it affects before treating it as final, not just after, and I link it from the code or infrastructure it governs so someone hits it while doing the work, not only while searching a wiki. For an organization with no existing practice, I'd introduce it on the highest-friction decision available (the one that generated the most Slack debate recently), because a record that resolves a real, felt disagreement earns buy-in faster than a template introduced in the abstract ever does.

Worked example

A team needed to decide whether a new internal feature flag should default to on or off during rollout, a decision genuinely likely to be revisited within weeks as usage data came in. The record was short: two to three hundred words covering the specific rollout risk, the two options (default-on with a kill switch vs. default-off with opt-in), the choice (default-off, opt-in, given the blast radius of the affected workflow), and an explicit trigger to revisit once a defined fraction of eligible users had opted in or two weeks had passed, whichever came first. That review date is what separated it from silently becoming the permanent behavior nobody remembered to reconsider.

Trade-offs and pitfalls

• Writing the context section as if the reader shares your assumptions. The point of the record is that they don't; skipping context is the single most common way these documents fail their actual purpose.
• Listing alternatives without saying why they lost. A bare list invites a future reader to re-argue an option that was already ruled out, wasting the exact effort the record was meant to save.
• No review trigger on a decision meant to be temporary. Without one, "temporary" decisions are the ones most likely to still be running in production two years later, unexamined.
• Treating the record as done once it's written. A decision record nobody reads or links to provides none of its intended value; distribution is part of the job, not an afterthought.

A latency budget is an allocation of the total allowable end-to-end latency among components of a system so each stage knows its target and owners can design, monitor, and trade off performance vs. cost and reliability.

Suggested per-stage budgets for a 2s SLA (API end-to-end):

• Event ingestion: 300 ms — include network hop, broker enqueue, acknowledgement.
• Transformation pipeline: 1,100 ms — includes queuing, batch/window wait, processing (e.g., Spark/Flink job).
• Query/serve layer: 600 ms — cache/DB read, serialization, and API handler.

Enforcement mechanisms:

• Timeouts: set per-stage timeouts slightly above budgeted mean (e.g., ingestion timeout 350 ms) to fail fast and avoid cascading waits.
• Retries: use limited, exponential-backoff retries where idempotency is guaranteed; avoid retries in synchronous critical path that blow past SLA.
• Backpressure & circuit breakers: when pipeline lags, apply backpressure or degrade features to keep core API within SLA.
• SLIs/SLOs & monitoring: define SLIs per stage (p99 latency, success rate). Example SLOs: p95 < budget, error rate < 0.5%. Alert when burn rate exceeds threshold.
• Observability: distributed traces, metrics (latency histograms), and dashboards to attribute latency to stages.

Trade-offs:

• Strict budgets improve predictability but increase cost (more headroom, faster infra) and reduce fault tolerance if you fail fast without graceful degradation.
• Looser budgets allow retries and more tolerant processing (e.g., eventual consistency) but risk violating SLA spikes and harder root-cause isolation.
• Practical approach: enforce strict budgets on the critical sync path (ingest + serve), push noncritical heavy work to async pipelines, use adaptive degradation (cache staleness, feature flags) to preserve core SLA while maintaining fault tolerance.

This split aligns owners to measurable SLIs, enables clear alerts, and supports architectural choices (sync vs async, batching, caching) to meet the 2s API requirement.