Lyft Site Reliability Engineer (Mid-Level) Interview Preparation Guide

Direct answer

Default to additive-only changes (new optional fields with sane defaults), never silently repurpose an existing field's name or meaning, and when the meaning genuinely has to change, introduce it as a new versioned field and dual-emit both the old and new during a defined deprecation window so every consumer (dashboards, alerts, downstream jobs) has time to migrate before the old one disappears.

Structured elaboration

Additive changes are the default and the cheap case
Adding a brand-new field with a sensible default (or simply absent, if consumers already tolerate unknown fields) is safe: existing dashboards and alerts that don't reference it are unaffected, and new tooling can start using it immediately. Most schema evolution should fit this case; if it doesn't, that's a signal the change is more than "add a field."

Never repurpose a field in place
Changing what an existing field means (e.g., a latency field that used to be measured in milliseconds and is now measured in microseconds, keeping the same name) is the most dangerous kind of change, because it fails silently: old dashboards keep running the same query and now show numbers that are wrong by a constant factor, with no error to alert anyone. A rename or unit change should always get a new field name (latency_ms retired in favor of latency_us, both emitted for a transition period), never an in-place redefinition.

Version the schema explicitly
Tag every emitted record with a schema_version. Consumers that need to branch on shape (a downstream parser, a strict dashboard query) can check the version rather than guessing from field presence. This also gives you a clean place to document exactly which version introduced which change.

Deprecation as a process, not an event

1. Announce the field's replacement and the planned sunset date.
2. Dual-emit: write both the old and new field for a fixed window.
3. Track actual usage of the old field (query logs, dashboard/alert definitions referencing it) to confirm consumers have migrated, not just assume they have.
4. Only stop emitting the old field once usage has genuinely dropped to zero (or the sunset date passes and remaining consumers have been explicitly notified they'll break).

Testing the transition
Contract tests (automated checks that a producer's output still satisfies what a known consumer expects) and shadow validation (running the old and new emission side by side and diffing the derived metrics they produce) catch the case where the "safe" additive change turns out to interact badly with an existing aggregation, before it reaches production dashboards.

Worked example

A service currently emits {"latency": 245, ...} where latency is milliseconds, and the team wants to switch to microsecond precision.

Wrong approach (in-place redefinition): change the emitter to write {"latency": 245000, ...} under the same field name. A dashboard panel computing avg(latency) over the last hour now silently reports a number 1000x larger with zero errors or warnings; anyone glancing at the dashboard sees "avg latency: 245000ms" and either panics or, worse, doesn't notice because the panel has no sanity bound configured.

Correct approach: add latency_us alongside the existing latency field, dual-emit both for a stated transition window (e.g., until every dashboard query referencing latency has been rewritten to use latency_us, confirmed by grepping the dashboard/alert config repository for the old field name), then drop latency only after that grep returns zero references.

The key diagnostic in this example: the failure mode is not "the pipeline throws an error," it's "the pipeline keeps running and produces a wrong number that looks plausible." That's why additive-with-a-new-name is the default, not an optional extra step.

Trade-offs & pitfalls

• Dual-emitting indefinitely accumulates cost and confusion; every deprecation needs an explicit sunset date, not an open-ended "eventually."
• Tracking actual field usage (rather than assuming consumers migrated because you announced it) is the step most teams skip, and it's exactly the step that prevents a surprise outage when the old field is finally dropped.
• Additive changes still need CI-enforced schema compatibility checks (backward/forward compatibility validation), because "just add a field" can still break a strict consumer that rejects unknown fields.
• A silent semantic change is strictly worse than a loud break: a query that errors gets noticed and fixed; a query that keeps returning a plausible-looking wrong number can go unnoticed for months.

Different system shapes need genuinely different SLIs, because "is it working" means something different for a synchronous API than for a background queue or a streaming pipeline.

Structured elaboration

Eight representative SLIs across common system types: (1) HTTP API - proportion of requests returning within a latency threshold and a non-error status, measured continuously; pitfall: sampling only a subset of endpoints can hide a slow, low-traffic endpoint that matters disproportionately to a few customers. (2) HTTP API (errors) - proportion of non-5xx responses, measured continuously; pitfall: conflating client errors (4xx) with server errors (5xx) hides real backend degradation behind legitimate bad-request noise. (3) Background job queue (throughput) - proportion of jobs completed within their expected SLA time, measured per batch or per hour; pitfall: averaging completion time across job types of very different sizes masks a specific job type silently regressing. (4) Background job queue (correctness) - proportion of jobs that complete without needing manual reprocessing, measured daily; pitfall: silent partial failures that don't raise an exception can go completely uncounted. (5) Database (query latency) - proportion of queries under a latency threshold, measured continuously; pitfall: mixing OLTP point-queries with occasional heavy analytical queries in the same SLI badly skews the distribution. (6) Database (replication lag) - proportion of time replication lag stays under an acceptable bound, measured continuously; pitfall: a brief lag spike during a backup job can look identical to a real problem without additional context. (7) Streaming service (freshness) - proportion of events processed within an acceptable end-to-end latency, measured per window; pitfall: high-cardinality per-partition tracking can blow up metric storage cost. (8) ML inference (latency + validity) - proportion of predictions returned within a latency bound AND passing a basic sanity/schema check, measured continuously; pitfall: tracking only latency and ignoring prediction validity means a service can look "up" while silently returning garbage.

Worked example

Taking SLI #1 (HTTP API request success): numerator = count of requests where status is not 5xx AND latency <= 300ms; denominator = total requests to that endpoint; measurement window = 5-minute rolling for alerting, rolled up to 30-day rolling for the SLO target. If in a given 5-minute window there were 12,000 requests and 11,880 met both conditions, the SLI for that window is $11880/12000 = 0.99$, i.e. 99.0%.

Trade-offs and pitfalls

The common thread across all eight is that a naive, generic "is it erroring" SLI is rarely the right one; each system type has a specific failure mode (partial job failure, replication lag, prediction validity) that a purely availability-shaped SLI would miss entirely. High-cardinality measurement (per-partition, per-customer) gives better diagnostic granularity but at real storage and query cost, so most teams track a coarse aggregate SLI for the SLO itself and reserve high-cardinality breakdowns for debugging once the aggregate signals a problem.

Direct answer

A message queue absorbing write spikes ahead of a relational database is a genuinely good pattern, but "just add a queue" is not a complete design: it trades one problem (the database can't take the burst) for a set of operational pitfalls the direct-write path never had to think about, namely ordering, transactional boundaries between the queue and the database, schema evolution across producers and consumers, unbounded queue growth, backpressure propagation, and safe recovery after an outage. Every one of those has a real mitigation, but each mitigation adds its own complexity and cost, so the honest framing in an interview is "here is what this buys you and here is what you now own," not "queues solve write spikes."

The six operational pitfalls

Getting to effectively-once processing over an at-least-once queue

Most queues guarantee at-least-once delivery, meaning a message can be redelivered (after a consumer crash mid-processing, for example) and the consumer must be built to tolerate that. Effectively-once processing, where a redelivered message doesn't produce a duplicate effect even though the queue itself only promises at-least-once, comes from three pieces working together: an idempotency key attached to each message (a stable identifier for "this exact logical write," not the queue's own delivery ID, which changes on redelivery), a dedup store that records which idempotency keys have already been processed so a redelivered message is recognized and skipped, and the same transactional outbox pattern from the table above, so the message that could be redelivered and the database write it corresponds to were never two independently-failing operations in the first place.

Choosing delivery semantics per job type in a background-job system

Not every job needs the same guarantee, and treating all jobs as if they did either over-invests in low-stakes jobs or under-invests in high-stakes ones:

Deduplication implementation choices at high throughput

Whichever approach is chosen, the dedup store's own growth needs active management: an unbounded exact store retaining every key forever eventually becomes its own capacity problem, so a TTL tied to the maximum realistic redelivery window (not "forever"), periodic compaction, or tiering older entries to cheaper storage are all standard ways to keep the dedup store itself from becoming the next bottleneck.

Worked example: two numbers that motivate bounding the dedup store and partitioning by key

Partitioning for per-key ordering at scale. Getting exactly-once, per-key-ordered processing at high throughput typically means routing to thousands of partitions rather than a handful, since per-key ordering forces every message for a given key onto one partition. As an illustrative assumption, not measured data, suppose the target throughput is 2,000,000 messages/sec and a single partition can sustain roughly 500 messages/sec while preserving strict per-key order within it:

$\text{partitions needed} = \frac{2{,}000{,}000}{500} = 4{,}000 \text{ partitions}$

This is the throughput-versus-latency trade-off in concrete terms: adding partitions scales aggregate throughput almost linearly, but any single key is still pinned to one partition for ordering, so that key's messages are bounded by that one partition's 500 messages/sec, no matter how many of the other 3,999 partitions have spare capacity. A downstream stateful processor consuming a partition also needs exactly-once handling of its own state mutations (typically by committing its offset and its state update together, atomically), and that per-partition coordination cost is part of why partition count is a real tuning decision, not "more is always better."

Why the dedup store needs a bound. As a further illustrative assumption, suppose every one of those 2,000,000 messages/sec needs a dedup entry retained for a 24-hour redelivery window, and each entry costs roughly 100 bytes:

$\text{events/day} = 2{,}000{,}000 \text{ /s} \times 86{,}400\text{s} = 172{,}800{,}000{,}000 \text{ events}$

$\text{dedup store size} = 172{,}800{,}000{,}000 \times 100\text{ bytes} = 17{,}280{,}000{,}000{,}000\text{ bytes} \approx 17.28\text{ TB}$

At this throughput, an unbounded exact dedup store retaining a full day of keys grows into a multi-terabyte system on its own, which is exactly why compaction, a shorter TTL calibrated to the actual maximum redelivery delay rather than a full day, or a bloom filter pre-filter in front of a smaller exact store are standard choices at this scale rather than optional polish.

Trade-offs and pitfalls

• Treating the queue as having solved the write-spike problem once it's in place, without addressing any of the six pitfalls above; the queue moved the bottleneck, it did not remove the underlying complexity.
• Reaching for full effectively-once machinery (idempotency key, dedup store, outbox) for every job type uniformly, which over-invests engineering effort in jobs where an occasional duplicate or dropped event genuinely does not matter.
• Choosing a bloom filter for deduplication without accounting for its false-positive rate, which silently drops legitimate new messages rather than failing loudly; that trade-off needs to be a deliberate choice, not a surprise discovered in production.
• Partitioning aggressively for throughput without recognizing that per-key ordering caps any single key's throughput at one partition's capacity; more partitions help aggregate throughput and hot-key isolation, not a single hot key's own ceiling.
• Skipping a rate-limited, tested recovery drill for the outage-recovery path; the first time a large backlog gets replayed should not be during a real outage.

Direct answer
Limiting a canary's blast radius to a single region behind a global load balancer means routing based on BOTH region AND canary assignment together, so users in the target region get split between canary and stable while every other region stays entirely on stable, with session affinity handled so a user doesn't flip between versions mid-session, and a cross-region failover path that doesn't accidentally expose the canary to a region it was never meant to reach.

Structured elaboration

1. Region-scoped canary: configure the global load balancer's routing so only requests already destined for the target region are further split by the canary weighting; requests to every other region bypass the canary logic entirely and go straight to stable, keeping the blast radius genuinely contained to one region's traffic.
2. Session affinity: within the target region, use a stable hash of the user's identity (not a random per-request choice) to decide canary-vs-stable, so once a user lands on the canary, they consistently stay there for the DURATION of their session rather than flip-flopping between versions on each request, which would both confuse metrics and give users an inconsistent experience.
3. Cross-region failover: if the target region fails over to another region (a genuine regional outage, unrelated to the canary itself), the failover target region needs to know NOT to apply the canary split, since the canary was only meant to affect that one specific region's traffic; failing over should route everyone, including the canary cohort, to STABLE in the failover-target region, rather than accidentally expanding canary exposure to a region it was never validated in.
4. Metrics scoped to the region: canary-vs-stable comparison metrics need to be filtered to the target region specifically, since aggregating in metrics from unaffected regions (which are 100% on stable) would dilute or distort the comparison.

Worked example

flowchart TB
  GLB[Global Load Balancer] -->|region=US-target| Split[Canary/Stable split, 10/90]
  GLB -->|region=EU| Stable_EU[100% stable]
  GLB -->|region=APAC| Stable_APAC[100% stable]
  Split --> Canary_US[Canary, US only]
  Split --> Stable_US[Stable, US]
  Canary_US -->|failover| Stable_EU

A user in the target region hashed into the canary cohort stays on canary consistently across their session (via the stable-hash session affinity); if that region experiences an unrelated outage and traffic fails over to the EU region, the failover path routes explicitly to EU's STABLE tier, not attempting to preserve the canary assignment across a region boundary it was never validated for.

Trade-offs and pitfalls
The specific risk this design guards against is a REGIONAL FAILOVER accidentally becoming a canary-exposure EXPANSION, silently putting canary-cohort users onto a fresh region where the canary was never tested against that region's specific infrastructure, traffic patterns, or configuration; the common mistake is a failover mechanism built independently of the canary logic that doesn't know to override the canary assignment during a cross-region failover event.

Direct answer

For a boolean flag (or a small set of category memberships) tracked across millions of entities, a bitset (one bit per entity per flag, packed into a flat array of machine words) beats a hash set of member IDs on both memory and multi-flag query cost whenever a meaningful fraction of entities actually carry the flag. A hash set only wins when the flag is genuinely sparse, since its memory scales with the number of members, not the population size.

Structured elaboration

Bitset representation: N entities need $N/8$ bytes flat (1 bit per entity), addressed directly by entity index. "Has flag A and flag B" becomes a single bitwise AND across the two byte arrays, one machine word at a time (e.g. 64 bits = 64 entities per instruction) - the work is O(N) bits total, but with a tiny constant factor, since it's pure word-parallel ALU work with no hashing and no pointer chasing.

Hash set representation: only the entities WITH the flag are stored, each as (say) an 8-byte id plus hashing/bucket overhead; a reasonably tuned open-addressing set needs load-factor slack and per-slot metadata on top of the raw id, so assume roughly 24 bytes per entry all-in as an illustrative estimate. Total memory is then $k \cdot b$ where k is the member count and b the per-entry byte cost, independent of N. "A and B" becomes an intersection of two hash sets: O(min(|A|, |B|)) hash lookups into the larger set - cheap when both sets are small, but each lookup costs a hash computation and a probe, not a single ALU instruction.

Break-even point:

Sparse middle ground: compressed bitmap formats (such as Roaring bitmaps) adaptively switch between array, bitmap, and run-length containers per chunk of the index range, approaching hash-set-sized memory when a flag is sparse and raw-bitset speed when it's dense, at the cost of extra implementation complexity and a small per-operation overhead versus a flat bitset.

The same idea, one level down (folding the game-engine collision-layer case): a per-entity collision-layer bitmask (up to 64 layers) is the identical idea with the axes swapped. Instead of one bitmap spanning the whole population for each flag, each individual entity carries its own fixed-width integer, where a single 64-bit word covers up to 64 category bits, and two entities' membership overlap becomes one AND across those two words, rather than a hash lookup on either side.

Worked example

Plugging concrete numbers into the break-even formula above:

N = 100_000_000       # total entities
b = 24                 # assumed bytes per hash-set entry (8-byte id + ~16 bytes overhead)

bitset_bytes = N / 8
breakeven_k = bitset_bytes / b
breakeven_fraction = breakeven_k / N

print(f"bitset size: {bitset_bytes:,.0f} bytes")
print(f"breakeven k (members): {breakeven_k:,.0f}")
print(f"breakeven fraction of N: {breakeven_fraction:.6f} ({breakeven_fraction*100:.4f}%)")

Output:

bitset size: 12,500,000 bytes
breakeven k (members): 520,833
breakeven fraction of N: 0.005208 (0.5208%)

So with these assumptions, a hash set is smaller only while fewer than about 0.52% of the 100 million entities carry the flag; past that density, the flat bitset is both smaller AND cheaper to query.

The per-entity collision-mask version of the same idea:

LAYER_PLAYER = 1 << 0
LAYER_ENEMY = 1 << 1
LAYER_TERRAIN = 1 << 2

entity_mask = LAYER_PLAYER | LAYER_TERRAIN
other_mask = LAYER_ENEMY | LAYER_TERRAIN

collides = (entity_mask & other_mask) != 0  # shares the "terrain" layer bit
print("collision check (shared layer bit set):", collides)

Output:

collision check (shared layer bit set): True

Whether the "many entities x few flags" bitset lives on the population axis (feature flags) or the "one entity x many categories" bitmask lives on the per-object axis (collision layers), the payoff is the same O(1) word-parallel membership or overlap test instead of a hash lookup.

Trade-offs & pitfalls

• A bitset needs entities to have small, dense, stable integer indices; if entity IDs are sparse (e.g. UUIDs), you need an id-to-dense-index table regardless, which itself costs memory a hash set skips.
• Bitset updates (flip a flag) are O(1) but require already knowing the entity's index; hash set add/remove is also O(1) average, with a higher constant cost from hashing and resize amortization.
• Iterating "give me every entity with flag A" requires a bit-scan over the bitset (fast, but not free); a hash set hands you the member list directly at no extra cost.
• Reaching for a hash set out of habit on a flag that is actually dense (say 40% of entities) wastes memory relative to a bitset AND makes multi-flag boolean queries meaningfully slower.
• Compressed bitmaps trade a little per-operation overhead for adapting automatically across the sparse/dense spectrum, which pays off once you have many flags of very different densities rather than hand-picking bitset vs. hash set per flag.

Direct answer

By default, DynamoDB's GetItem and Query calls perform eventually consistent reads, which can be served from a replica that hasn't yet applied the most recent write, so a read immediately after a write can come back stale or missing. The fix is either to request a strongly consistent read on that specific call, or to redesign the flow so the downstream step doesn't need to re-read a value it just wrote.

Structured elaboration

Why it happens: DynamoDB replicates every write across multiple storage nodes in the region before acknowledging the write as successful, but a default GetItem/Query can be routed to a replica that hasn't received that write yet. This is a deliberate cost and latency trade-off, not a bug: eventually consistent reads use half the read capacity and typically have lower latency than strongly consistent reads.

Your options as the operator:

1. Pass ConsistentRead: true on the specific read that needs the fresh value. It costs twice the read capacity units of an eventually consistent read and only works within the same region, but it guarantees you see every write that was acknowledged before the read started.
2. Avoid the read-after-write pattern entirely: have the writer hand the value it just wrote directly to the downstream step instead of making it re-read from the table.
3. If a separate process genuinely has to re-read, add a short retry with backoff, since same-region replication lag is typically single-digit milliseconds.

Where the ConsistentRead flag has limits: DynamoDB Global Tables (cross-region replicas) replicate asynchronously, so a strongly consistent read in one region's replica still only guarantees you see every write already acknowledged in that region, never a write still in flight from another region; there is no cross-region strong-consistency option. DynamoDB Accelerator (DAX, an in-memory cache in front of DynamoDB) is a different case: requesting ConsistentRead: true through DAX does work, DAX simply passes that request straight to DynamoDB without serving or populating it from cache, so you get a genuinely fresh read at the cost of losing DAX's cache acceleration for that one call.

Worked example

A checkout service writes an order row, then a fulfillment worker in the same request path reads it back to grab the shipping address. For a sub-4KB item, an eventually consistent GetItem consumes 0.5 RCU while a strongly consistent one consumes 1 RCU, so setting ConsistentRead: true on that one call costs an extra 0.5 RCU and removes the race entirely, versus a blind retry loop that adds latency and still isn't guaranteed to succeed on the first attempt.

Trade-offs and pitfalls

• Turning on strongly consistent reads everywhere "to be safe" roughly doubles read capacity cost and latency across the service; it should be applied surgically to the one call with the race, not the whole read path.
• Strongly consistent reads do nothing for a Global Table's cross-region replica (there is no cross-region equivalent), and while they do work through DAX, they lose all cache acceleration when they do, a gap teams often discover only after a multi-region or DAX rollout.
• The most robust fix is usually architectural (pass the value forward instead of re-reading it), since it removes the timing dependency entirely and also removes the extra read-capacity cost.

What the interviewer probes next

Expect a follow-up comparing this to S3, which has provided strong read-after-write consistency for every operation (new objects, overwrites, and deletes) since late 2020, unlike DynamoDB's opt-in ConsistentRead, and how you'd catch this class of race in production before a customer reports it.

Direct answer

Use asset inventories and access logs to rapidly enumerate every system the suspicious activity could have touched, confirm what actually left by correlating access and transfer logs against known-legitimate patterns, and contain the exfiltration channel itself while you finish scoping rather than waiting for a complete picture first.

Structured elaboration

Rapidly identifying affected systems. Start from the specific account, host, or service where the exfiltration was first observed, then use asset inventories, service dependency maps, and access logs to trace outward: what else did this account or host have access to, and does activity on those systems show the same anomalous pattern. Automated queries against your asset inventory (which services does this account have access to, which hosts share this network segment) are far faster than manually working through a list, especially at any meaningful scale.

Confirming what left and when. Correlate access logs (who or what accessed the data) against transfer or egress logs (what actually moved, and to where) to build a defensible summary distinguishing data the attacker could reach from data that was actually copied out; access alone doesn't prove exfiltration, but combined with unusual outbound transfer volume or destination, it becomes a credible evidentiary chain.

Producing an evidentiary summary. Document what you found, from what sources, with timestamps, in a form that can support both the technical remediation decision and any downstream legal or regulatory reporting obligation, since this summary may need to stand up to scrutiny well after the immediate incident is resolved.

Containing while still scoping. Don't wait for a complete picture before acting on the exfiltration channel itself; block or revoke the specific access path in use as soon as you have reasonable confidence, and continue scoping in parallel, since every additional minute of uncontained access is potential additional harm regardless of how complete your current understanding is.

For a customer-facing API caught exfiltrating data, this plays out on a compressed timeline: in the first 60 minutes, contain the specific vulnerable endpoint or credential and begin evidence collection; in the first 24 hours, complete the scoping (what data, how much, which customers affected) well enough to make an informed decision on customer notification and regulatory reporting obligations. Where the data involved is personal or otherwise sensitive, balance preserving evidence against minimizing further exposure: don't leave a vulnerable path open longer than necessary just to gather more forensic detail, since the harm from continued exposure typically outweighs the incremental investigative value once the channel is well understood.

Worked example

A service account is found to have downloaded an unusually large volume of records overnight. Using the asset inventory, the team quickly confirms this account only has access to two specific databases, narrowing the scope immediately rather than needing to check every system in the environment. Correlating the account's access logs against network egress logs confirms not just that records were queried, but that a matching volume of data left via an outbound transfer to an unfamiliar destination in the same window, distinguishing "could have accessed" from "actually exfiltrated." The vulnerable credential is revoked within the first hour, well before the full scope (exactly which record types and roughly how many) is finalized over the following several hours, since delaying containment to finish the count first would have left the channel open longer than necessary.

Trade-offs and pitfalls

Waiting until scoping is fully complete before containing the channel is the most common and costly mistake, trading a small amount of investigative completeness for a real, ongoing exposure window. The opposite mistake, treating mere access as proof of exfiltration without corroborating actual data transfer, risks overstating the incident's severity and triggering unnecessary notification obligations on data that was reachable but never actually left.

Direct answer
For a CPU spike causing timeouts: containment is isolating the affected host or throttling the traffic causing the spike so the problem stops spreading to healthy instances; mitigation is shedding non-critical load or scaling out so timeouts stop for most users even before you know the root cause; recovery is restoring the service to its normal capacity and confirming latency has genuinely returned to baseline, not just dropped once.

Structured elaboration

• Containment answers 'how do I stop this from getting worse.' For the CPU spike: pull the overloaded instance out of the load balancer pool so it stops receiving new traffic and can't drag healthy instances down with it (for example through a shared connection pool or retry storm). Safety check: confirm the load balancer has enough remaining healthy capacity before pulling the instance, or you've just made the outage worse for everyone else.
• Mitigation answers 'how do I reduce what customers feel right now,' independent of understanding the root cause yet. For the CPU spike: shed non-critical requests (defer background jobs, disable an expensive feature) or scale out horizontally to spread load. Safety check: shedding load has to be reversible and clearly logged, so nobody forgets a feature is disabled after the incident ends.
• Recovery answers 'is the service actually back to normal.' For the CPU spike: bring capacity back to its normal level, watch CPU and latency hold at baseline for a sustained window, and re-enable anything that was shed. Safety check: recovery isn't declared on a single good data point, since a transient dip can look like recovery for a minute and then relapse.

Worked example
Suppose CPU on the affected instance is pinned at 95% and p99 latency has risen from a normal 120ms to 4 seconds, causing client-side timeouts. Containment: the on-call engineer removes the instance from the load balancer pool, redistributing its traffic across the remaining healthy instances. Mitigation: because the remaining instances are now each carrying more load, they scale out from 4 to 6 instances and temporarily disable a CPU-heavy recommendation feature; CPU across the fleet drops to a more sustainable 60% and p99 latency returns to around 300ms, well below the timeout threshold even if not yet fully back to baseline. Recovery: once the root cause (an inefficient query introduced by a recent deploy) is identified and fixed, the team restores the original instance count, re-enables the recommendation feature, and watches p99 latency hold near 120ms for 30 minutes before declaring the incident resolved.

Trade-offs and pitfalls
Containment that's too aggressive (pulling too many instances, or isolating a component that other services still depend on) can turn a partial degradation into a full outage; containment that's too narrow (missing that the real blast radius includes a shared resource like a connection pool or cache) leaves the spread unaddressed. Mitigation can also mask the symptom in a way that delays real diagnosis: shedding load makes CPU look healthy again, but if nobody tracks that the shed feature is still disabled, the team can lose track of what 'fully recovered' actually means. The general pattern (same containment options: read-only mode, feature toggles, traffic shaping, or temporary scaling) applies just as well to a database write-outage as it does to a CPU spike; the phase you're in, not the specific technology, determines what action is appropriate.

Approach: store a deduplication key per client+message in Redis with a TTL. Use an atomic Redis operation so two concurrent workers can't both decide to process the same message. Two common options: (1) Redis SET key value NX EX ttl (simpler) or (2) Lua script that checks, sets, and returns a definitive boolean (safer if additional logic required). I'll show both (SET NX) and a Lua example.

import uuid
import redis
from typing import Tuple

r = redis.Redis(host='localhost', port=6379, db=0)

DEDUP_PREFIX = "dedup"  # optional namespace

def dedup_key(client_id: str, message_id: str) -> str:
    return f"{DEDUP_PREFIX}:{client_id}:{message_id}"

def should_process_message(client_id: str, message_id: str, ttl_seconds: int = 3600) -> bool:
    """
    Returns True if the message is new and should be processed.
    Returns False if the message has already been processed (drop).
    Uses SET NX EX to atomically create the key only if absent.
    """
    key = dedup_key(client_id, message_id)
    # Value can hold metadata (timestamp, worker id) if desired
    value = "1"
    # SET key value NX EX ttl -> creates key only if not exists, atomically
    created = r.set(key, value, nx=True, ex=ttl_seconds)
    return bool(created)

# Alternative: Lua script that returns 1 if set, 0 if exists
LUA_SET_IF_ABSENT = """
if redis.call('exists', KEYS[1]) == 0 then
  redis.call('set', KEYS[1], ARGV[1], 'EX', ARGV[2])
  return 1
else
  return 0
end
"""
set_if_absent = r.register_script(LUA_SET_IF_ABSENT)

def should_process_message_lua(client_id: str, message_id: str, ttl_seconds: int = 3600) -> bool:
    key = dedup_key(client_id, message_id)
    return bool(set_if_absent(keys=[key], args=["1", str(ttl_seconds)]))

Key points and race conditions:

• SET NX EX is atomic in Redis: concurrent callers cannot both get True. This prevents duplicate processing.
• Race: if processing fails after marking, message is considered processed; to support retry-on-failure, use a two-state pattern (e.g., store "processing:<ts>" and a worker heartbeat or short TTL) or include an explicit confirmation step that upgrades the key to "done" with a longer TTL—use Lua to implement multi-step atomic transitions.
• TTL bounds storage. Choose TTL larger than max expected re-delivery window.
• Consider idempotent handlers downstream and monitoring for keys growth.

Direct answer

For a data pipeline whose bill keeps creeping despite flat traffic, the usual suspects are idle or oversized compute clusters, unbounded intermediate storage, unnecessary full-history reprocessing, and re-runs triggered by silent upstream failures, each with a same-day mitigation and a slower, more durable fix.

Structured elaboration

1. Idle or oversized compute clusters (clusters running around the clock, sized for a peak batch job): quick mitigation, turn on auto-termination for idle clusters and shrink the default cluster size. Long-term fix, move to ephemeral clusters that spin up per job and shut down automatically, or migrate to a serverless processing engine that only bills for compute actually used.
2. Unbounded intermediate or staging data (temp files and checkpoints from failed or retried jobs that never get cleaned up): quick mitigation, add a lifecycle rule that deletes staging data after a few days. Long-term fix, build cleanup into the pipeline itself so it doesn't depend on someone remembering to sweep it.
3. Expensive joins re-run against the full historical dataset every run: quick mitigation, add a date filter or partition pruning so a daily job only touches data that actually changed. Long-term fix, redesign the pipeline to be incremental, processing only new or changed records instead of recomputing the full history each run.
4. Reprocessing caused by upstream schema drift or silent failures: quick mitigation, add data-quality checks that fail fast instead of silently producing bad output that later forces a full re-run. Long-term fix, version the schema and pipeline logic so an upstream change doesn't force a full backfill.

Worked example

A nightly job that reprocesses a full year of history instead of just the previous day's partition is doing exactly 365 times more work than it needs to for a dataset that only grows by one day at a time. That ratio is the clearest signal that a pipeline should be redesigned to be incremental: no amount of cluster right-sizing closes a 365x gap, only changing what gets recomputed does.

Trade-offs and pitfalls

The quick mitigations are usually safe to apply immediately, since they only remove genuine waste (idle clusters, orphaned files) without touching pipeline logic. The long-term fixes, especially incremental processing, need real engineering time and testing, because a badly implemented incremental pipeline can silently miss late-arriving data, which is a correctness bug that costs more to fix later than the money it saved. If a pipeline is reprocessing full history on every run, prioritize that fix first, it's usually the single biggest driver in this space.