Lyft Site Reliability Engineer (Mid-Level) Interview Preparation Guide

Direct answer

Default to additive-only changes (new optional fields with sane defaults), never silently repurpose an existing field's name or meaning, and when the meaning genuinely has to change, introduce it as a new versioned field and dual-emit both the old and new during a defined deprecation window so every consumer (dashboards, alerts, downstream jobs) has time to migrate before the old one disappears.

Structured elaboration

Additive changes are the default and the cheap case
Adding a brand-new field with a sensible default (or simply absent, if consumers already tolerate unknown fields) is safe: existing dashboards and alerts that don't reference it are unaffected, and new tooling can start using it immediately. Most schema evolution should fit this case; if it doesn't, that's a signal the change is more than "add a field."

Never repurpose a field in place
Changing what an existing field means (e.g., a latency field that used to be measured in milliseconds and is now measured in microseconds, keeping the same name) is the most dangerous kind of change, because it fails silently: old dashboards keep running the same query and now show numbers that are wrong by a constant factor, with no error to alert anyone. A rename or unit change should always get a new field name (latency_ms retired in favor of latency_us, both emitted for a transition period), never an in-place redefinition.

Version the schema explicitly
Tag every emitted record with a schema_version. Consumers that need to branch on shape (a downstream parser, a strict dashboard query) can check the version rather than guessing from field presence. This also gives you a clean place to document exactly which version introduced which change.

Deprecation as a process, not an event

1. Announce the field's replacement and the planned sunset date.
2. Dual-emit: write both the old and new field for a fixed window.
3. Track actual usage of the old field (query logs, dashboard/alert definitions referencing it) to confirm consumers have migrated, not just assume they have.
4. Only stop emitting the old field once usage has genuinely dropped to zero (or the sunset date passes and remaining consumers have been explicitly notified they'll break).

Testing the transition
Contract tests (automated checks that a producer's output still satisfies what a known consumer expects) and shadow validation (running the old and new emission side by side and diffing the derived metrics they produce) catch the case where the "safe" additive change turns out to interact badly with an existing aggregation, before it reaches production dashboards.

Worked example

A service currently emits {"latency": 245, ...} where latency is milliseconds, and the team wants to switch to microsecond precision.

Wrong approach (in-place redefinition): change the emitter to write {"latency": 245000, ...} under the same field name. A dashboard panel computing avg(latency) over the last hour now silently reports a number 1000x larger with zero errors or warnings; anyone glancing at the dashboard sees "avg latency: 245000ms" and either panics or, worse, doesn't notice because the panel has no sanity bound configured.

Correct approach: add latency_us alongside the existing latency field, dual-emit both for a stated transition window (e.g., until every dashboard query referencing latency has been rewritten to use latency_us, confirmed by grepping the dashboard/alert config repository for the old field name), then drop latency only after that grep returns zero references.

The key diagnostic in this example: the failure mode is not "the pipeline throws an error," it's "the pipeline keeps running and produces a wrong number that looks plausible." That's why additive-with-a-new-name is the default, not an optional extra step.

Trade-offs & pitfalls

• Dual-emitting indefinitely accumulates cost and confusion; every deprecation needs an explicit sunset date, not an open-ended "eventually."
• Tracking actual field usage (rather than assuming consumers migrated because you announced it) is the step most teams skip, and it's exactly the step that prevents a surprise outage when the old field is finally dropped.
• Additive changes still need CI-enforced schema compatibility checks (backward/forward compatibility validation), because "just add a field" can still break a strict consumer that rejects unknown fields.
• A silent semantic change is strictly worse than a loud break: a query that errors gets noticed and fixed; a query that keeps returning a plausible-looking wrong number can go unnoticed for months.

Direct answer
Decomposing a large service into smaller, well-bounded ones limits how far a single failure can spread, because a bug, resource exhaustion, or outage confined to one small service only takes down the functionality that service owns, instead of a shared process where the same failure could take down every feature that happened to be bundled into it.

Structured elaboration
Blast radius in a monolithic (or overly broad) service is large because everything runs in the same process, shares the same resource pool (memory, connection pool, thread pool), and typically deploys together; a memory leak or a slow downstream dependency in one code path can exhaust shared resources and degrade or crash the whole service, taking unrelated features down with it. Splitting that broad service into smaller, independently-deployed pieces along genuine bounded contexts means each piece has its own process, its own resource pool, and its own deploy and rollback cycle, so a failure in one is naturally contained to the functionality it owns, and recovering it (restarting it, rolling it back) doesn't require touching or redeploying the unrelated pieces.

Worked example
A service that originally handled both order processing and a resource-intensive report-generation feature in the same process: a runaway report-generation job consuming excessive memory could previously degrade order processing too, since they shared the same process and resource pool. Splitting reporting into its own service means a runaway report job now only affects reporting; order processing, running in a completely separate process with its own resources, is unaffected, and the reporting service alone needs to be restarted or scaled to fix the issue, without any customer-facing order-processing impact.

Trade-offs and pitfalls
The trade-off this introduces is more inter-service communication to reason about: what used to be a function call within one process (order processing needing something from the reporting logic, if it ever did) now potentially becomes a network call, which can fail in ways an in-process call can't (timeouts, partial failures, network partitions), and needs its own error handling. Splitting purely for blast-radius reduction without also handling those new failure modes at the boundary (timeouts, retries, fallback behavior) can trade one class of failure (a shared-process crash) for another (an unhandled network failure cascading anyway, just through an HTTP call instead of a function call); the blast-radius benefit only fully materializes when the new inter-service calls are also built defensively.

Direct answer

A message queue absorbing write spikes ahead of a relational database is a genuinely good pattern, but "just add a queue" is not a complete design: it trades one problem (the database can't take the burst) for a set of operational pitfalls the direct-write path never had to think about, namely ordering, transactional boundaries between the queue and the database, schema evolution across producers and consumers, unbounded queue growth, backpressure propagation, and safe recovery after an outage. Every one of those has a real mitigation, but each mitigation adds its own complexity and cost, so the honest framing in an interview is "here is what this buys you and here is what you now own," not "queues solve write spikes."

The six operational pitfalls

Getting to effectively-once processing over an at-least-once queue

Most queues guarantee at-least-once delivery, meaning a message can be redelivered (after a consumer crash mid-processing, for example) and the consumer must be built to tolerate that. Effectively-once processing, where a redelivered message doesn't produce a duplicate effect even though the queue itself only promises at-least-once, comes from three pieces working together: an idempotency key attached to each message (a stable identifier for "this exact logical write," not the queue's own delivery ID, which changes on redelivery), a dedup store that records which idempotency keys have already been processed so a redelivered message is recognized and skipped, and the same transactional outbox pattern from the table above, so the message that could be redelivered and the database write it corresponds to were never two independently-failing operations in the first place.

Choosing delivery semantics per job type in a background-job system

Not every job needs the same guarantee, and treating all jobs as if they did either over-invests in low-stakes jobs or under-invests in high-stakes ones:

Deduplication implementation choices at high throughput

Whichever approach is chosen, the dedup store's own growth needs active management: an unbounded exact store retaining every key forever eventually becomes its own capacity problem, so a TTL tied to the maximum realistic redelivery window (not "forever"), periodic compaction, or tiering older entries to cheaper storage are all standard ways to keep the dedup store itself from becoming the next bottleneck.

Worked example: two numbers that motivate bounding the dedup store and partitioning by key

Partitioning for per-key ordering at scale. Getting exactly-once, per-key-ordered processing at high throughput typically means routing to thousands of partitions rather than a handful, since per-key ordering forces every message for a given key onto one partition. As an illustrative assumption, not measured data, suppose the target throughput is 2,000,000 messages/sec and a single partition can sustain roughly 500 messages/sec while preserving strict per-key order within it:

$\text{partitions needed} = \frac{2{,}000{,}000}{500} = 4{,}000 \text{ partitions}$

This is the throughput-versus-latency trade-off in concrete terms: adding partitions scales aggregate throughput almost linearly, but any single key is still pinned to one partition for ordering, so that key's messages are bounded by that one partition's 500 messages/sec, no matter how many of the other 3,999 partitions have spare capacity. A downstream stateful processor consuming a partition also needs exactly-once handling of its own state mutations (typically by committing its offset and its state update together, atomically), and that per-partition coordination cost is part of why partition count is a real tuning decision, not "more is always better."

Why the dedup store needs a bound. As a further illustrative assumption, suppose every one of those 2,000,000 messages/sec needs a dedup entry retained for a 24-hour redelivery window, and each entry costs roughly 100 bytes:

$\text{events/day} = 2{,}000{,}000 \text{ /s} \times 86{,}400\text{s} = 172{,}800{,}000{,}000 \text{ events}$

$\text{dedup store size} = 172{,}800{,}000{,}000 \times 100\text{ bytes} = 17{,}280{,}000{,}000{,}000\text{ bytes} \approx 17.28\text{ TB}$

At this throughput, an unbounded exact dedup store retaining a full day of keys grows into a multi-terabyte system on its own, which is exactly why compaction, a shorter TTL calibrated to the actual maximum redelivery delay rather than a full day, or a bloom filter pre-filter in front of a smaller exact store are standard choices at this scale rather than optional polish.

Trade-offs and pitfalls

• Treating the queue as having solved the write-spike problem once it's in place, without addressing any of the six pitfalls above; the queue moved the bottleneck, it did not remove the underlying complexity.
• Reaching for full effectively-once machinery (idempotency key, dedup store, outbox) for every job type uniformly, which over-invests engineering effort in jobs where an occasional duplicate or dropped event genuinely does not matter.
• Choosing a bloom filter for deduplication without accounting for its false-positive rate, which silently drops legitimate new messages rather than failing loudly; that trade-off needs to be a deliberate choice, not a surprise discovered in production.
• Partitioning aggressively for throughput without recognizing that per-key ordering caps any single key's throughput at one partition's capacity; more partitions help aggregate throughput and hot-key isolation, not a single hot key's own ceiling.
• Skipping a rate-limited, tested recovery drill for the outage-recovery path; the first time a large backlog gets replayed should not be during a real outage.

Direct answer

For a boolean flag (or a small set of category memberships) tracked across millions of entities, a bitset (one bit per entity per flag, packed into a flat array of machine words) beats a hash set of member IDs on both memory and multi-flag query cost whenever a meaningful fraction of entities actually carry the flag. A hash set only wins when the flag is genuinely sparse, since its memory scales with the number of members, not the population size.

Structured elaboration

Bitset representation: N entities need $N/8$ bytes flat (1 bit per entity), addressed directly by entity index. "Has flag A and flag B" becomes a single bitwise AND across the two byte arrays, one machine word at a time (e.g. 64 bits = 64 entities per instruction) - the work is O(N) bits total, but with a tiny constant factor, since it's pure word-parallel ALU work with no hashing and no pointer chasing.

Hash set representation: only the entities WITH the flag are stored, each as (say) an 8-byte id plus hashing/bucket overhead; a reasonably tuned open-addressing set needs load-factor slack and per-slot metadata on top of the raw id, so assume roughly 24 bytes per entry all-in as an illustrative estimate. Total memory is then $k \cdot b$ where k is the member count and b the per-entry byte cost, independent of N. "A and B" becomes an intersection of two hash sets: O(min(|A|, |B|)) hash lookups into the larger set - cheap when both sets are small, but each lookup costs a hash computation and a probe, not a single ALU instruction.

Break-even point:

Sparse middle ground: compressed bitmap formats (such as Roaring bitmaps) adaptively switch between array, bitmap, and run-length containers per chunk of the index range, approaching hash-set-sized memory when a flag is sparse and raw-bitset speed when it's dense, at the cost of extra implementation complexity and a small per-operation overhead versus a flat bitset.

The same idea, one level down (folding the game-engine collision-layer case): a per-entity collision-layer bitmask (up to 64 layers) is the identical idea with the axes swapped. Instead of one bitmap spanning the whole population for each flag, each individual entity carries its own fixed-width integer, where a single 64-bit word covers up to 64 category bits, and two entities' membership overlap becomes one AND across those two words, rather than a hash lookup on either side.

Worked example

Plugging concrete numbers into the break-even formula above:

N = 100_000_000       # total entities
b = 24                 # assumed bytes per hash-set entry (8-byte id + ~16 bytes overhead)

bitset_bytes = N / 8
breakeven_k = bitset_bytes / b
breakeven_fraction = breakeven_k / N

print(f"bitset size: {bitset_bytes:,.0f} bytes")
print(f"breakeven k (members): {breakeven_k:,.0f}")
print(f"breakeven fraction of N: {breakeven_fraction:.6f} ({breakeven_fraction*100:.4f}%)")

Output:

bitset size: 12,500,000 bytes
breakeven k (members): 520,833
breakeven fraction of N: 0.005208 (0.5208%)

So with these assumptions, a hash set is smaller only while fewer than about 0.52% of the 100 million entities carry the flag; past that density, the flat bitset is both smaller AND cheaper to query.

The per-entity collision-mask version of the same idea:

LAYER_PLAYER = 1 << 0
LAYER_ENEMY = 1 << 1
LAYER_TERRAIN = 1 << 2

entity_mask = LAYER_PLAYER | LAYER_TERRAIN
other_mask = LAYER_ENEMY | LAYER_TERRAIN

collides = (entity_mask & other_mask) != 0  # shares the "terrain" layer bit
print("collision check (shared layer bit set):", collides)

Output:

collision check (shared layer bit set): True

Whether the "many entities x few flags" bitset lives on the population axis (feature flags) or the "one entity x many categories" bitmask lives on the per-object axis (collision layers), the payoff is the same O(1) word-parallel membership or overlap test instead of a hash lookup.

Trade-offs & pitfalls

• A bitset needs entities to have small, dense, stable integer indices; if entity IDs are sparse (e.g. UUIDs), you need an id-to-dense-index table regardless, which itself costs memory a hash set skips.
• Bitset updates (flip a flag) are O(1) but require already knowing the entity's index; hash set add/remove is also O(1) average, with a higher constant cost from hashing and resize amortization.
• Iterating "give me every entity with flag A" requires a bit-scan over the bitset (fast, but not free); a hash set hands you the member list directly at no extra cost.
• Reaching for a hash set out of habit on a flag that is actually dense (say 40% of entities) wastes memory relative to a bitset AND makes multi-flag boolean queries meaningfully slower.
• Compressed bitmaps trade a little per-operation overhead for adapting automatically across the sparse/dense spectrum, which pays off once you have many flags of very different densities rather than hand-picking bitset vs. hash set per flag.

I use a mix of active learning, curated feeds, hands‑on labs, and deliberate prioritization to stay current with SRE tech (Kubernetes, observability, cloud infra).

Weekly routine:

• Monday quick scan (1h): RSS/Feedly, Twitter lists (K8s SIGs, CNCF), and newsletters (KubeWeekly, SRE Weekly) to surface major updates.
• Twice/week (2–3h total): Read one deep blog or RFC and watch a short talk (KubeCon talks, CNCF webinars).
• Hands‑on Friday (2–4h): Small lab—upgrade a minikube/ Kind cluster, try a new Prometheus/OTel exporter, or experiment with a new Terraform provider in a sandbox repo with automated tests.
• Ongoing: Participate in Slack/Discord communities, triage one relevant GitHub issue/PR monthly to learn maintainers’ priorities.

Monthly routine:

• One focused learning sprint (4–8h): follow a curated course (Pluralsight, A Cloud Guru) or complete a lab (Katacoda, cloud free tier) and document a short runbook or doc for the team.
• Share learnings in a 15‑minute brown‑bag and update our playbooks.

How I prioritize:

• Impact-first filter: choose topics that reduce toil, improve SLOs, or unblock major projects.
• Signal from incidents and roadmap: prioritize tech directly tied to recurring incidents or upcoming migrations.
• Cost/effort heuristic: prefer small, high‑ROI experiments; escalate bigger investments when cross-team benefit is clear.

This combination keeps knowledge fresh, practical, and aligned to team needs.

Direct answer

By default, DynamoDB's GetItem and Query calls perform eventually consistent reads, which can be served from a replica that hasn't yet applied the most recent write, so a read immediately after a write can come back stale or missing. The fix is either to request a strongly consistent read on that specific call, or to redesign the flow so the downstream step doesn't need to re-read a value it just wrote.

Structured elaboration

Why it happens: DynamoDB replicates every write across multiple storage nodes in the region before acknowledging the write as successful, but a default GetItem/Query can be routed to a replica that hasn't received that write yet. This is a deliberate cost and latency trade-off, not a bug: eventually consistent reads use half the read capacity and typically have lower latency than strongly consistent reads.

Your options as the operator:

1. Pass ConsistentRead: true on the specific read that needs the fresh value. It costs twice the read capacity units of an eventually consistent read and only works within the same region, but it guarantees you see every write that was acknowledged before the read started.
2. Avoid the read-after-write pattern entirely: have the writer hand the value it just wrote directly to the downstream step instead of making it re-read from the table.
3. If a separate process genuinely has to re-read, add a short retry with backoff, since same-region replication lag is typically single-digit milliseconds.

Where the ConsistentRead flag has limits: DynamoDB Global Tables (cross-region replicas) replicate asynchronously, so a strongly consistent read in one region's replica still only guarantees you see every write already acknowledged in that region, never a write still in flight from another region; there is no cross-region strong-consistency option. DynamoDB Accelerator (DAX, an in-memory cache in front of DynamoDB) is a different case: requesting ConsistentRead: true through DAX does work, DAX simply passes that request straight to DynamoDB without serving or populating it from cache, so you get a genuinely fresh read at the cost of losing DAX's cache acceleration for that one call.

Worked example

A checkout service writes an order row, then a fulfillment worker in the same request path reads it back to grab the shipping address. For a sub-4KB item, an eventually consistent GetItem consumes 0.5 RCU while a strongly consistent one consumes 1 RCU, so setting ConsistentRead: true on that one call costs an extra 0.5 RCU and removes the race entirely, versus a blind retry loop that adds latency and still isn't guaranteed to succeed on the first attempt.

Trade-offs and pitfalls

• Turning on strongly consistent reads everywhere "to be safe" roughly doubles read capacity cost and latency across the service; it should be applied surgically to the one call with the race, not the whole read path.
• Strongly consistent reads do nothing for a Global Table's cross-region replica (there is no cross-region equivalent), and while they do work through DAX, they lose all cache acceleration when they do, a gap teams often discover only after a multi-region or DAX rollout.
• The most robust fix is usually architectural (pass the value forward instead of re-reading it), since it removes the timing dependency entirely and also removes the extra read-capacity cost.

What the interviewer probes next

Expect a follow-up comparing this to S3, which has provided strong read-after-write consistency for every operation (new objects, overwrites, and deletes) since late 2020, unlike DynamoDB's opt-in ConsistentRead, and how you'd catch this class of race in production before a customer reports it.

Direct answer
Decide what error detail reaches a client by defaulting to the minimum that's actually actionable for THAT client (a correlation id and a stable error code, always; a human-readable message only if it's genuinely safe and useful; never a stack trace or internal identifiers), keeping the full detail in internal logs correlated by the same id.

Structured elaboration

• Safe client-facing format: {error_code, message, correlation_id} at minimum; the message should describe what went wrong from the CLIENT's perspective ("the email field is required") never from the server's internal perspective ("NullPointerException in UserValidator.java line 42").
• Correlation ids without leaking internals: a correlation id is safe to expose (it's an opaque token, not information about your system) and is exactly what lets support/engineering find the FULL internal detail later, without the client ever seeing that detail directly.
• Localized user messages: keep the machine-readable error_code stable and English-invariant; localize the human-readable message separately based on the client's locale, so client code branching on error_code never breaks when message wording/translation changes.
• Automated tests for no leakage: a test suite that deliberately triggers every known internal exception type and asserts the CLIENT-FACING response contains none of a blocklist of sensitive patterns (stack trace markers, internal hostnames, SQL fragments, raw exception class names for internal errors) catches this class of leak before it ships, since manual review alone reliably misses it under time pressure.

Worked example
An internal psycopg2.OperationalError: could not connect to server: Connection refused... host "10.2.4.19" must never reach a client; the sanitized response is {"error_code": "internal_error", "message": "Something went wrong on our end. Please try again.", "correlation_id": "7f3e-9c"}, while the full raw exception (including the internal hostname) is logged server-side, findable by an engineer searching for correlation_id: 7f3e-9c.

Trade-offs and pitfalls
The hardest cases are 5xx errors that ARE genuinely useful for the client to know more about (a specific downstream service being down, which the client's own retry logic might want to know about specifically); resist the urge to pass through the raw exception message even here, and instead define a small, deliberate set of STRUCTURED, safe detail fields ({"error_code": "dependency_unavailable", "dependency": "payment_gateway"}) rather than either a blanket generic message or a raw leak.

Direct answer
Limiting a canary's blast radius to a single region behind a global load balancer means routing based on BOTH region AND canary assignment together, so users in the target region get split between canary and stable while every other region stays entirely on stable, with session affinity handled so a user doesn't flip between versions mid-session, and a cross-region failover path that doesn't accidentally expose the canary to a region it was never meant to reach.

Structured elaboration

1. Region-scoped canary: configure the global load balancer's routing so only requests already destined for the target region are further split by the canary weighting; requests to every other region bypass the canary logic entirely and go straight to stable, keeping the blast radius genuinely contained to one region's traffic.
2. Session affinity: within the target region, use a stable hash of the user's identity (not a random per-request choice) to decide canary-vs-stable, so once a user lands on the canary, they consistently stay there for the DURATION of their session rather than flip-flopping between versions on each request, which would both confuse metrics and give users an inconsistent experience.
3. Cross-region failover: if the target region fails over to another region (a genuine regional outage, unrelated to the canary itself), the failover target region needs to know NOT to apply the canary split, since the canary was only meant to affect that one specific region's traffic; failing over should route everyone, including the canary cohort, to STABLE in the failover-target region, rather than accidentally expanding canary exposure to a region it was never validated in.
4. Metrics scoped to the region: canary-vs-stable comparison metrics need to be filtered to the target region specifically, since aggregating in metrics from unaffected regions (which are 100% on stable) would dilute or distort the comparison.

Worked example

flowchart TB
  GLB[Global Load Balancer] -->|region=US-target| Split[Canary/Stable split, 10/90]
  GLB -->|region=EU| Stable_EU[100% stable]
  GLB -->|region=APAC| Stable_APAC[100% stable]
  Split --> Canary_US[Canary, US only]
  Split --> Stable_US[Stable, US]
  Canary_US -->|failover| Stable_EU

A user in the target region hashed into the canary cohort stays on canary consistently across their session (via the stable-hash session affinity); if that region experiences an unrelated outage and traffic fails over to the EU region, the failover path routes explicitly to EU's STABLE tier, not attempting to preserve the canary assignment across a region boundary it was never validated for.

Trade-offs and pitfalls
The specific risk this design guards against is a REGIONAL FAILOVER accidentally becoming a canary-exposure EXPANSION, silently putting canary-cohort users onto a fresh region where the canary was never tested against that region's specific infrastructure, traffic patterns, or configuration; the common mistake is a failover mechanism built independently of the canary logic that doesn't know to override the canary assignment during a cross-region failover event.

Requirements & constraints:

• Move 20 TB from hot block storage to cheaper infrequent-access (IA) tier.
• Minimize egress charges and user-visible latency for occasional reads.
• Preserve availability, allow rollback, verify integrity, automate lifecycle.

High-level approach:

• Use an object-store-based IA tier (e.g., S3 Standard-IA or S3 Intelligent-Tiering / S3 Glacier Flexible Retrieval depending on access pattern). Keep frequently-read objects in hot storage for a short probation period; cold candidates moved gradually.
• Implement phased migration: pilot (1%), ramp (10%, 50%), full (100%).

Architecture & components:

1. Migration controller (K8s job or Lambda) that:
   • Reads candidate list from manifest (tagged objects or DB).
   • Copies objects server-side (S3 CopyObject or provider-equivalent) to IA bucket or same-bucket with different storage-class to avoid egress.
   • Verifies checksum (ETag / MD5) after copy.
   • Updates metadata/DB to mark new tier and copy timestamp.
   • Optionally retains original for a retention window before delete.
2. Lifecycle rules:
   • After successful verification and retention window (e.g., 30 days), transition original/remaining copies to delete.
   • For S3: Set lifecycle to transition to Standard-IA after 30 days, Glacier after 180 days, and expiration after 365 days as business policy.
3. Read-path optimization:
   • Serve reads via gateway service that checks metadata: if object is in IA, use direct GET with provisioned range reads + caching (CloudFront or edge cache) and adaptive prefetch for whole-object on first access.
   • For Glacier/Deep Archive, use asynchronous restore with on-demand caching: when a user requests an archived object, return quick 202 + notification or provide partial object if possible (use select/byte-range if storage supports) and warm into hot cache.
4. Cost & egress minimization:
   • Use server-side copy within same region to avoid egress.
   • Deduplicate copies; do not download-upload.
   • Use bulk multipart copy for large objects to reduce request costs.

Data flow:

• Migration controller issues CopyObject (in-region) -> verify checksum -> update DB/tag -> schedule original delete after retention.

Testing:

• Pilot 1% with representative object sizes/types. Validate:
  • Correct storage-class assignment and checksum equality.
  • Read latency from IA vs hot; simulate occasional reads (Poisson).
  • Cost model: simulate request and retrieval fees.
• Ramp up in stages; run integrity scans and reconcile counts/sizes.
• Load-test read-path including cache misses to ensure SLOs (e.g., 95th percentile read < target).

Monitoring & alerts:

• Metrics: migration rate (GB/hr), failed copies, verification failures, cost projection, read latency, restore request rate.
• Alerts on >1% failure rate, migration slower than schedule, unexpected read spike from IA.

Rollback & safety:

• Retention window: keep original hot copy or snapshot for X days (e.g., 30 days) before deletion. That enables quick rollback by switching metadata back to hot.
• If mass failures, migration controller pauses; automated retries with exponential backoff; manual rollback process to re-mark objects hot and cancel lifecycle deletion.
• Test rollback during pilot.

Trade-offs:

• Using Standard-IA reduces retrieval latency vs Glacier but costs more storage; choose based on expected read frequency and retrieval SLA.
• Keeping originals increases short-term cost but allows fast rollback.

Operational runbook:

• Pre-check: inventory, cost estimate, tagging plan.
• Execute pilot, validate, ramp.
• Post-migration: enable lifecycle to remove originals after retention.
• Postmortem and cost/latency review.

This strategy balances minimizing egress (in-region server-side copy), controlling user-visible latency (use IA or caching; avoid deep-archive for occasionally-read data), and providing rollback windows and monitoring to ensure reliability.

Direct answer

A good game day has a tightly bounded scope, a named set of stakeholders who signed off before the experiment starts, a real-time comparison of the system's behavior against its SLIs (service level indicators: the specific numbers you track, like latency and error rate, that tell you whether the system is healthy) during the run, and a retrospective that turns findings into tracked action items, not just a summary email. The hard part is not running the experiment; it's building the recurring program and organizational trust that lets you run harder ones over time.

Scoping the experiment

Pick a single, realistic failure mode against a bounded slice of traffic: a specific dependency (cache, database replica, a downstream API), a specific service, and ideally a canary or staging slice of load rather than 100% of production on the first run. Define upfront what "done" looks like: which SLIs you'll watch, what the abort condition is, and who has authority to hit the kill switch.

Who to involve

• Service owners and on-call engineers for the system under test, since they know the failure modes and own the runbook being validated.
• A designated incident commander for the exercise itself, separate from whoever is executing the fault injection, so there's a clear decision-maker if things go sideways.
• Product or support stakeholders when the blast radius could touch real users, so they understand what "the recommendations service is intentionally broken for 20 minutes" means for anyone who notices.
• Observability or SRE tooling owners to make sure dashboards and alerting are actually wired up to catch what you're about to do, not just to catch organic incidents.

Measuring impact against SLIs

Capture a baseline of your SLIs (latency percentiles, error rate, saturation) before injecting the fault, then watch the same SLIs in real time during the run and compare against the SLOs (service level objectives: the target values you've committed to for those same indicators, e.g. 99.9% success rate). The goal is not "did it break" (you know it will) but "did it break within the bounds you predicted, and did the defenses (timeouts, circuit breakers, autoscaling) behave the way the runbook assumes they do."

Turning findings into a recurring program

A single successful game day proves one thing worked once. Standing up a recurring practice requires a roadmap: start with low-risk, staging-only experiments to build muscle memory and trust, then progressively widen scope (larger blast radius, real production traffic, less-scripted scenarios) as the team demonstrates it can run these safely. Getting buy-in usually means showing leadership a concrete finding from an early, low-risk drill (a specific gap the exercise surfaced) rather than asking for blanket permission to break production up front. Once a cadence is established (for example, monthly), track a maturity metric across runs, such as the fraction of prior findings that were actually remediated before the next drill, so the program itself is accountable.

Worked example

Consider a payments API game day: inject 200ms of added latency into its database replica for a scoped window, on a canary slice of traffic, with a monthly error budget of 43.2 minutes at a 99.9% SLO:

During the drill, the induced latency causes synchronous retries to queue up, and the service is measurably degraded (error rate above SLO) for 12 minutes before the circuit breaker trips and the fallback path kicks in. That single test consumed:

That is a legitimate, alarming finding on its own: a single scoped drill burning over a quarter of the monthly error budget means either the blast radius needs to be tightened further (smaller canary percentage) or the circuit breaker's failure threshold needs to trip faster. Either way it's a concrete, numeric input for the retrospective and the case for continued investment in the program, rather than a vague "went well."

Trade-offs & pitfalls

Widening scope too fast is the single biggest risk to the program's survival: one game day that causes a real customer-visible incident before the team has built confidence can kill the practice for a year. The opposite failure is scoping every drill so conservatively that it never surfaces anything new, which also erodes stakeholder buy-in because the exercise starts to look like theater. The retrospective is where most of the value is either captured or lost; findings that don't get a tracked owner and a re-test in the next cycle tend to silently repeat.