Lyft Site Reliability Engineer (Junior Level) Interview Preparation Guide

Liveness, readiness, and startup probes all ask whether a container is okay, but each answer drives a different Kubernetes action: a failing liveness probe gets the container restarted, a failing readiness probe gets the pod pulled out of Service traffic without touching the container at all, and a startup probe simply delays the other two until the app has had time to boot.

What each probe gates

Worked example: sizing a startup budget by workload archetype

What 'booting' means differs a lot by workload, and the startup probe has to be sized for the actual archetype, not guessed at: a machine learning (ML) inference service loading model weights into memory might need several minutes; a batch worker doing asynchronous Java Virtual Machine (JVM) warmup, classloading, and connection-pool initialization for an extract-transform-load (ETL) job might need under a minute; a stateless HTTP handler might be ready in under a second. Whichever number applies, it has to be encoded as periodSeconds times failureThreshold. Budgeting 5 minutes of startup headroom with a 10-second check interval for the ML case:
$10 \times 30 = 300\text{s} = 5\text{ min}$
means periodSeconds: 10 and failureThreshold: 30. Too tight in this calculation and the startup probe itself kills a healthy-but-slow container before it ever gets a chance to serve; too loose, and a genuinely stuck container burns minutes before anything reacts.

Trade-offs and pitfalls

• Swapping liveness and readiness is the classic mistake: pointing liveness at a deep dependency check (database reachability) means a transient database blip restarts every application pod at once instead of simply pulling them from rotation, turning a recoverable dependency issue into a self-inflicted outage.
• Using liveness as a substitute for a startup probe on a slow-booting app causes a restart loop before the app ever finishes initializing, since the container never survives long enough to pass a liveness check tuned for steady-state behavior.
• A readiness probe that is too permissive, common with the JVM-async pattern where the process starts accepting connections before its dependency pools are actually warm, reports the pod as ready while real requests still fail; that failure mode never shows up as a probe failure at all, only as user-visible errors.

Direct answer
I'll use Amazon's "Customer Obsession" as the example: in plain terms it means starting from the customer's actual experience and working backward to the decision, rather than starting from what's easiest or cheapest for the team and working forward to how it will land on the customer. In day-to-day work that shows up as a specific, repeatable habit: before finalizing a decision, explicitly write down what the customer will experience as a result, not just what the team will ship.

Structured elaboration

1. State the value in plain language first, in one or two sentences, before layering on any nuance. A stated value is only useful if you can restate it without jargon; if you can't, you probably don't understand it well enough to apply it.
2. Trace two or three concrete decisions the value would actually change, not just decisions it would be compatible with. The test is not "does this decision fit the value" (almost any reasonable decision can be described as fitting almost any value after the fact); the test is "would I have decided differently without this value in mind."
3. Be specific about the mechanism, not just the outcome. It's not enough to say "I'd focus on the customer"; describe the actual practice (writing the customer-facing consequence down explicitly, reviewing a metric that measures customer impact rather than only internal effort, asking a specific question in a design review) that operationalizes the value day to day.
4. Acknowledge the value has a cost or a trade-off, because a value with no real cost usually is not being taken seriously. A genuinely operative value changes what you'd otherwise have done, which means it sometimes means doing the harder or slower thing.
5. Connect it back to your own role specifically, since the same value plays out differently for different functions; the mechanism for a backend engineer, a designer, and an analyst are all different concrete practices in service of the same underlying value.

Worked example
Say you're building a dashboard intended to help a seller reduce order defects. A team NOT applying customer obsession as a working discipline might ship the dashboard once the underlying data pipeline is stable and the metrics are technically correct, treating "the data is right" as the finish line. Applying the value changes the finish line: before shipping, you'd sit with two or three actual sellers using an early version and ask what decision they're trying to make when they open it, which might surface that they need same-day defect data to catch a bad batch before it ships further, not a metric that's accurate but a day stale. The concrete decision that changes: you invest in a same-day data refresh even though it's more engineering effort than the weekly batch job you'd planned, because the customer's real decision-making need, not the easier technical path, is what determines what "done" means. The cost is real (more pipeline complexity, tighter SLAs to maintain) which is exactly why it's evidence the value is actually operative rather than decorative.

Trade-offs & pitfalls
The most common failure is reciting the value's definition fluently and then giving an example so generic it would apply to any company with any stated value ("I always think about the user"), which demonstrates you've read the careers page rather than that you understand the mechanism. A second pitfall is picking an example where the value cost nothing: if every example you give was also simply the obviously correct engineering or business call regardless of the stated value, you haven't actually shown the value did any independent work in your reasoning. A third is over-indexing on one company's specific phrasing so heavily that the answer would sound out of place at any other employer; the goal is to show you can genuinely reason from a stated principle to a concrete decision, a transferable skill, not that you've memorized one company's vocabulary.

Direct answer

Given app-server CPU around 30%, RAM around 60%, database CPU around 85% with connections near the configured limit, and low network utilization, the leading hypothesis is that the database is the bottleneck, most likely from expensive queries, missing indexes, or lock contention, not application-tier capacity. The app tier's low utilization rules it out as the primary constraint; the investigation should start at the database, not by adding app servers.

Structured elaboration

Diagnosis, in order:

1. Confirm the correlation. Verify the response-time increase tracks with database CPU and connection saturation, using application performance monitoring (APM) traces and p95/p99 (95th/99th-percentile) latency, not just an average.
2. Look inside the database. Check currently running queries, the slow-query log, and query plans (using EXPLAIN ANALYZE or the equivalent) for the most expensive statements; check for lock contention and long-running transactions; check cache hit ratio and I/O wait.
3. Check the connection layer. Inspect the application's connection-pool configuration (a connection pool is a capped, reusable set of already-open database connections that app instances share rather than opening a new one per request; pool size, leaks, and retry storms are the specific things to check) and any pooling middleware between app and database.
4. Rule out a recent change. Check for a recent deployment, schema migration, or genuine traffic-pattern shift that coincides with the onset.

Short-term mitigations (reduce user impact while investigating):

• Add or raise caching for read-heavy endpoints to take load off the database directly.
• Add a targeted index for the single most expensive query identified in the slow-query log.
• Route read traffic to a read replica if one exists, as a read-scaling measure (not a failover measure).
• Throttle non-critical background jobs and batch imports competing for the same database.
• Tune the application's connection pool to prevent a connection storm, and kill any identified runaway transaction holding locks.

Long-term architectural fixes:

• Add read replicas with a defined read-routing strategy, and a connection-pooling proxy in front of the database to cap total connection count.
• Introduce a dedicated caching layer for high-QPS (queries per second) reads.
• Move heavy synchronous work into an asynchronous job queue to smooth load spikes.
• Revisit schema and query design (indexing, selective denormalization, partitioning for very large tables).
• Add durable observability: query-level monitoring, wait-event alerting, and connection-saturation alerts, so the next occurrence is caught before it degrades user-facing latency.

Worked example

Applying the framework to the stated numbers: app CPU (30%) and RAM (60%) are both comfortably below saturation, and network utilization is low, which together rule out the app tier and the network as the primary constraint. Database CPU at 85% combined with connections near the configured limit is the standout signal. Pulling the slow-query log would likely surface either (a) a small number of expensive queries consuming disproportionate CPU, or (b) lock contention from a long-running transaction blocking others and causing connections to pile up waiting rather than completing quickly, which would independently explain both the high CPU and the near-limit connection count. The next concrete action is reading that slow-query log and pg_stat_activity (or the equivalent process list for the database in use) before touching any infrastructure.

Related bottleneck patterns. The same diagnostic posture (confirm with metrics, then narrow with tracing) generalizes to trickier variants:

Trade-offs & pitfalls

• High database CPU does not automatically mean the database needs more hardware. It's equally consistent with an application sending inefficient queries; scaling the database vertically before finding the offending query is a common expensive mistake that only masks the problem temporarily.
• Short-term mitigations can hide the root cause if you stop there. Adding a cache or an index reduces symptoms; without the long-term fixes, the same failure mode returns at the next traffic level.
• Killing a runaway transaction is a blunt, disruptive tool. It has side effects (failed requests, potential partial writes to roll back) and should be a deliberate, logged action, not a routine response.
• Resource metrics that look fine can still hide the real problem, as the "CPU and memory both normal" variant above shows; a diagnostic process anchored only on the big three resource metrics (CPU, memory, network) will miss GC pauses, lock waits, and thread starvation.

Situation: During a P0 outage a junior SRE on the call froze and stopped contributing while the incident was unfolding.

In-the-moment coaching (what I’d say and do):

• Calm, short cues: “Breathe with me — two deep breaths. You’re doing fine.” (physiological reset helps)
• Give one clear, low-risk task they can complete in 2–5 minutes — e.g., “Run the health-check script from runbook and paste output to the channel.” Keep instructions step-by-step.
• Use pair debugging: “I’ll share your screen / stay on this call; do step 1 while I narrate.” This reduces cognitive load and builds momentum.
• Reassure and normalize: “It’s okay to ask questions. Focus on this one thing.” Avoid public shaming; keep voice calm.
• If needed, temporarily reassign them to an observer/documenter role (timeline, commands tried) so they stay engaged and useful.

Reassigning/partitioning tasks to keep mitigation moving:

• Immediately triage and split work into parallel threads: diagnostics, mitigation (feature flags/rollback), communications, and monitoring.
• Assign concise roles with owners and SLAs (e.g., diagnostics: junior + senior buddy; mitigation: senior engineer; comms: incident lead).
• Use runbooks/checklists and small, atomic tasks to reduce ambiguity.
• Create a “safe” mitigation queue for the junior: non-destructive tasks (logs, metrics, reproductions, postmortem notes).

Supportive debrief (post-incident, learning-focused):

• Blameless 1:1 within 48 hours: discuss what happened, what they felt, and specific moments. Validate emotions, ask how they’d prefer support.
• Conduct blameless postmortem focusing on system and process fixes; highlight what they did well (e.g., asked for help, stayed on call).
• Actionable development plan: paired on-call shifts, runbook walkthroughs, tabletop drills, and a checklist for high-stress moments.
• Follow-up after 2–4 weeks to review progress and adjust support (mentorship, training sessions).
• Update runbooks and incident playbooks with clearer micro-tasks and escalation points inspired by the incident.

Why this works:

• Reduces cognitive load, preserves dignity, and keeps the person contributing.
• Parallel tasking and clear owners maintain momentum on mitigation.
• Blameless, actionable learning builds competence and preserves confidence for future on-call duty.

Direct answer

At hundreds-of-services scale, resilience can't be a per-team decision made independently for each service, because the inconsistency itself becomes the risk: one team's excellent circuit-breaker tuning doesn't help if the team three hops upstream never implemented one at all. The strategy has four layers that reinforce each other: reusable patterns shipped as shared libraries so teams don't reinvent (and mis-implement) the same primitives, governance that makes resilience a gate rather than a suggestion, telemetry that gives every team and the org as a whole a common picture of dependency health, and runbooks that turn "we know this failure mode exists" into "here's exactly what an on-call engineer does about it at 3am." Given an existing high-MTTR baseline, the fastest reduction in customer impact rarely comes from writing more resilience code first; it comes from instrumenting what's actually failing today and fixing the highest-blast-radius gaps, because at this scale intuition about "which dependency is riskiest" is usually wrong.

Reusable patterns

Ship standardized client libraries (one per major language in use) that implement the standard resilience toolkit for calling another service: something to stop hammering a dependency that's already failing, something that spaces out and caps retries so they don't pile on, something that enforces how long a call is allowed to wait, and something that stops one slow dependency from exhausting resources other calls need. (Named precisely, for readers who want the specific patterns: circuit breakers, jittered exponential backoff with retry budgets, timeout and deadline propagation, and bulkhead-style connection pooling.) Ship these as versioned SDKs or a sidecar proxy (a small helper process deployed alongside the service, in its own container but sharing the same host or pod, that intercepts outbound network calls and applies the retry/timeout/circuit-breaker logic on the service's behalf, so the service's own code doesn't have to implement the pattern itself) for languages that can't easily share a library. The goal isn't just code reuse, it's that every team's circuit breaker behaves the same way under the same conditions, so an incident responder who understands one service's failure behavior can reason about any other service's failure behavior too. A pattern catalog with runnable examples for each pattern (service-to-service versus third-party dependency, since third parties often need more conservative defaults) makes correct usage the path of least resistance.

Governance

Telemetry

Every SDK emits standardized telemetry (circuit-breaker state transitions, retry counts, latency histograms, per-dependency error codes) into a shared observability stack, feeding a small number of org-wide dashboards: a dependency heatmap showing which services are the riskiest single points of failure by fan-in (the number of other services that call into this one; a high fan-in means many things break at once if it fails), a per-service SLO burn-rate view, and a "top failing third parties" view that surfaces vendor issues before they've caused five separate team-level incidents that nobody connected. Alerting on SLO burn rate and on sudden spikes in open-circuit count catches emerging problems before they cascade, rather than after an incident is already customer-visible.

Runbooks

Per-dependency runbooks covering detection, mitigation (force-close or force-open a circuit, apply emergency throttling, shift traffic to a degraded mode), rollback, and escalation contacts, with the common actions automated (a CLI or button to toggle a circuit breaker, not a manual code deploy) so response time doesn't depend on someone remembering the right kubectl incantation under pressure. Runbooks that are only tested during real incidents are unreliable; quarterly tabletop exercises (a facilitated walkthrough where the on-call team talks through their response to a scripted incident scenario out loud, step by step, without touching any real production system) simulating a specific third-party outage validate that the documented steps actually work and that the on-call rotation knows where to find them.

Prioritizing remediation against a high-MTTR baseline

Given limited engineering time, the fastest reduction in customer impact comes from ranking services by (fan-in × current failure rate × missing-resilience-pattern count), not by which team is loudest or which service feels intuitively risky. Tracing the formula through a small, illustrative example makes the ranking concrete: Service X has fan-in 50, a current failure rate of 2% (0.02), and 3 missing resilience patterns, scoring $50 \times 0.02 \times 3 = 3.0$; Service Y has fan-in 5, a much higher failure rate of 10% (0.10), but only 1 missing pattern, scoring $5 \times 0.10 \times 1 = 0.5$; Service Z has fan-in 200, a low failure rate of 0.5% (0.005), and 2 missing patterns, scoring $200 \times 0.005 \times 2 = 2.0$. Ranked by score, the fix order is X (3.0), then Z (2.0), then Y (0.5), even though Y's raw failure rate is the highest of the three, because Y's small blast radius (only 5 callers) and already-thin gap list make it a low-leverage fix by comparison. A service with 50 upstream callers and no circuit breaker is a much higher-leverage fix than a service with 2 callers and a full resilience suite already in place, even if the second service "feels" more important because it's customer-facing. Concretely: instrument telemetry first (you can't rank what you can't see), fix the highest fan-in gaps first (biggest blast-radius reduction per engineering-hour), and treat "ship the shared SDK" and "mandate its use via the architecture-review gate" as sequential, not simultaneous, since a library nobody's required to adopt doesn't move the MTTR number regardless of how good it is.

Trade-offs & pitfalls

Uniform SDK adoption is the ideal, but legacy or polyglot systems make a single shared library impractical everywhere; a sidecar-proxy approach extends the same governed behavior to services that can't easily embed the SDK, at the cost of an extra network hop and an additional piece of infrastructure to operate. Mandating resilience patterns through architecture-review gates works for new services but does nothing for the hundreds of already-shipped services that predate the gate, so rollout has to include a deliberate retrofit plan (again, prioritized by the fan-in ranking above) rather than assuming the gate alone will fix the fleet over time. The biggest governance failure mode at this scale is treating the checklist as a one-time approval rather than a continuously monitored property. A service that passed its architecture review with a correctly-configured circuit breaker two years ago can silently regress (a config change, a library version bump that changed defaults) with nobody noticing until the next incident, which is exactly what the telemetry layer's continuous SLO-burn alerting is meant to catch that a point-in-time review cannot.

Direct answer

PACELC, short for "if Partition, Availability vs. Consistency; Else, Latency vs. Consistency", says that CAP's dilemma, choose Consistency or Availability when a network Partition is happening, is only half the story. Even when there is no partition at all, a system still has to choose between Latency and Consistency for every write it replicates, because making a write durable on every replica before acknowledging it takes longer than acknowledging it once it's durable on a single node. PACELC packages this as: if Partition occurs, trade off Availability against Consistency (exactly what CAP already says); Else, meaning no partition, trade off Latency against Consistency.

Restating CAP precisely first

CAP says that during an actual network partition, a distributed system can guarantee only one of Consistency (every read sees the latest completed write) or Availability (every request gets a non-error response) for the nodes on either side of the split, not both. A common misreading treats CAP as "pick two of three, always"; it isn't. CAP's teeth are specifically about behavior during a partition. Most systems are both consistent and available almost all of the time, precisely because a true network partition is a rare event relative to total uptime, not something happening continuously.

flowchart TD
    Start[Write occurs] --> P{Partition active?}
    P -->|Yes| AC[Choose Availability or Consistency]
    P -->|No| LC[Choose Latency or Consistency]

What PACELC adds

PACELC names the trade-off CAP is silent about: during normal operation, with no partition, you still choose between Latency (L) and Consistency (C), because synchronous replication that waits for a majority of replicas costs a round trip before it can acknowledge a write, while asynchronous or single-node-acknowledged replication returns faster but risks a reader seeing stale data, or the acknowledged write being lost outright if that one node fails before it propagates. Systems are commonly labeled by both branches together, for example PA/EL (favor Availability under partition, favor Latency otherwise, the Cassandra/Dynamo-style default) or PC/EC (favor Consistency in both cases, the HBase-style default).

Worked example: a decision with no partition occurring

A write to a piece of user data must be replicated to three nodes: R1 in the local region, and R2, R3 in two remote regions. All three are reachable; no partition is happening anywhere in this example.

• Favor consistency (the "C" side of the Else branch): the write path waits for acknowledgment from a majority, at least two of the three replicas, say R1 and R2, before returning success to the caller. Any subsequent read from a majority quorum is now guaranteed to see this write. Cost: the caller's write waits on the round trip to R2, a remote replica, even though R1, the local one, already has it durably.
• Favor latency (the "L" side of the Else branch): the write path acknowledges as soon as R1 has it durably, and replicates to R2 and R3 asynchronously in the background. Cost: the caller gets a fast, local acknowledgment, but a read served from R2 immediately afterward, before the async replication catches up, will not see the write yet. If R1 crashes before that background replication completes, the already-acknowledged write can be lost entirely, with zero partition ever occurring.

This decision, wait for two of three versus acknowledge on one, is made on every single write regardless of whether any partition is happening, which is exactly the trade-off PACELC's Else branch names and CAP alone has nothing to say about, since CAP only speaks to a system that is not fully connected.

Trade-offs & pitfalls

A common misconception is treating a database's PACELC label as a fixed law of the software rather than a description of its typical default: most systems let you tune the replication wait per request (via quorum size), so "Cassandra is PA/EL" describes its usual configuration, not something it's incapable of changing. It's also easy to blur this Else-branch trade-off with an availability discussion; in the worked example above, no node was ever unreachable, so the trade being made is purely about how long the write path waits before acknowledging, not about surviving an outage, which is a separate concern belonging to the partition branch of the theorem.

Direct answer
A layered test strategy for error paths runs unit tests that simulate a dependency failure in isolation, integration tests using test doubles for more realistic multi-component interaction, contract tests that verify a downstream dependency's actual failure-mode behavior matches what your code assumes, and failure-injection/chaos tests against real dependencies, gated in CI as blocking (fast, deterministic tiers) or non-gating/scheduled (slower, real-dependency tiers).

Structured elaboration

• Unit tests, simulated failure: mock a single dependency to raise a specific exception, asserting the function under test handles it exactly as designed (falls back, retries, propagates a specific error); fast, deterministic, runs on every commit.
• Integration tests, test doubles: exercise a fuller call path (multiple real internal components, but an external dependency still faked via a test double) so you catch integration bugs a pure unit test's narrow mock can't (an internal component genuinely NOT calling the retry logic the unit tests assumed it would use, say).
• Contract tests: verify that your MOCK of an external dependency's failure behavior actually matches the dependency's REAL behavior (a contract test run periodically against the real dependency's staging environment, or a shared, versioned contract both sides agree to and test against independently); without this, a unit test can pass forever against an increasingly-stale assumption about how the real dependency actually fails.
• Failure-injection/chaos tests: exercise the REAL dependency under an injected real fault (see the companion chaos-game-day survivor), the only tier that can catch a bug in the ACTUAL integration, not just your code's internal logic or a test double's approximation of the real thing.
• CI gating: unit and integration tests (fast, deterministic) gate every merge; contract and chaos tests (slower, sometimes dependent on external infrastructure availability) typically run on a schedule or a separate, non-blocking pipeline, with failures triaged and acted on promptly rather than being blocking-but-flaky in the main merge path.

Worked example
Testing a client's retry-on-503 logic: the UNIT test mocks the HTTP client to return a 503 once then a 200, asserting the retry fires and the final result is correct; the INTEGRATION test runs the real client code against a local test-double HTTP server configured to behave the same way, catching any bug in how the client code is actually WIRED UP (a config value not actually being passed through, say); the CONTRACT test periodically confirms the real downstream service genuinely returns a 503 (not, say, a 500 or a connection reset) for the specific failure condition your retry logic assumes, since a downstream API change could silently invalidate that assumption; a CHAOS test in staging injects a REAL transient network failure against the real dependency and confirms the whole system, end to end, recovers as expected.

Trade-offs and pitfalls
The contract-test tier is the one most commonly skipped, and its absence is exactly what lets a downstream dependency's behavior silently drift away from what your unit tests still (incorrectly) assume, with every unit test continuing to pass the whole time; treat contract-test coverage of your genuinely load-bearing failure-mode assumptions as a first-class investment, not an afterthought.

Direct answer

A distributed monolith is a system that is split into separately deployed services but still behaves like one program: the services cannot be changed, tested or released independently. You pay every cost of a distributed system (network latency, partial failures, harder debugging) and get none of the main benefit, which is independent teams shipping independently. Two strong signs are lockstep deployments (a feature routinely requires several services to release together, in a specific order) and long synchronous call chains (one user request cannot complete unless many services are all up at the same moment). A concrete first mitigation is to make deploys independent with versioned, backward-compatible APIs checked by consumer-driven contract tests, so each service can ship without waiting for the others.

What it looks like

Think of an online store split into Cart, Pricing, Inventory, Orders and Payments. On paper these are five microservices. In practice:

• Adding a "gift wrap" option means changing all five, and the release plan is a spreadsheet saying "deploy Pricing first, then Orders, then Cart, within 30 minutes, or checkout breaks".
• Placing an order calls Cart → Pricing → Inventory → Orders → Payments, one after another, synchronously.
• Orders and Inventory read and write the same products table.

Sign 1: lockstep deployment

What you observe: releases are coordinated across services; there is a shared release train (a fixed, recurring release schedule that several services are bundled onto and ship together, rather than each shipping on its own whenever its change is ready) or a "deploy order" document; rolling back one service requires rolling back others.

Why it happens: services share a data model or API that changes without versioning. If Pricing renames a field in its response, Cart breaks until Cart is updated, so they must deploy together.

How to detect it: from the deploy log, compute the fraction of releases in which a service was deployed within the same short window as another service for the same ticket. If Cart ships with Pricing in most of its releases, those two are one deployable in disguise.

Sign 2: synchronous call chains that multiply failure

What you observe: a distributed trace (a record, stitched together from the timestamped spans each service emits for the same request, showing every service that request touched and how long each one took) of one request shows a deep chain of blocking calls, and an outage in any one service takes down the whole user flow.

Why it matters, with numbers: if a request needs 5 services in series and each is independently available 99.9% of the time, the chain is available only when all 5 are up:

• 0.999 × 0.999 × 0.999 × 0.999 × 0.999 = 0.999^5 ≈ 0.99501, about 99.50%.
• Over a 30-day month (30 × 24 × 60 = 43,200 minutes), one service at 99.9% is down about 0.001 × 43,200 = 43.2 minutes.
• The 5-service chain is down about (1 − 0.99501) × 43,200 ≈ 215.6 minutes, roughly five times as much.

A monolith would have had one component's worth of downtime; the distributed monolith has five, with the same features. (This simple model assumes failures are independent; correlated failures change the number but not the lesson.)

Other signs worth naming

• Services sharing a database schema, so a column change needs coordination.
• A shared library of domain objects that every service must upgrade together.
• End-to-end tests that need every service running before anyone can merge.

The mitigation: break deploy coupling first

Why start here: lockstep deploys are the most expensive symptom (they slow every team, every release) and fixing them is incremental. Steps:

1. Version the contract. API changes become additive: add a new field, never rename or remove one in place. Consumers use a tolerant reader (ignore fields you do not recognise, do not fail on missing optional ones).
2. Add consumer-driven contract tests. Each consumer (Cart) publishes the exact requests and response fields it depends on; the provider (Pricing) runs those contracts in its own CI. Tools such as Pact implement this. Now Pricing learns in its own build, before deploying, whether it would break Cart.
3. Deploy in expand-then-contract order. To rename price to unit_price: Pricing ships a version returning both; Cart switches to unit_price whenever it likes; once contracts show no consumer reads price, Pricing removes it. Three independent deploys, no coordination window.
4. Measure the result. Track the lockstep share from the deploy log. If Cart and Pricing co-deployed in 8 of 10 releases before, the target is near zero within a couple of quarters.

If the contracts reveal that two services cannot evolve separately no matter what (they change together on every feature), the honest fix is to merge them back into one service. A boundary that always moves together was never a real boundary.

Pitfalls

• Fixing the symptom with tooling. A better release-orchestration tool makes lockstep deploys easier to run, which hides the coupling instead of removing it.
• Adding more services. Splitting further to "fix" coupling usually adds more edges.
• Replacing sync calls with async messages carrying the same shared data model. The deploy coupling survives; it just moved into the message schema.

The architecture is a pipeline of five stages, each trading some fidelity for cost, plus a reconstruction path at query time for the cases where someone genuinely needs the detail back. The stages: label reduction, aggregation and rollups, sampling, storage tiering, and retention.

Pipeline

flowchart LR
  A[Raw High-Cardinality Metrics] --> B[Label Reduction]
  B --> C[Aggregation and Rollups]
  C --> D[Sampling]
  D --> E[Storage Tiering]
  E --> F[Retention Enforcement]
  F --> G[Query Layer]
  G --> H[On-demand Rehydration: top-K exemplars]

• Label reduction: the single highest-leverage step. Cardinality is combinatorial across independent label dimensions: if endpoint has 80 distinct values, status has 6, pod has 800, and region has 4, the bounded series count is their product.

Adding one more label with effectively unbounded cardinality (like a raw request_id, close to one unique value per request) multiplies this by request volume instead of by a fixed factor, turning a bounded 1.5M-series metric into an unbounded one. Label reduction means identifying and removing or bucketing exactly those unbounded dimensions before ingestion, keeping the combinatorially-bounded ones.

• Aggregation and rollups: pre-compute coarser-grained series (per-service instead of per-pod, for example) at ingestion time so most dashboard and alert queries never touch the raw high-cardinality series at all.
• Sampling: for label values that are useful individually but too numerous to keep in full (e.g., per-customer-ID series for a B2B product with thousands of customers), keep exact series for the top-K by volume or spend, and sample or bucket the long tail.
• Storage tiering: recent raw data on fast storage, older data downsampled and moved to cheaper storage, as in a standard hot/warm/cold retention policy.
• Retention enforcement: hard expiry so the pipeline's cost stays bounded over time regardless of how ingestion volume trends.

Reconstructing detail on demand

The trick that makes this acceptable to users is that "finer detail" doesn't mean "keep everything forever," it means "keep enough breadcrumbs to go get the detail from a cheaper source when someone actually asks." Two mechanisms:

1. Exemplars: attach a sampled trace ID or raw event reference to an aggregated metric bucket, so a spike in the aggregate can be drilled into by fetching the handful of exemplar traces/logs that were kept in full, even though the metric itself was aggregated.
2. On-demand reprocessing: if raw pre-aggregation data still exists in a cheap cold tier (e.g., unindexed compressed blobs), a rare deep-dive query can trigger an offline reprocessing job rather than requiring the hot path to keep everything queryable in real time.

Sizing against a cost ceiling

Given an illustrative monthly storage budget of $5,000 and an illustrative unit cost of $0.02/GB-month (a labeled assumption for this worked example, not a live vendor quote):

Using the 1,536,000 bounded series from the label-reduction step above, at a 5-minute downsampled resolution with 8 bytes/point (consistent with the multi-aggregate downsampling estimate used for retention-tier design):

series = 1_536_000
bytes_per_point = 8
interval_s = 300
points_per_day = 86400 / interval_s
bytes_per_day = series * points_per_day * bytes_per_point   # 3.539 GB/day
max_days = (250_000 * 1e9) / bytes_per_day

That affords roughly 70,600 days of 5-minute-resolution history under the budget, which is obviously far beyond any real retention need, so the budget is not actually the constraint at this series count and resolution. Repeating the same calculation for raw 15-second resolution at 2 bytes/sample instead gives 17.69 GB/day and about 14,128 affordable days, roughly 5x shorter than the downsampled case (the 20x fewer points at 5-minute resolution is partly offset by needing 4x more bytes per point to store min/max/sum/count instead of a single value, netting exactly 20/4 = 5x). The concrete lesson: at this series count, the budget comfortably covers years of retention either way, so cost pressure at 1.5M series is not what forces sampling or tiering, it forces label reduction to happen so the series count never gets to the point where the arithmetic above breaks down (e.g., adding the unbounded request_id label would blow past 250 TB in a matter of hours).

Trade-offs and pitfalls

• Treating sampling as the first line of defense (instead of label reduction) is the most common mistake: sampling a metric whose cardinality is unbounded because of a labeling error still leaves an unbounded number of series, each just sampled less; the series count itself, not just the sample rate, is what needs to be bounded first.
• Aggregation destroys the ability to answer "which specific instance caused this" without exemplars; a design that aggregates without keeping any drill-down path trades away debuggability that's expensive to get back later.
• A fixed cost ceiling naturally reframes the problem: it's not "how do we store everything cheaper," it's "what data can we afford to keep at what fidelity," and that framing should drive which stage of the pipeline (label reduction vs. sampling vs. tiering) absorbs the cost pressure, since as shown above they don't interact linearly.
• Query-time reconstruction only works if the cheap cold tier is actually queryable, even slowly; if cold data is written in a format nothing can read without a bespoke recovery process, "reconstruct on demand" is really "data is gone" with extra steps.

The workload. 20% of requests arrive in short spikes; the rest is steadier. The core tension: provisioned capacity reacts too slowly to cover a short spike safely (launching a new instance can take on the order of a minute or more, too slow for a spike measured in seconds to minutes), so a provisioned-only design has to keep permanent headroom "just in case," while serverless/FaaS scales per request but at a higher unit price and with its own cold-start latency on the first invocation after idle.

Illustrative pricing (round numbers, not any vendor's real published rates). Instances at $0.10/hr; serverless at $0.20 per million invocations plus $0.00002 per GB-second, average invocation using 0.5 GB for 0.3 seconds.

per-request serverless cost = (0.5 GB * 0.3 s * $0.00002/GB-s) + ($0.20 / 1,000,000)
                             = $0.000003 + $0.0000002 = $0.0000032 per request

Cost model, three options, for 10 million requests/month (8M steady, 2M in spikes)

(a) provisioned for steady load only, drops/degrades during spikes:
    2 instances always-on = 2 * $0.10 * 24 * 30 = $144/month  (doesn't meet the requirement)

(b) provisioned with permanent peak headroom, spike-safe:
    5 instances always-on = 5 * $0.10 * 24 * 30 = $360/month

(c) pure serverless for all 10M requests:
    10,000,000 * $0.0000032 = $32/month

(d) hybrid: provisioned baseline for the steady 8M + serverless overflow for the 2M spike:
    $144 + (2,000,000 * $0.0000032) = $144 + $6.40 = $150.40/month

Option (a) is cheapest but is disqualified immediately: it doesn't meet the requirement to serve the spike. Among the options that do, pure serverless (c) is the cheapest here by a wide margin, because per-request pricing is only expensive relative to always-on capacity once request volume is high enough to justify paying for a dedicated instance around the clock.

The general decision rule (crossover volume). Set the always-on cost equal to volume times per-request cost to find the request volume where the two options cost the same:

crossover requests/month = always-on monthly cost / per-request cost
vs the 2-instance baseline: $144 / $0.0000032 ~= 45,000,000 requests/month
vs the 5-instance peak-headroom baseline: $360 / $0.0000032 ~= 112,500,000 requests/month

At 10 million requests/month, this workload is far below both crossover points, which is exactly why serverless wins the pure cost comparison here. If this same workload later grew to over 100 million requests/month at a similar spike ratio, the always-on peak-headroom option would become the cheaper one, and the decision should flip.

What the cost model leaves out. Serverless's per-request price is the easy part to model; the trade-off that doesn't show up in this table is cold-start latency on the first invocation after an idle period. If the service has a tight per-request p99 (99th percentile) SLO (service level objective), a straight serverless choice needs to be checked against how often a genuine cold invocation would occur (which itself depends on how spiky the traffic really is) and whether a small amount of provisioned concurrency is needed just to keep the SLO-sensitive path warm, which would nudge the total cost up from the pure-serverless number above without changing the overall conclusion.

Decision for this example. Given the actual 10M/month volume, serverless is both cheaper and natively spike-safe; keep the plain provisioned option in mind only if request volume grows well past the crossover point, or if the cold-start latency turns out to violate the SLO, in which case a small amount of provisioned concurrency layered on top of serverless is the next thing to try before reverting to always-on instances.