Airbnb Site Reliability Engineer Interview Preparation Guide (Mid-Level)

Direct answer

Extending an LRU (least-recently-used) cache with per-entry time-to-live (TTL, a lifespan after which an entry is no longer considered valid) means treating expiry and eviction as two independent mechanisms layered on the same structure: TTL decides whether an entry is even valid to serve at all, LRU decides who gets removed when the cache is full, regardless of whether that entry's TTL has expired yet. Safe concurrent access comes from sharding the cache into N independently locked pieces by key hash, so unrelated keys never contend for the same lock, rather than one global lock serializing every request. The one genuine race to guard against is a reader and an expiry-driven cleanup touching the same key at the same instant, which a single lock per shard around each operation's full check-then-mutate sequence eliminates by construction.

Structured elaboration

Data structures. Per shard: an ordered map (a doubly linked list plus a hash index, or collections.OrderedDict in Python) storing key -> (value, expires_at). get first checks expires_at against the current time; if expired, the entry is deleted and treated as a miss, never served stale. If still valid, it is moved to the most-recently-used end, same as a plain LRU. put inserts or updates the entry with a fresh expires_at, then evicts from the least-recently-used end while the shard is over capacity, exactly as a TTL-unaware LRU would, an unexpired entry can still be evicted early purely for space if it is cold enough.

Locking and sharding to minimize contention. A single lock around the whole cache serializes every request regardless of which key it touches, the same contention problem as a naive shared counter. The fix is the same: partition keys across N shards by hash(key) % N, each with its own lock, so two threads touching different shards never block each other; within a shard, the critical section (a dict lookup, a move, maybe an eviction) is small enough that a simple mutex around the whole operation is both correct and fast, a lock-free linked-list-based LRU is real engineering effort that rarely pays for itself compared to just adding more, smaller shards.

Eviction rules on TTL expiry. Two independent triggers remove an entry: lazily, on access, if its expires_at has passed; and capacity-driven, on put, evicting the least-recently-used entry regardless of that entry's own TTL state once the shard is full. A background sweep thread that proactively scans for and removes expired-but-never-accessed entries is an optional memory-hygiene addition, not a correctness requirement, lazy expiry alone already guarantees an expired entry is never served, it just might sit unreclaimed in memory until LRU pressure or a sweep removes it.

Race conditions between expiry and access. Because the expiry check and the touch-or-evict mutation both happen inside the same lock acquisition as one atomic operation, no other thread on that shard can observe a half-expired state in between. Two remaining subtleties matter: use a monotonic clock (one that only ever moves forward, unaffected by wall-clock adjustments) for expiry comparisons, since a wall-clock adjustment (an NTP correction, a daylight-saving shift) can jump time backward and silently corrupt TTL math; and treat the exact boundary consistently, an access at precisely expires_at should be treated as expired, not as a coin flip.

Code

import threading, time
from collections import OrderedDict

class TTLLRUCache:
    def __init__(self, capacity, default_ttl):
        self._capacity = capacity
        self._default_ttl = default_ttl
        self._store = OrderedDict()  # key -> (value, expires_at)
        self._lock = threading.Lock()

    def get(self, key, now=None):
        now = time.monotonic() if now is None else now
        with self._lock:
            entry = self._store.get(key)
            if entry is None:
                return None
            value, expires_at = entry
            if now >= expires_at:
                del self._store[key]
                return None
            self._store.move_to_end(key)
            return value

    def put(self, key, value, ttl=None, now=None):
        now = time.monotonic() if now is None else now
        ttl = self._default_ttl if ttl is None else ttl
        with self._lock:
            if key in self._store:
                del self._store[key]
            self._store[key] = (value, now + ttl)
            self._store.move_to_end(key)
            while len(self._store) > self._capacity:
                self._store.popitem(last=False)


class ShardedTTLLRUCache:
    def __init__(self, shard_count, capacity_per_shard, default_ttl):
        self._shards = [TTLLRUCache(capacity_per_shard, default_ttl) for _ in range(shard_count)]
        self._shard_count = shard_count

    def _shard_for(self, key):
        return self._shards[hash(key) % self._shard_count]

    def get(self, key, now=None):
        return self._shard_for(key).get(key, now=now)

    def put(self, key, value, ttl=None, now=None):
        self._shard_for(key).put(key, value, ttl=ttl, now=now)


if __name__ == "__main__":
    # basic TTL expiry
    cache = TTLLRUCache(capacity=2, default_ttl=10)
    cache.put("a", 1, now=0)
    print("get a at t=5 (not expired) ->", cache.get("a", now=5))
    print("get a at t=15 (expired, ttl=10) ->", cache.get("a", now=15))

    # LRU eviction independent of TTL (long ttl so eviction is purely LRU-driven)
    cache2 = TTLLRUCache(capacity=2, default_ttl=100)
    cache2.put("x", 100, now=0)
    cache2.put("y", 200, now=1)
    cache2.get("x", now=2)          # touch x -> y becomes LRU
    cache2.put("z", 300, now=3)     # evicts y (LRU), not due to TTL (ttl=100, still valid)
    print("y evicted by LRU (not TTL) ->", cache2.get("y", now=4) is None)
    print("x survived ->", cache2.get("x", now=4) == 100)

    # concurrency check: 16 threads, 200 put-then-get round trips each, shared 8-shard cache
    shared = ShardedTTLLRUCache(shard_count=8, capacity_per_shard=64, default_ttl=1000)
    errors = []

    def worker(tid):
        for i in range(200):
            key = f"t{tid}-{i}"
            shared.put(key, (tid, i), now=0)
            val = shared.get(key, now=0)
            if val != (tid, i):
                errors.append((key, val))

    threads = [threading.Thread(target=worker, args=(t,)) for t in range(16)]
    for th in threads:
        th.start()
    for th in threads:
        th.join()
    print("concurrent put/get across 16 threads, errors ->", len(errors))

    # exact boundary: an access at precisely expires_at counts as expired
    cache3 = TTLLRUCache(capacity=2, default_ttl=1.0)
    cache3.put("k", 42, now=0.0)
    print("exactly-at-expiry (t=1.0) counts as expired ->", cache3.get("k", now=1.0) is None)

Output (executed as shown, using an injected clock so TTL behavior is deterministic and reproducible):

get a at t=5 (not expired) -> 1
get a at t=15 (expired, ttl=10) -> None
y evicted by LRU (not TTL) -> True
x survived -> True
concurrent put/get across 16 threads, errors -> 0
exactly-at-expiry (t=1.0) counts as expired -> True

The concurrency check ran 16 threads, each performing 200 put-then-get round trips against a shared 8-shard cache, and confirmed every value was read back exactly as written with zero errors, evidence the sharded locking is actually safe under real concurrent access, not just single-threaded correctness.

Trade-offs and pitfalls

• Lazy-only expiry can leave cold, expired entries resident indefinitely if the cache has generous capacity and that key is never looked up again; add a background sweep if bounding memory tightly matters more than the extra CPU cost of scanning.
• Shard count is a real tuning knob, not a fixed choice. Too few shards under-spreads lock contention; too many wastes memory on mostly-empty per-shard structures and adds bookkeeping overhead.
• Distributed sharding is the same idea one layer up. The identical hash-the-key-to-a-shard principle extends from in-process locks to independent nodes in a distributed cache: shard by consistent hashing across nodes so the cache's total capacity and throughput scale past one machine, with a routing layer (client-side or proxy-side) sending each request to the node that owns its key's shard.
• Persistence across restarts should stay proportionate to what TTL data actually needs. Since every entry already has a bounded lifespan, a periodic best-effort snapshot to disk or an external store, reloaded at startup and immediately filtered to discard anything whose expires_at has already passed relative to current wall-clock time, is sufficient; treating this like a database's durability guarantee (distributed consensus, write-ahead logs) is usually over-engineering for data that was always meant to expire.
• A common wrong turn: using wall-clock time (time.time()) instead of a monotonic clock for TTL comparisons. It works in every normal test run and fails exactly once, silently, during a real clock adjustment in production, which is precisely why it is easy to ship and hard to catch without deliberately testing for it.

Direct answer. Decompose by responsibility, not by line count: pull out one function per distinct concern (parse, validate, compute, persist, notify), give each a name that states its single job, and compose them in a thin orchestrator that reads like the original steps.

Before (one function, five jobs)

def handle_order(raw):
    order_id = raw["id"]; email = raw["email"]; items = raw["items"]      # parse
    if not items: raise ValueError("order has no items")                  # validate
    if "@" not in email: raise ValueError("bad email")
    total = sum(i["price"] * i["qty"] for i in items)                     # compute
    if total > 100: total *= 0.9
    DB[order_id] = {"email": email, "total": total}                       # persist
    OUTBOX.append(f"receipt to {email}: ${total:.2f}")                    # notify
    return total

After

def parse_order(raw: dict) -> dict:
    return {"id": raw["id"], "email": raw["email"], "items": raw["items"]}

def validate_order(order: dict) -> None:
    if not order["items"]: raise ValueError("order has no items")
    if "@" not in order["email"]: raise ValueError("bad email")

def compute_total(items, *, bulk_discount_threshold=100, bulk_discount_rate=0.9) -> float:
    total = sum(i["price"] * i["qty"] for i in items)
    return total * bulk_discount_rate if total > bulk_discount_threshold else total

def save_order(db, order_id, email, total) -> None:
    db[order_id] = {"email": email, "total": total}

def send_receipt(outbox, email, total) -> None:
    outbox.append(f"receipt to {email}: ${total:.2f}")

def handle_order_v2(raw, db, outbox) -> float:
    order = parse_order(raw)
    validate_order(order)
    total = compute_total(order["items"])
    save_order(db, order["id"], order["email"], total)
    send_receipt(outbox, order["email"], total)
    return total

Verified against two cases: a two-item order totaling 120 (discounted to 108.0) and a single-item order totaling 20 (no discount) -- both versions return identical totals.

How to confirm you didn't change behavior

• Run the OLD function and the NEW orchestrator against the same set of inputs (including edge cases: empty items, exactly-at-threshold totals, malformed email) and diff the outputs.
• If there's an existing test suite, run it unchanged against the new code first -- a passing suite that never exercised the discount boundary is itself a gap worth flagging, not just a green check mark.
• Where no tests exist yet, write characterization tests against the OLD function first, then refactor, then confirm the same tests still pass -- this way the safety net exists before you touch anything.

Trade-offs and pitfalls

• compute_total's discount threshold is now a named, testable, overridable parameter instead of a buried magic number -- a deliberate improvement, but note it as a design decision so a reviewer knows it wasn't accidental scope creep.
• Over-decomposing (a function per line) makes you trace five files to understand one flow; stop splitting when each piece maps to one clear reason to change.
• The orchestrator itself now has a job: sequencing. Keep it free of business logic so it doesn't quietly become a sixth responsibility.

Situation: During a high-severity outage (database failover flapping) at 2am, the on-call lead and I were both paged; I was coordinating cross-team comms and needed someone to execute a rollback and runbooks because the lead engineer was tied up with upstream vendor calls. A junior SRE with 6 months on call experience was available.

Task: I needed them to perform a controlled rollback of a recent schema migration, run health checks, and gather logs—without making the production state worse.

Action:

• Clarified the objective and success criteria: "Rollback migration X to version v2, verify DB connections <1% error rate, app latency <200ms, and collect app/db logs for 15 minutes."
• Broke the work into explicit steps and handed them a checklist: pre-checks, rollback command, post-checks, and escalation triggers.
• Paired over screen share for the pre-checks (10 minutes): I watched them run the safety queries, confirm backups and replica sync, and validate maintenance window flags.
• Added safety guards: prevented destructive commands unless two approvals were given (me + on-call lead), used a read-only snapshot to rehearse, and set a temporary traffic shift to a healthy region via the load balancer before the rollback.
• Gave clear escalation thresholds (e.g., error rate >5% or replication lag >30s) and stayed available on a dedicated Zoom while they executed.
• Had them narrate each command and expected result out loud to ensure understanding.

Result: The rollback completed in 22 minutes, error rate dropped from 8% to 0.7%, and latency returned to normal. No additional data loss occurred. The junior SRE executed precisely and gained confidence.

Follow-up coaching & process changes:

• Conducted a 1:1 debrief the next day using the incident timeline; praised decisions and corrected minor gaps (improving query timeouts).
• Ran a tabletop session with the team to practice the checklist and refine escalation thresholds.
• Updated the runbook to include the safety rehearsal step, added automated pre-check scripts to enforce backup and replica health, and introduced a mandatory two-person approval for production rollbacks.
• Scheduled mentorship: weekly pairing sessions and a mandatory on-call runbook certification for new SREs.

What I learned: Breaking high-pressure tasks into clear steps, pairing during execution, and baking safety checks into runbooks lets junior engineers contribute safely while accelerating their growth.

Direct answer
Default to Terraform for the foundational, provider-broad layer, since it has the deepest multi-cloud provider coverage and the most mature ecosystem for infra teams. Reach for Pulumi specifically when the team is already strong in a general-purpose language and the win is expressiveness, real loops and functions and existing test frameworks, rather than provider coverage. Reach for Crossplane (a Kubernetes-native IaC tool that manages cloud resources as Kubernetes objects) specifically when continuous reconciliation matters more than plan/apply control, typically because the platform is already Kubernetes-centric. Treat cloud-native templates as a single-cloud tool, not a multi-cloud contender, useful only for the narrow slice of a platform that's genuinely cloud-specific anyway.

Structured elaboration

The axis that actually decides this
"Multi-cloud parity" collapses into three separable questions most comparisons blur together: does the tool have mature provider support for everything needed across all the clouds in play; is drift corrected continuously or only when someone runs a plan; and does the team's existing skill set make the tool's config language an asset or a liability. Answering those three first makes the tool choice close to mechanical; debating "Terraform vs Pulumi" in the abstract without them just becomes a preference argument.

Comparison

What tips the recommendation in practice

• If the team has no strong Kubernetes investment yet, Crossplane's continuous-reconciliation advantage isn't free: adopting it means adopting Kubernetes as infrastructure for infrastructure, a bigger commitment than the reconciliation benefit alone justifies for most platforms.
• If provider coverage for a specific, load-bearing service is the actual blocker, a niche managed service only has a mature cloud-native or Terraform provider, not a Pulumi or Crossplane one yet, that single gap can decide the whole platform's tooling, since running two different IaC tools for two different resource types is its own maintenance cost.
• If the team is already writing the application in TypeScript or Python and wants infra tests to run in the same CI job with the same test framework as the app, that's Pulumi's actual differentiator, the shared tooling and testability, not "the language is nicer."
• Cloud-native templates earn a place even in a multi-cloud platform for the narrow slice of resources that are genuinely cloud-specific, where the vendor's own template format gets day-one support for a new feature before any third-party provider catches up; that's a deliberate, scoped exception, not a sign the whole platform should move to vendor templates.

Worked example
Applying the axis to one concrete requirement: the platform needs a managed Kubernetes cluster on both AWS (EKS) and GCP (GKE), plus continuous enforcement that no cluster ever drifts from its baseline node count and network policy.

• Provider coverage: both EKS and GKE have mature, long-standing Terraform providers; this doesn't rule anything out yet.
• Continuous drift correction: this is the requirement that actually narrows the field, since "no cluster ever drifts" is a continuous-reconciliation ask, not a periodic-plan ask, which points toward Crossplane, or a Terraform-plus-a-separate-reconciler-loop hybrid, over plain Terraform.
• Team skill: if the platform team already operates Kubernetes clusters as their day job, Crossplane's operational model, another set of controllers to run, is a marginal addition, not a new discipline; that tips the recommendation to Crossplane for this specific requirement, while Terraform still provisions the surrounding account-level resources, networking, IAM, that don't need continuous reconciliation.

This is the actual decision process: none of the four tools wins as a general answer, the specific requirement, continuous drift correction here, determines which axis matters most.

Trade-offs & pitfalls

• Running two tools, Terraform for foundational resources and Crossplane for continuously-reconciled workload resources, for example, is a legitimate hybrid, but it's an explicit architectural decision with its own coordination cost, not a default to reach for without a specific requirement driving it.
• Pulumi's expressiveness, real loops, conditionals, functions, is also its biggest risk: it's easy to write infra code that's hard to review because the actual resources created depend on runtime logic, not something visible from reading the file the way a Terraform for_each mostly is.
• Crossplane's continuous reconciliation means an unintended manual change gets reverted automatically, a feature until someone makes an emergency manual fix during an incident and Crossplane quietly reverts it a few minutes later; break-glass procedures need to account for that.
• Cloud-native templates being excellent for their own cloud is real, but it's also the lock-in mechanism; a platform leaning on them for anything beyond the narrow cloud-specific exception has quietly opted out of the multi-cloud parity goal the question started with.

Direct answer

When a security or ops team resists a proposed architecture change and insists on stricter controls or the status quo, don't argue that they're wrong. Treat their risk concern as a real constraint to engineer around, and negotiate a bounded path (compensating controls, a scoped and timeboxed exception, defined monitoring and rollback) that earns approval without asking them to accept the full, permanent risk up front.

Structured elaboration

Negotiating a bounded exception:

1. Listen and clarify the specific risk. Meet the resistant team and document exactly which control they believe would be violated and what residual risk they'd accept, before proposing anything.
2. Run a focused risk assessment scoped to that specific control, not the whole proposal.
3. Propose compensating controls mapped to that exact risk: segmentation, encryption in transit, least-privilege identities, tuned detection rules, whatever addresses the named concern.
4. Offer a timeboxed, limited-scope pilot: one workload, one region, a defined window, with explicit rollback criteria and no automatic expansion.
5. Define monitoring and a governance path: who reviews it, on what cadence, and what the escalation and sign-off chain looks like.
6. Keep a rollback plan ready, not just a forward plan.

This generalizes past pure technical controls. When the resistance is to a business or pricing model change rather than a security control (for example, a finance team resisting a new contract term), the tactic doesn't change: propose a bounded compromise or a timeboxed pilot with defined success criteria and an exit path, rather than asking for the permanent change up front.

Worked example

Situation. At a B2B platform, the architecture team proposed moving a batch reporting service from a fully isolated on-prem network segment to a shared cloud VPC, to cut latency and cost. The client's security team refused, citing existing segmentation policy.

Stakes. The current setup was measurably slower and more expensive to operate, but security's sign-off was a hard gate, and escalating over their head would have burned the relationship for every future proposal.

The influence moves.

• Met with security leads first to get the specific control at stake; the real blocker turned out to be network segmentation policy, not the data-residency concern initially assumed.
• Ran a focused risk assessment scoped only to that control, not the whole migration.
• Proposed compensating controls mapped to it: a dedicated micro-segment for the new service, mutual TLS to the rest of the VPC, and short-lived service credentials instead of long-lived ones.
• Offered a timeboxed pilot: one non-critical reporting workload, one region, 60 days, with an explicit rollback trigger tied to any anomalous traffic alert.
• Defined the monitoring security would get during the pilot: SIEM-visible logs, a named point of contact, a weekly review, so their sign-off wasn't an act of faith.

Resolution. Security approved the scoped pilot. When it cleared review with no incidents, the exception expanded to additional workloads under the same monitoring terms.

What a senior candidate does differently. Doesn't treat the security team's "no" as an obstacle to route around; treats their specific risk objection as the actual spec to engineer against, and doesn't ask for the full, permanent change until the bounded version has already produced evidence.

Trade-offs and pitfalls

• Compensating controls have real operational cost (more logging, tighter IAM, monitoring overhead). If that cost isn't acknowledged and budgeted, the pilot degrades over time and becomes the next audit finding.
• A scoped exception with no expansion or sunset review quietly becomes a permanent, ungoverned exception, worse than either the original status quo or a properly approved change.
• Escalating past the resistant team can work once, but it costs the relationship. Reserve it for cases where the objection is provably not risk-based (organizational turf, not risk), not as a default first move.

Where you measure latency (server-side versus client-side) changes what the number actually represents, and conflating the two, or trusting either blindly, produces an SLO that looks precise but measures the wrong thing.

Structured elaboration

Server-side instrumentation measures only the time the backend itself spends processing a request; it's cheap, low-variance, and easy to instrument consistently, but it is BLIND to everything that happens on the network between the client and the server (DNS lookup, TLS handshake, queueing, actual transit time), which for many users is a large fraction of perceived latency. Client-side instrumentation captures the full, true user-perceived latency, but introduces its own noise: clock skew between client devices (a client's own clock can be seconds off, corrupting any timestamp-difference calculation done client-side without a server-synchronized reference), highly variable network conditions across users, and sampling bias if only a subset of clients (e.g. those with a certain browser or app version) report telemetry at all.

Worked example

A service reports a server-side p95 of 80ms, comfortably inside its SLO, while client-side RUM data shows real users experiencing p95 latency of 450ms; the gap is almost entirely network transit and client-side rendering time invisible to the server-side number. An SLO built only on the server-side figure would report perfect health while the actual user experience is materially worse, which is exactly why it matters to declare, explicitly, which of the two the SLO is based on, and why most mature setups track both and set the CUSTOMER-facing SLO on the client-observed number, using the server-side one as an internal diagnostic signal.

Trade-offs and pitfalls

Histogram bucket placement compounds this problem: if buckets are chosen assuming server-side-scale latencies (fine-grained near 50-100ms) but applied to client-side data with true tails out past 1 second, the histogram's percentile estimate near p95/p99 becomes inaccurate because there aren't enough bucket boundaries where the real tail actually lives. Sampling client-side telemetry (common for cost reasons at very high traffic) also needs to be checked for bias: if only users on a fast, stable network path are more likely to successfully report their client-side telemetry (e.g. because a flaky connection also drops the telemetry beacon), the sampled data will look artificially fast, systematically underestimating the true tail.

Approach: aggregate entries by job_id, treat each log entry as an attempt. For success rate use count(successful attempts) / total attempts per job. For average duration, compute durations only for attempts with valid end_ts; treat missing end_ts as in-progress — exclude them from average but count them in attempts. Optionally, if end_ts missing and start_ts older than a cutoff (e.g., 24h), treat as failed/stale — shown as configurable.

from datetime import datetime, timedelta
from typing import Iterable, Dict, Any
import math

ISO = "%Y-%m-%dT%H:%M:%S"  # assume seconds precision; use dateutil.parser for robust parsing

def compute_job_metrics(logs: Iterable[Dict[str, Any]], stale_threshold: timedelta = None):
    """
    logs: iterable of JSON-like dicts:
      {"job_id","worker_id","start_ts","end_ts","status"}
    stale_threshold: if provided, entries with end_ts==None and now-start_ts > threshold are counted as failed attempts
    Returns dict job_id -> {"success_rate": float, "avg_duration_s": float or None, "attempts": int}
    """
    from collections import defaultdict
    now = datetime.utcnow()
    stats = defaultdict(lambda: {"attempts":0, "successes":0, "durations":[], "inprogress":0})
    
    for e in logs:
        jid = e.get("job_id")
        if jid is None:
            continue
        stats[jid]["attempts"] += 1
        if e.get("status") == "ok":
            stats[jid]["successes"] += 1
        start = None
        try:
            start = datetime.fromisoformat(e["start_ts"])
        except Exception:
            # malformed start -> skip duration but count attempt
            start = None
        end_ts = e.get("end_ts")
        if end_ts:
            try:
                end = datetime.fromisoformat(end_ts)
                dur = (end - start).total_seconds() if start else None
                if dur is not None and dur >= 0:
                    stats[jid]["durations"].append(dur)
            except Exception:
                pass
        else:
            stats[jid]["inprogress"] += 1
            if stale_threshold and start and (now - start) > stale_threshold:
                # treat as failed/stale attempt
                pass  # already counted as attempt; you might decrement or mark differently depending on policy
    
    result = {}
    for jid, s in stats.items():
        avg = (sum(s["durations"]) / len(s["durations"])) if s["durations"] else None
        success_rate = s["successes"] / s["attempts"] if s["attempts"] else 0.0
        result[jid] = {"success_rate": success_rate, "avg_duration_s": avg, "attempts": s["attempts"], "inprogress": s["inprogress"]}
    return result

Key points:

• We treat each log as an independent attempt (retries included).
• Missing or malformed end_ts: excluded from duration average; optionally flagged as in-progress or stale using stale_threshold.
• Success rate counts status field regardless of end_ts presence.
• Use robust ISO parsing (dateutil.parser.parse recommended in production).
  Complexity: O(n) time, O(m) memory for m distinct jobs. Edge cases: duplicate logs, clock skew (negative durations), malformed timestamps — handled by skipping durations and logging/alerting in real system.

Situation & goal: Validate that an optimization reduces CPU usage by ≥10% without increasing P99 latency (no regression). We'll run a randomized online experiment (canary/A–B) with clear instrumentation, stats plan, and rollback criteria.

Control & treatment:

• Control: current production binary/config on group A.
• Treatment: optimized version on group B.
• Traffic split: 50/50 (or weighted if riskier). Randomly assign requests/hosts.

Warmup & duration:

• Warmup: 10 minutes after rollout per host to let JIT/caches/GC stabilize (or N warmup requests/containers).
• Run for at least the time needed to collect planned sample size (see below); ensure coverage of typical traffic patterns (include peak window, at least 1 full business day).

Metrics to collect (sampled at 1-min granularity and per-request traces):

• Primary:
  • Mean CPU% per host (1-min aggregates)
  • P99 request latency (end-to-end) per minute and raw traces
• Safety: error rate (5xx), throughput (rps), GC/pause metrics, queue lengths, instance churn
• Context: request mix, payload size, instance type

Sampling unit & sample size (power calc example):

• Unit: 1-minute host-level CPU% aggregates. Baseline mean μ0=60%, σ≈8% (estimate from monitoring). Want Δ = 10% of μ0 = 6% absolute reduction.
• Two-sided t-test, α=0.05, power=80% (β=0.2):
  n ≈ 2*(Zα/2+Zβ)^2σ^2/Δ^2 = 2(1.96+0.84)^2*8^2/6^2 ≈ 28 minutes per group (hosts × minutes). If using fewer hosts, extend minutes; if CPU variance higher, increase n. Capture more for safety (recommend ≥60 samples per group).
• P99: treat as non-inferiority check. P99 estimates have heavy tails; collect many requests (recommend ≥100k requests per group) and use bootstrap to estimate CI.

Statistical tests:

• CPU: two-sample t-test (or Welch’s t-test) on minute-aggregates; report mean difference and 95% CI. Success if mean CPU reduction ≥6% and p<0.05.
• P99 latency: non-inferiority test using bootstrap CIs on difference (treatment - control). Define non-inferiority margin (e.g., +5ms or +2% relative). Success if upper bound of 95% CI ≤ margin.
• Also check error rates (chi-square test) and throughput stability.

Multiple comparisons & robustness:

• Pre-register primary/secondary metrics. Adjust for multiple testing (Bonferroni or hierarchical testing) if needed.
• Run sensitivity: stratify by region, instance type, request path.

Rollback / launch criteria:

• Immediate rollback during experiment if:
  • Error rate increases >0.1 percentage points or by >50% relative
  • P99 increases beyond non-inferiority margin in two consecutive 5-minute windows
  • Throughput drops or instance crashes occur
• Post-experiment promotion if:
  • CPU mean reduction ≥10% with p<0.05
  • P99 non-inferiority satisfied
  • No safety metric regressions and SLOs not violated for 24-hour extended canary

Observability & runbook:

• Dashboards for CPU, P99, errors, GC, and hosts; alerting on rollback thresholds; automated rollback pipeline to flip traffic and deploy previous version.

This plan balances statistical rigor with operational safety; adjust σ, Δ, and sample unit based on real telemetry and repeat or increase sample size if P99 variance is high.

Direct answer
When the root cause lives in a service another team owns, my first move is getting the right person from that team engaged directly and fast, usually by paging their on-call rather than routing through a manager, and then working in parallel rather than blocking: I keep making progress on whatever I can control while they investigate their side.

Structured elaboration

• Get the right person, not just any person. Page the owning team's on-call directly if your investigation clearly points at their service, rather than escalating through management layers that add delay without adding expertise.
• Be specific about what you need from them. Rather than a vague 'something's wrong with your service,' share exactly what you've observed and why you believe the root cause is there, which lets them start from your findings instead of re-deriving them from scratch.
• Work in parallel, not sequentially. While the owning team investigates their side, continue anything you can independently do on your side (further mitigation, additional monitoring, keeping stakeholders updated), rather than sitting idle waiting for their update.
• Don't take over their system without context. Even if you technically have the access to poke at their service directly, doing so without their domain knowledge risks causing a second problem; the better move is close collaboration, not unilateral action on a system you don't own.
• Don't silently wait either. If you've reached out and haven't heard back within a reasonable window given the severity, escalate again rather than assuming they're already on it.

Worked example
An incident's root cause traces to a shared authentication service owned by a different team. Rather than waiting for a formal handoff process, the responder directly pages that team's on-call with specific findings ('auth requests from our service are timing out starting at 14:02, correlating with your deploy at 13:58'), which lets the other team's engineer start investigating their deploy immediately rather than starting from scratch. While waiting, the original responder adds a client-side retry with backoff on their own service as a partial mitigation, something within their own control, rather than being fully blocked on the other team's fix.

Trade-offs and pitfalls
The most common failure here is silently waiting on the other team without actively escalating, which can leave an incident stalled far longer than necessary if that team is slow to notice or prioritize it. The opposite failure, someone outside the owning team taking matters into their own hands and directly modifying a system they don't fully understand, risks introducing a second, unrelated incident on top of the first. The right balance is proactive, specific engagement paired with continuing to make progress on what you do control, rather than either extreme.

Requirements and constraints:

• Detect nodes that are intermittently or persistently NotReady/have failing health checks without causing excessive churn.
• Safe automated remediation: cordon/drain, reprovision/replace, verify, and rejoin.
• Avoid cascading failures and respect cluster capacity/SLOs.

Detection

• Use Node Problem Detector + kubelet health, plus Prometheus alerts for kube_node_status_condition{condition="Ready"} and node-exporter metrics (heartbeats, disk, network, cpu).
• Define two tiers: transient (short blips) vs persistent (thresholds). Example: mark persistent if NotReady > 3m within last 10m OR 3 kubelet restarts in 30m.
• Enrich with cloud provider VM status and system logs (journalctl) before acting.

Remediation workflow

1. Pre-checks: verify alerts, check node drainability (pods with localStorage, PDB impacts), available capacity in cluster.
2. Cordon node: kubectl cordon <node>.
3. Graceful drain: kubectl drain --ignore-daemonsets --delete-local-data --force --grace-period=<sensible> <node>, respecting PDBs by waiting or using disruption budgets.
4. Health triage hook: attempt in-place fixes (restart kubelet, restart problematic services, remove bad mounts). Run automated diagnostics script and report results.
5. If diagnostics fail or repeated failures: reprovision:
   • For cloud: terminate/replace instance using autoscaling group/Instance Template or API; boot new node, ensure bootstrap (cloud-init) installs correct kubelet and joins.
   • For on-prem: trigger PXE reinstall or provision replacement hardware, or recreate VM via virtualization API.
6. Post-provision validation: wait for node Ready, run smoke tests (pull small image, schedule probe pods), remove cordon when healthy.

Safety and avoiding cascading failures

• Rate limit repairs: global concurrent-repair limit (e.g., max 5% of nodes or N nodes) and per-AZ limit.
• Backoff and retry policy: exponential backoff with escalation to human if >2 automated attempts.
• Respect cluster capacity: check cluster utilization and error budget—if insufficient capacity, defer repair and escalate.
• Canary and staged rollout: run repairs first in non-production AZs/nodes, or mark a pool as repair-candidate.
• Panic mode: if many nodes fail rapidly, pause automated repairs, increase observability, run coordinated manual response.
• Audit & alerting: log all actions, create tickets, notify on-call with remediation summary.

Metrics and observability

• Track MTTR, repair success rate, number of automated vs manual escalations, and impact on SLOs.
• Provide dashboards for ongoing failure patterns to drive root-cause fixes.

This design balances automation speed with safety—automate common fixes and replacements, but gate actions with capacity checks, rate limits, diagnostics, and human escalation to avoid causing or worsening outages.