Apple Site Reliability Engineer (Senior Level) - Comprehensive Interview Preparation Guide

Direct answer

Enforcing "at most R requests per rolling T-second window" per user, at millions-of-users scale, comes down to picking how much state you keep per user and how precisely that state approximates a true rolling window. A fixed counter per window is cheapest ($O(1)$ per user) but allows up to 2R requests to slip through right across a window boundary; a rolling log of exact timestamps is perfectly precise but costs $O(R)$ per user; a token-refill (token-bucket) scheme and a two-counter sliding-window approximation both give $O(1)$ per-user memory with only small, bounded imprecision near boundaries, which is why they are the usual production choice at this scale.

Structured elaboration

Three structural approaches compared

Why a fixed counter's imprecision happens specifically at boundaries

If the window resets every T seconds, a user can send R requests in the last instant of one window and another R in the first instant of the next: both windows individually respect the R-per-window limit, but a true rolling T-second view sees up to 2R requests in a span far shorter than T. The rolling log fixes this by definition (it only ever counts requests actually within the trailing T seconds), at the cost of storing up to R timestamps per user. The two-counter and token-bucket schemes recover most of the precision of the rolling log at the memory cost of the fixed counter, by using the previous window's count as a fading estimate of "how many of those requests are still within the trailing T seconds," rather than discarding it entirely at the reset boundary.

Sharding for millions of users

Regardless of which per-user scheme is chosen, per-user state should be sharded by a hash of the user ID across many limiter nodes or partitions, so no single node holds all users and no single lock serializes all traffic. Route each user consistently to the same shard (consistent hashing keeps this stable as shards are added or removed) so all requests for one user hit the same counter state, and evict counters for inactive users on a time-to-live (TTL, an expiration timer after which an idle entry is dropped) so memory tracks active users rather than the full lifetime user base.

Worked example

The two-counter sliding-window approximation, concretely:

class SlidingWindowCounter:
    """
    Approximate sliding-window limiter: O(1) memory per user (two counters),
    O(1) time per check. Weights the previous fixed window by how much of it
    still overlaps the current rolling window.
    """

    def __init__(self, limit: int, window_seconds: float):
        self.limit = limit
        self.window = window_seconds
        self.curr_window_start = 0.0
        self.curr_count = 0
        self.prev_count = 0

    def _roll_window(self, now: float) -> None:
        elapsed = now - self.curr_window_start
        if elapsed >= 2 * self.window:
            self.prev_count = 0
            self.curr_count = 0
            self.curr_window_start = now
        elif elapsed >= self.window:
            self.prev_count = self.curr_count
            self.curr_count = 0
            self.curr_window_start += self.window

    def allow(self, now: float) -> bool:
        self._roll_window(now)
        elapsed_in_curr = now - self.curr_window_start
        overlap = max(0.0, (self.window - elapsed_in_curr) / self.window)
        estimated = self.prev_count * overlap + self.curr_count
        if estimated + 1 > self.limit:
            return False
        self.curr_count += 1
        return True


if __name__ == "__main__":
    limiter = SlidingWindowCounter(limit=5, window_seconds=1.0)
    # 5 requests at t=0.0 fill the first window
    results_first = [limiter.allow(0.0) for _ in range(5)]
    # a 6th request in the same window must be rejected
    sixth = limiter.allow(0.05)
    # at t=1.5 we are 50% into the new window; the estimate blends 50% of the
    # old window's 5 requests (2.5) with 0 new ones, so 2.5 + 1 <= 5 fits
    seventh = limiter.allow(1.5)
    print(results_first, sixth, seventh)

Running this prints:

[True, True, True, True, True] False True

Five requests at t=0.0 fill the first one-second window exactly to the limit of 5. A sixth request at t=0.05 (still inside that same window) is rejected, since the count is already at 5. At t=1.5, half a second into the next window, the estimate blends 50% of the previous window's 5 requests ($5 \times 0.5 = 2.5$) with the 0 requests so far in the current window: $2.5 + 1 \le 5$, so the seventh request is allowed. This is the boundary smoothing a fixed counter does not give you: a fixed counter would have simply reset to 0 at t=1.0 and allowed 5 fresh requests immediately, permitting the same 2x-at-the-boundary burst described above.

Complexity

Per-user check and update: $O(1)$ time for the fixed counter, token bucket, and two-counter sliding window; $O(\log R)$ or $O(1)$ amortized (averaged over a sequence of operations) for the rolling log depending on whether old timestamps are pruned lazily or with a deque. Per-user memory: $O(1)$ for the first three approaches, $O(R)$ for the rolling log.

Edge cases

• A burst exactly at a window boundary is the scenario every design above is explicitly trying to bound; state which imprecision (if any) your chosen scheme accepts.
• Clock skew between distributed limiter nodes can make the "current time" disagree slightly across shards; keep window arithmetic tolerant of small skew rather than assuming a perfectly synchronized clock.
• A user with no prior activity needs a cold-start default (empty counters, full token bucket) rather than an error.
• Inactive users must be evicted (TTL-based) so memory does not grow without bound across millions of distinct users who each showed up once.

Trade-offs & pitfalls

The common wrong turn is presenting the rolling log as strictly "the correct one" without acknowledging its $O(R)$-per-user memory cost: at millions of users and even a modest R, that can dwarf the memory of the $O(1)$ approaches by orders of magnitude, which is exactly why production rate limiters favor the token-bucket or sliding-window-counter approximation instead. A second common gap is proposing a single global lock or single-node counter for correctness: that eliminates any cross-shard race but reintroduces the exact contention problem millions of distinct users at high volume were meant to avoid; sharding by user ID sidesteps this because a fully correct answer only needs to be correct per user, not globally serialized. A third pitfall is conflating the token bucket's smoothing behavior with the sliding window's counting behavior: a token bucket happily allows a burst up to its full capacity the instant it has accumulated enough tokens, which is a different guarantee from "at most R in any rolling T-second window," and the two should not be presented as interchangeable without naming that difference.

Requirements & constraints:

• Allow temporary emergency release bypasses for on-call SREs during outages.
• Every bypass must be fully auditable, require immediate post-facto justification, and be time-limited.
• Enforce RBAC, 2-person validation where appropriate, and automated notifications/alerts.
• Prevent silent/unaudited use of bypass.

High-level architecture:

• Approval Service (centralized workflow API)
• RBAC/Identity Provider (IdP) with groups & attributes
• Release Orchestrator (CI/CD gate that enforces approvals)
• Immutable Audit Store (WORM or append-only ledger + cryptographic signing)
• Notification Engine (email, Slack, PagerDuty)
• SIEM/Alerting & Postmortem Ticketing system integration

Core components & responsibilities:

1. RBAC model

   • Roles: Developer, Release Approver, On-Call SRE, Emergency Approver, Compliance Auditor.
   • Attributes: on_call=true, escalation_level, rotation_id.
   • Policies: Only On-Call SREs (with valid on_call attribute) can request emergency bypass. Emergency Approvers can grant overrides; Compliance Auditors can only read.
   • Principle of least privilege; default deny for bypass endpoints.

2. Emergency override flow (enforced by Release Orchestrator + Approval Service)

   • Step A: On-call creates a “Break-Glass Request” via the Approval Service. Must include incident ticket ID, justification text, impacted services, and remediation plan stub.
   • Step B: Requester must authenticate with IdP + step-up MFA and attach ephemeral session token from IdP (short TTL).
   • Step C: Approval Service creates a signed override token (JWT) with strict claims: requester id, approver id(s) (if pre-approved), scope (service, pipeline stage), TTL (e.g., 15 min), nonce, request_id, and mandatory post-facto due-by timestamp.
   • Step D: If policy requires two-person rule, a second Emergency Approver must acknowledge within X minutes; otherwise auto-expire.
   • Step E: Release Orchestrator accepts only signed override tokens from Approval Service and validates nonce + signature + TTL + scope; executes release.
   • Step F: Approval Service writes entire request, attached MFA evidence, signed override token and every state transition into Immutable Audit Store (WORM or append-only DB with SHA-256 chaining and offsite backups). Also emit real-time alerts to Notification Engine and SIEM.

Data flow / guarantees against silent use:

• Enforce that the CD pipeline refuses to run bypassed actions unless presented an override token signed by Approval Service. Tokens are single-use (nonce) and recorded upon redemption into Audit Store.
• Immutable store uses chained hashes/timestamps and cryptographic signatures so any tampering is detectable. Optionally ship hashes to an external ledger (e.g., blockchain anchor or external KMS-signed checkpoint).
• All actions (request, MFA evidence, token issuance, token redemption, execution logs) are correlated by request_id and stored atomically. SIEM monitors for any execution in Release Orchestrator lacking a matching audit entry and raises immediate alerts / auto-rollbacks.

Time-limited overrides & revocation:

• Tokens include short TTL (e.g., 10–30 minutes) and single-use nonce.
• Immediate revocation path: Approval Service can push a revocation to Release Orchestrator and orchestrator checks revocation list before each critical step.
• Auto-expiration and enforced re-authentication for extended windows.

Notifications & compliance:

• On issuance: immediate notifications to on-call rotation, release approvers, engineering manager, and compliance channel (Slack, PagerDuty, email).
• On redemption: second wave of alerts with execution logs and links to artifact checksums.
• Post-facto requirement: system enforces a mandatory “justification completion” form within N hours; until completed, the requester is blocked from creating future emergency overrides and the event is escalated to Compliance Auditor. This is enforced by Approval Service policy checks.
• Daily digest and automated generation of post-incident ticket in ticketing system if justification unresolved.

Observability, monitoring & SLO considerations:

• Monitor counts of emergency bypasses per service/period; alert on anomalies (rate thresholds).
• Track time from bypass to justification; incorporate into on-call KPIs and blameless postmortems.
• Integrate with SLO error budgets to limit frequency of emergency releases.

Trade-offs & mitigations:

• Latency vs safety: Short TTLs increase safety but can slow recovery. Mitigate with pre-authorized emergency approvers and clear escalation paths.
• Central Approval Service is a single point of failure: make it highly available, replicated, with local failover modes that still log to immutable store (buffered writes).
• Usability vs friction: use step-up MFA and single-use tokens to avoid risk; invest in streamlined UX + templates to reduce cognitive load during incidents.

Why this guarantees no silent bypass:

• CD pipeline enforces cryptographic token validation and single-use nonce checks; any execution without matching audit entry triggers automated alerts/rollback.
• All artifacts and state transitions are immutably logged and externally anchored; tampering is detectable.
• Post-facto enforcement (blocking future overrides, managerial escalation) ensures social and process accountability in addition to technical controls.

Implementation roadmap:

1. Define RBAC attributes & policies in IdP; add attributes for on_call rotations.
2. Implement Approval Service with signed token issuer, single-use nonce store, WORM audit writes, and revocation list.
3. Integrate Release Orchestrator to validate overrides and perform single-step atomic redemption logs.
4. Hook Notification Engine and SIEM; implement post-facto enforcement and ticket automation.
5. Run drills and audits; tune TTLs and policies based on ops experience.

When two services disagree about a transaction's outcome, the safest default for computing a reliable SLI is to trust whichever service is the AUTHORITATIVE source of truth for that specific outcome, and treat any disagreement itself as a signal worth investigating, not just resolving and moving on.

Structured elaboration

Reconciliation rules: identify which service is genuinely authoritative for the specific claim in question (e.g. for a payment transaction, the payment processor's own confirmation is usually more authoritative than an internal service's inferred success/failure based on an HTTP response, since the internal service could have logged "success" based on receiving a request while the actual downstream processing failed asynchronously, or vice versa); where no single service is unambiguously authoritative, define an explicit, DOCUMENTED tiebreaker (e.g. "if A and B disagree and neither is authoritative for this specific field, default to the more conservative outcome, i.e. treat as failed, since falsely counting a transaction as successful when it wasn't is worse than the reverse for this use case"). Instrumentation changes to reduce future inconsistency: propagate a shared, canonical transaction-outcome field written by whichever service IS authoritative, and have all other services read (not independently re-derive) that field for their own logging and SLI computation, rather than each service independently inferring and logging its own opinion about the same event.

Worked example

For an e-commerce checkout where service A (the order service) logs success based on receiving an acknowledgment, and service B (the payment processor integration) logs failure because the payment actually failed asynchronously after A's acknowledgment was sent: B is authoritative here, since payment success/failure is squarely B's domain of truth, and A's SLI computation should be corrected to READ B's authoritative outcome rather than relying on its own earlier, now-known-to-be-wrong acknowledgment-based inference. Going forward, A's logging is changed to write "acknowledged" (not "success") for A's own internal event, and to derive its own "transaction succeeded" SLI directly from B's authoritative confirmation, closing the specific inconsistency this incident revealed.

Trade-offs and pitfalls

Defaulting to "trust whichever service reports first" or "trust whichever reports success" (an easy but wrong shortcut under time pressure) systematically biases the SLI toward looking better than reality, since a service that reports optimistically and early will systematically override a later, more accurate failure signal from a slower but more authoritative source. It's also worth treating every discovered A/B disagreement as an OPPORTUNITY to fix the underlying instrumentation gap that allowed two services to develop divergent views of the same event in the first place, rather than treating reconciliation as a one-off patch applied after the fact each time a new disagreement surfaces.

Direct answer
The TCP three-way handshake establishes a reliable connection before any data flows: the client sends a SYN, the server replies with a combined SYN-ACK, and the client finishes with an ACK. Its job is to let both sides agree on starting sequence numbers and confirm that both directions of the path actually work before committing application data to the wire.

Structured elaboration

1. SYN: the client picks an initial sequence number (ISN, essentially a large pseudo-random 32-bit number) and sends a segment with the SYN flag set and that sequence number. The client moves to the SYN-SENT state.
2. SYN-ACK: the server, if it's listening on that port, picks its OWN initial sequence number, and replies with a segment that has both the SYN flag set (announcing the server's own sequence number) AND the ACK flag set (acknowledging the client's sequence number + 1). The server moves to the SYN-RECEIVED state.
3. ACK: the client acknowledges the server's sequence number + 1 with a plain ACK segment. Both sides now move to ESTABLISHED, and either side may now send data.

Why three steps rather than two: TCP needs BOTH sides' sequence numbers acknowledged, since TCP is full-duplex (both directions need independent sequence tracking). A two-way handshake could confirm only one direction; the third message is what confirms the client's original SYN actually arrived, closing the loop for the client's own sequence space.

Worked example
Suppose a client opens a TCP connection to a web server on port 443. The client sends SYN, seq=1000. The server responds SYN, ACK, seq=5000, ack=1001 (acknowledging the client's SYN by number+1). The client responds ACK, seq=1001, ack=5001. From this point, the client's next data byte will carry sequence number 1001, and the server's next data byte will carry sequence number 5001; each side tracks the OTHER side's sequence space independently via the ACK field of every following segment.

Trade-offs & pitfalls
A frequent mistake is describing the handshake as three round trips; it's actually one and a half round trips of latency, because the SYN-ACK piggybacks the server's SYN onto its ACK of the client's SYN. This is also exactly why TCP always incurs at least one round trip of setup latency before any data can flow, which is the whole motivation behind newer mechanisms like TCP Fast Open that try to send data alongside the very first SYN.

This architecture has three single points of failure once you look past the app tier: the load balancer, and the primary database, are both singletons that the whole request path depends on; the app-server tier looks redundant on paper (three instances) but is only actually redundant if those three instances sit in different fault domains, so it's worth confirming rather than assuming.

Walking the diagram

flowchart LR
  U[Users] --> LB[Load Balancer\nSINGLE instance]
  LB --> A1[App Server 1]
  LB --> A2[App Server 2]
  LB --> A3[App Server 3]
  A1 --> DB[(Primary DB\nSINGLE writer)]
  A2 --> DB
  A3 --> DB

Load balancer (single instance). Every request passes through it, so its failure is a total outage regardless of how healthy the three app servers are behind it. Mitigation: run an active-active pair of LB nodes behind a floating IP or DNS-based failover, or use a managed cloud load balancer where the provider owns that redundancy. Cost: a small amount of extra infrastructure and configuration; the bigger cost is usually operational (health-check tuning, avoiding split traffic during LB failover), not dollars.

App servers (three instances, conditionally redundant). If all three run in the same rack, same availability zone, or share an underlying host, they aren't actually independent, a single power or network event takes out all three at once. Mitigation: spread them across at least two, ideally three, availability zones and confirm the LB health-checks each independently and routes around a dead one automatically. Cost: cross-AZ data transfer costs and slightly higher latency on some requests; this is usually the cheapest SPOF to fix since it's mostly a placement decision, not new infrastructure.

Primary database (single writer, no replica). This is the highest-blast-radius SPOF: if it fails, every write path is down and, depending on the failure mode, recent unreplicated data can be at risk. Mitigation: add at least one synchronous or semi-synchronous replica in another AZ with automated failover (promote-on-failure), plus continuous backups for protection against logical corruption that replication alone wouldn't catch. Cost: this is the most expensive fix of the three, both in infrastructure (a standing replica) and in write latency if replication is synchronous.

Worked example: quantifying the SPOFs

Assume, for illustration, per-component annual availability of 99.95% for the load balancer, 99.9% for each app-server instance, and 99.9% for the database, all figures pinned as inputs for this calculation, not measured facts about any real vendor.

The three-server app tier, if truly independent across fault domains, only fails when all three fail simultaneously, so its unavailability multiplies:

That's an app-tier availability of essentially 99.9999999%, negligible. But the LB and DB are each in series with the whole request path (either one being down takes the whole system down), so their unavailabilities add through multiplication of the availabilities:

That's about $525{,}600 \times (1 - 0.998500) \approx 788.4$ minutes of downtime per year, roughly the sum of the LB's own downtime (about 262.8 min/yr at 99.95%) and the DB's own downtime (about 525.6 min/yr at 99.9%), because the redundant app tier contributes essentially nothing to the failure budget while the two singletons dominate it completely. This is the concrete version of "fix the SPOFs first": no amount of extra app-server redundancy moves that 788-minute number until the LB and DB are addressed.

Trade-offs and pitfalls

The most common wrong turn is stopping at "add more app servers," which is the SPOF that's already effectively solved in this diagram and contributes the least to the real number above; teams do this because it's the cheapest, least disruptive change, not because it's the highest-leverage one. A second pitfall is fixing the database with synchronous cross-region replication by default: it does reduce RPO to near zero, but the added write latency (and reduced availability during a partition, since a strict quorum, requiring a majority of replicas to agree before a write is accepted, can block writes when too few replicas are reachable) is often the wrong trade for a service that would have been fine with an in-region synchronous replica and async cross-region for disaster recovery only. The same "look for the singleton" walk generalizes past this exact diagram: in a streaming ingestion pipeline the SPOF is usually a single partition leader or a schema registry with no standby; in an ML-serving stack it's a lone model server or a feature store with no fallback; and at a more abstract level, any shared control plane (service discovery, config store, secrets manager) or shared cache that every downstream service depends on is a SPOF even when nobody draws it on the diagram.

"-euo pipefail" is a common safety trio to make Bash scripts fail early and avoid silent errors.

• -e (errexit): exit immediately if a simple command returns non-zero. Helps catch failures early.
• -u (nounset): treat unset variables as errors; referencing one causes immediate exit. Prevents surprising empty-string behavior.
• -o pipefail: make a pipeline return the exit status of the rightmost failing command (instead of the last command). Prevents hiding failures in earlier pipeline stages.

Common pitfalls (commands that legitimately return non-zero):

• grep returns 1 when no match — often not an error for scripts that check existence.
• test/[ ] returns 1 when condition false.
• kubectl get or psql may return non-zero for "not found" semantics you expect to handle.
• commands used only for side-effects (rm file || true) or status checks.

Examples and safe patterns:

set -euo pipefail

# Problem: grep no matches -> script exits
if grep -q 'needle' file; then
  echo found
fi

# Explicitly handle expected non-zero:
if ! grep -q 'needle' file; then
  echo "not found, continuing"
fi

# Use || true when non-zero is acceptable
rm /tmp/maybe || true

# Temporarily disable errexit for a command
set +e
cmd_that_may_fail
rc=$?
set -e

Best practices for SRE automation:

• Prefer explicit checks (if/then) instead of relying on exit codes.
• Use "|| true" or capture exit code when a command may legitimately fail.
• For pipelines that you expect to allow some commands to fail, inspect "${PIPESTATUS[@]}" or run parts separately.
• Use traps for cleanup: trap 'cleanup' EXIT
• Validate inputs early; check for required env vars with parameter expansion: : "${REQUIRED_VAR:?missing}"
• Keep scripts small and test them interactively with -x for debugging.

These options are valuable for reliability; use them with explicit handling where non-fatal failures are expected.

Direct answer
To know whether a postmortem process is actually working, track a small set of metrics on two levels: is the process itself being followed (leading indicators like action-item closure rate and time-to-close), and is it producing real outcomes (lagging indicators like incident recurrence rate and time between related incidents). Neither kind alone is enough: high process compliance with unchanged recurrence means the process is theater, and improving recurrence without process metrics gives you no early warning when things start slipping.

Structured elaboration
Useful metrics, split by what they tell you:

• Process health (leading): action-item closure rate within the committed deadline; median time from incident to a completed postmortem writeup; percentage of postmortems with at least one measurable, owned action item (a postmortem with zero action items is a red flag, not a sign nothing needed fixing); adoption rate, meaning the fraction of qualifying incidents that actually got a postmortem at all.
• Outcome (lagging): recurrence rate of the same or a closely related incident class; mean time between incidents in a given category; trend in overall incident severity over a quarter or two.
• Cultural signal (supporting): near-miss and self-reported-incident volume, and a periodic anonymized psychological-safety survey, since a process can look procedurally healthy while people quietly stop reporting things.

Collection should be mostly automatic: pull closure rates and time-to-close from whatever ticketing system tracks action items, rather than relying on manual reporting that decays over time. Report these on a regular cadence (monthly or quarterly) to both the engineering org and, in summary form, to leadership, since visibility is part of what keeps the process from quietly eroding.

Worked example
A team tracks action-item closure rate at 60% within the committed deadline and a database-related incident recurring three times in six months. Rather than treating these as separate facts, they cross-reference: two of the three recurring incidents trace back to the same never-closed action item from an earlier postmortem, which had been marked 'in progress' for four months with no owner actively working it. This tells the team the real problem isn't the postmortem process itself producing bad analysis, it's a downstream tracking gap: action items get created but nothing enforces follow-through. The fix is a lightweight escalation rule (any action item open past its deadline gets automatically flagged to the item owner's manager), and the team adds 'percentage of overdue action items escalated within a week' as a new leading metric to catch this earlier next time.

Trade-offs and pitfalls
A common failure is optimizing the metric instead of the outcome, for example closing action items quickly by scoping them down to something trivial just to hit a closure-rate target, which improves the number while leaving the real risk unaddressed. Guard against this by periodically auditing a sample of 'closed' items against whether the underlying incident class has actually stopped recurring, not just whether a ticket got marked done.

Direct answer

At two days of silence with a deadline approaching, keep working the problem in parallel on two tracks: escalate progressively (wider audience, shorter response window) instead of waiting indefinitely or jumping straight to someone's manager, and start a temporary workaround so your own deadline isn't hostage to someone else's response time.

Structured elaboration

1. Reconfirm the ask was clear before escalating. Silence sometimes means the original message was ambiguous or buried, not that it's being ignored. A quick, sharper re-send (what's needed, by when, what breaks if it slips) is worth trying before widening the audience.
2. Widen the channel and audience, not just the volume. Loop in a teammate of the owner's, or their tech lead, with a concise summary: what's blocked, since when, and what you need. This isn't going over anyone's head yet, it's making sure the request isn't sitting unseen in one inbox.
3. Escalate to management if there's still no response, framed around unblocking the work, not blaming the person: bring your own manager or a shared point of contact (like a PM) into a short, direct conversation rather than an open-ended thread.
4. Start a workaround in parallel, not sequentially after escalation: a mock, a stub, or a scoped assumption that lets you keep making progress while the real dependency gets resolved, clearly labeled as temporary so it doesn't quietly become permanent.
5. Close the loop afterward. Once unblocked, note what caused the delay (no on-call coverage, unclear ownership, a channel nobody monitors) so the same two-day silence doesn't repeat next time.

Worked example

Say another team owns a data pipeline, and a schema change they need to ship is blocking your dashboard launch, due in three days. You messaged the pipeline owner two days ago and got no reply.

1. Reconfirm: you send a sharper follow-up in the same thread: "Following up: I need the orders table schema change merged by Thursday EOD to hit our dashboard launch Friday. Anything blocking you on it, or should I loop in someone else?"
2. Widen: a few hours pass with no reply, so you message the pipeline team's tech lead directly (not a reply-all): "I've been blocked on the orders schema change since Monday and our Friday launch depends on it. Can you help me find the right person, or unblock it yourself?"
3. Escalate: by end of day, still nothing, so you bring it to your manager or a shared PM in a short conversation, not a long thread: "I've tried the owner directly and through their lead over two days with no response, and Friday's launch depends on this. Can you help get it unblocked?"
4. Workaround, run in parallel from day one: while those messages are going out, you build your dashboard against a stubbed version of the new schema (a local view with the expected new columns backfilled from sample data), clearly commented as temporary, so the launch timeline doesn't wait on the real merge landing.
5. Close the loop: once the schema change lands, you raise in the team retro that the pipeline team had no on-call coverage for urgent schema requests, and propose a shared "blocked on us" channel so a two-day silence doesn't happen again.

(The same five-step shape applies outside engineering: a designer blocked on a brand asset from marketing, or a QA engineer blocked on a test environment from infra, would reconfirm, widen, escalate, work around, and close the loop the same way.)

Trade-offs & pitfalls

• Pitfall: escalating too fast, before trying a second direct attempt, which can read as skipping over someone unnecessarily.
• Pitfall: waiting too long out of politeness, which puts your own deadline at risk and, in review, looks like you didn't flag a risk early enough.
• Pitfall: treating escalation and workaround as either/or. Doing them in parallel protects the deadline regardless of how fast the escalation resolves.
• Senior differentiator: framing every step (the re-send, the widened ask, the escalation) around getting unblocked, not around who's at fault, so the relationship with the owning team survives the deadline pressure.

Requirements & assumptions

• Baseline steady traffic: 1 RPS continuous.
• Bursts: to 1000 RPS, duration 2 minutes, N times/day (assume N=6 unless specified).
• Workload: single short HTTP request/response; avg execution CPU time = 50 ms; memory = 256 MB.
• Serverless provider pricing example (e.g., AWS Lambda):
  • Invocation cost = $0.20 per 1M requests ≈ $2.0e-7 per invocation.
  • GB-seconds = $0.0000166667 per GB-second.
  • Cold-start penalty = extra 200 ms added to first invocation latency when scaling new containers.
  • Provisioned concurrency cost = $0.0000041667 per GB-hour (example).
• VM cost example: single vCPU 2GB VM = $0.02/hr; can handle ~50 concurrent requests with 50 ms service time (approx); network, OS overhead ignored.
• Target metric: monthly cost (30 days) and expected p99 latency under observed bursts.

Model approach — serverless (no provisioned concurrency)

1. Capacity needed during burst: 1000 RPS * 50 ms service time = 50 concurrent executions. But to keep p99 low under spike ramp-up, concurrency needed ≈ 1000 * 0.05 = 50 concurrent containers.
2. Cold-start behavior: if scaling from baseline (1 RPS) to 1000 RPS quickly, many new containers created → a fraction f of requests incur 200 ms extra. Assume scale latency causes 80% of burst requests to hit cold starts on first burst second. Conservative: 30% of burst requests see cold start.
3. Monthly invocations = baseline + bursts:
   • Baseline: 1 RPS * 36002430 = 2,592,000
   • Bursts: 1000 RPS * 120 s * N30? No—N per day. For N=6/day: daily burst seconds = 6120=720s => per month 720*30=21,600s. Invocations during bursts = 1000 * 21,600 = 21,600,000
   • Total invocations ≈ 24,192,000
4. Serverless compute GB-seconds:
   • Each invocation uses 0.256 GB * 0.05 s = 0.0128 GB-s
   • Monthly GB-s = 24,192,000 * 0.0128 ≈ 309,657 GB-s
5. Serverless cost:
   • Invocation cost = 24.192M * $2.0e-7 ≈ $4.84
   • Compute cost = 309,657 * $0.0000166667 ≈ $5.16
   • Cold-start latency has no direct monetary cost but hurts p99.
   • Total ≈ $10/month (provider example; actual varies).

Serverless with provisioned concurrency sized to avoid cold starts during bursts

1. To avoid cold starts, provision concurrency = 50 concurrent containers. Provisioned GB-hours per month:
   • 50 containers * 0.256 GB * 24*30 = 50 * 0.256 * 720 = 9216 GB-hours
   • Cost = 9216 * $0.0000041667 ≈ $38.4
2. With provisioned concurrency, you still pay normal invocation + compute for execution (but you avoid many cold starts).
3. Total monthly ≈ base serverless ($10) + provisioned concurrency ($38) = ~$48/month.

VM-based provisioned capacity

1. Each VM handles ≈50 concurrent requests (50 ms service); to serve 50 concurrent need 1 VM. But to handle sustained 1000 RPS with headroom and p99, allow 20% buffer → 2 VMs.
2. Monthly VM cost: 2 VMs * $0.02/hr * 24*30 = 2 * 0.02 * 720 = $28.8
3. Add load balancer / autoscaling overhead (~$5) and utilization inefficiency due to baseline (low) → total ≈ $35-40/month.

p99 latency estimation

• Serverless without provisioned concurrency:
  • Baseline p99 ≈ function execution 50 ms + network = ~70 ms
  • During burst: many cold starts → assume 30% requests incur +200 ms ⇒ p99 dominated by cold starts. If 70 ms baseline, cold-start requests at 270 ms. p99 approximated ≈ 270 ms (if cold-starts concentrated in tail).
• Serverless with provisioned concurrency:
  • Cold starts eliminated for provisioned containers -> p99 ≈ 70–120 ms (small jitter).
• VM approach:
  • With 2 VMs and proper autoscaling, p99 ≈ 50 ms execution + queuing under peak. If capacity matches peak, p99 ≈ 70–100 ms. If underprovisioned, p99 spikes significantly.

Trade-offs & recommendation

• Cost: pure serverless cheapest ($10) but risk of high p99 during bursts due to cold starts. Provisioned concurrency raises cost ($48) but yields low p99.
• VM: steady cost similar to provisioned serverless ($30–40), predictable latency and control, but operational overhead (patching, scaling).
• For SRE with SLO p99 < 150 ms: provisioned concurrency or small VM pool recommended. If SLO allows p99 ≈ 300 ms, pure serverless acceptable and cheapest.

How to refine with real data

• Measure real cold-start rate and latency distribution, measured function memory vs CPU, burst ramp-up slope, and provider pricing. Use simulation (Poisson baseline + scheduled bursts) to compute tail percentiles more precisely.

Linux uses the OOM (Out‑Of‑Memory) killer when the system runs out of memory to pick a process to terminate and free memory. Selection is based on each process’s "badness" score: the kernel computes oom_score (0–1000) representing how expensive/appropriate it is to kill a process, considering RSS size, process age, and OOM adjustments.

Key controls and files:

• /proc/<pid>/oom_score — read-only current computed score.
• /proc/<pid>/oom_score_adj — writable integer in [-1000, +1000] to bias selection. -1000 prevents the process from being chosen (immune); positive values make it more likely to be killed. Use oom_score_adj (preferred) instead of deprecated oom_adj.
• /proc/sys/vm/overcommit_memory — controls allocation behavior:
  • 0: heuristic overcommit (default)
  • 1: always allow overcommit
  • 2: never overcommit; allocations fail if they exceed commit limit (safer but can break apps expecting allocations)
• /proc/sys/vm/overcommit_ratio — used when overcommit_memory=2 to set commit limit relative to RAM+swap.

How to protect a critical process:

• Set /proc/<pid>/oom_score_adj = -1000 (requires root) so the kernel will not pick it. For systemd services, use MemoryDenyWriteExecute? No—use systemd property OOMScoreAdjust=-1000 in service unit.
• In container/cgroup environments, set an appropriate oom_score_adj inside container or use cgroup v2 oom.group settings; ensure cgroup limits are set so the whole container isn't punished unpredictably.
• Reduce overcommit risk by setting vm.overcommit_memory=2 and tuning overcommit_ratio, or by increasing RAM/swap so the system is less likely to trigger OOM.

Trade-offs and caveats:

• Marking a process immune (-1000) can cause the kernel to kill other, possibly critical, system processes instead; you may push the system into instability if other processes are killed.
• overcommit_memory=2 prevents surprising allocation successes but can make some applications fail on allocation calls; it's safer for deterministic memory control but reduces flexibility.
• Relying on oom_score_adj is a last-resort guard; better long-term solutions are capacity planning, adding RAM/swap, memory limits per service (cgroups), OOM-aware app behavior, and good monitoring/alerts.

Practical steps:

• For a systemd service: add "OOMScoreAdjust=-1000" to the unit and restart.
• For a running PID (root): echo -1000 > /proc/<pid>/oom_score_adj.
• Also monitor memory usage, set cgroup memory limits, and adjust overcommit settings only after testing.