Comprehensive Interview Preparation Guide for Staff-Level Site Reliability Engineer (FAANG Standards)

Approach (brief)

• We rank candidates for the next on-call slot (one slot of duration slot_minutes) so that picking the top candidate reduces variance of cumulative on-call minutes across engineers while respecting availability/timezone constraints.
• Score each engineer by their "deficit" = (target_minutes - prev_minutes), scaled by timezone_weight and penalized if constraints make them less desirable. Higher score == more deserving.
• Target_minutes = (total_prev + slot_minutes) / n — the ideal equalized load after assigning this single slot (assumes we assign one slot). For selecting multiple slots, repeat greedily or solve globally (min-cost flow).

Code (Python)

from typing import List, Dict
import math

def rank_oncall_candidates(engineers: List[Dict], slot_minutes: int) -> List[Dict]:
    """
    engineers: list of dicts with keys:
      - name (str)
      - prev_minutes (int)
      - timezone_weight (float) >=0 (higher means can tolerate more minutes)
      - available (bool)
      - hard_unavailable (bool) -> exclude
      - soft_penalty (float) optional extra penalty 0..1
    Returns: list sorted by descending score (best candidate first)
    """
    n = len(engineers)
    if n == 0:
        return []

    # sanitize and defaults
    for e in engineers:
        e.setdefault('timezone_weight', 1.0)
        e.setdefault('available', True)
        e.setdefault('hard_unavailable', False)
        e.setdefault('soft_penalty', 0.0)
        e['prev_minutes'] = max(0, int(e.get('prev_minutes', 0)))

    total_prev = sum(e['prev_minutes'] for e in engineers)
    # target after assigning this one slot should be roughly equal for n people
    target = (total_prev + slot_minutes) / n

    ranked = []
    for e in engineers:
        if e['hard_unavailable']:
            score = -math.inf  # exclude
        else:
            deficit = target - e['prev_minutes']  # >0 means they need load
            # scale by timezone_weight: someone in a favorable timezone can accept more load
            tz = max(0.0, e['timezone_weight'])
            # If not available now (soft), apply a significant penalty but still allow if needed
            availability_factor = 1.0 if e['available'] else 0.3
            # soft_penalty reduces score (0..1)
            soft_pen = min(1.0, max(0.0, e.get('soft_penalty', 0.0)))
            # Compose score: favor large positive deficit; allow negatives (overloaded) but low score.
            score = (deficit * tz * availability_factor) - (soft_pen * slot_minutes * 0.5)
        ranked.append({'engineer': e, 'score': score})

    # sort descending by score
    ranked.sort(key=lambda x: x['score'], reverse=True)
    return ranked

Key concepts / reasoning

• Deficit-based scoring: picking the person with largest deficit moves total minutes toward equal distribution and thus reduces variance.
• Timezone weight multiplies deficit so people in favorable timezones get proportionally more load.
• Availability: hard_unavailable excludes; soft unavailability reduces score but doesn't always block.
• This greedy single-slot selection is O(n log n) due to sorting; computing scores is O(n).

Complexity

• Time: O(n log n) dominated by sort.
• Space: O(n) for ranked list.

Edge cases to handle

• All engineers hard_unavailable -> function returns all with -inf scores; caller should handle (raise, relax constraints, or escalate).
• Zero timezone_weight for everyone -> treat as 1.0 default or normalize to avoid dividing by zero.
• Very skewed prev_minutes -> greedy single-slot pick may be suboptimal for multi-slot assignments; consider global optimization.
• Negative or missing prev_minutes -> sanitized to 0.

Alternatives / improvements

• For assigning multiple future slots, model as min-cost flow or integer linear program: nodes=engineers, costs = increase in variance when assigning a slot; constraints for availability/timezone. This yields globally optimal minimal-variance assignment at higher complexity.
• Use squared-error marginal cost: cost of giving slot to engineer i = ( (prev_i + slot - mean_new)^2 - prev_i^2 ) to exactly minimize variance; select candidate minimizing that cost.
• Add fairness windows: prefer those who haven't been on-call recently using last_on_call timestamp as tie-breaker.

This implementation is practical for SRE teams as a fast, interpretable heuristic; swap in min-cost flow when you need provable optimality for many simultaneous slots.

NetworkPolicy in Kubernetes is purely additive: there is no "deny" rule you author directly. Once any policy selects a pod for a given direction (ingress or egress), that pod becomes default-deny in that direction, and only traffic matching some policy's explicit allow rule gets through. So finding "which policy is responsible" for a block usually means finding a policy that selects the destination pods and either has no matching rule at all, or a rule that is narrower than the traffic actually needs, not finding an explicit deny statement anywhere.

Step 1: find every policy that could affect the destination

kubectl get networkpolicy -n prod -o json | jq '.items[] | {name: .metadata.name, podSelector: .spec.podSelector, policyTypes: .spec.policyTypes}'

Example output:

{"name": "default-deny-ingress", "podSelector": {}, "policyTypes": ["Ingress"]}
{"name": "allow-internal-prod", "podSelector": {"matchLabels": {"tier": "internal"}}, "policyTypes": ["Ingress"]}

An empty podSelector: {} matches every pod in the namespace. Here, default-deny-ingress selects all pods in prod for Ingress and defines no ingress rules at all, meaning every pod in prod denies all inbound traffic unless a second, more specific policy also selects it and allows something. allow-internal-prod only helps pods labeled tier: internal; if the backend pods are not labeled that way, they get no exception and stay fully closed.

Step 2: reproduce and confirm which rule (or absence of one) is the block

kubectl -n web exec -it frontend-pod -- curl -sv --connect-timeout 5 http://backend.prod.svc.cluster.local:80

A connection that hangs until timeout (as opposed to an immediate connection refused) is the characteristic signature of a NetworkPolicy drop: the packet is silently discarded rather than actively rejected, since the destination pod's application never even sees the SYN.

Then inspect the specific policy selecting app=backend in prod:

kubectl describe networkpolicy default-deny-ingress -n prod

PodSelector:     <none> (Allowing the specific traffic to all pods in this namespace)
Allowing ingress traffic:
  <none> (Selected pods are isolated for ingress connectivity)

"Selected pods are isolated for ingress connectivity" with no rules listed is the exact confirmation: this policy is the one making backend pods deny-by-default, and no other policy in prod currently grants an exception for traffic from web.

Step 3: minimal fix

Add a narrowly scoped allow rule rather than widening or removing the existing deny-all; NetworkPolicy design should stay additive and specific.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-web-frontend-to-backend
  namespace: prod
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: web
      podSelector:
        matchLabels:
          app: frontend
    ports:
    - protocol: TCP
      port: 80

This uses kubernetes.io/metadata.name, the label Kubernetes automatically applies to every namespace (set to the namespace's own name) since it became a default-on feature around Kubernetes 1.21, rather than assuming a custom name=web label already exists on the web namespace. Relying on the built-in label avoids a second failure mode: a namespaceSelector that silently matches nothing because the expected custom label was never applied.

kubectl apply -f allow-web-frontend-to-backend.yaml
kubectl -n web exec -it frontend-pod -- curl -sS http://backend.prod.svc.cluster.local:80 -o /dev/null && echo OK

Trade-offs and pitfalls

• A podSelector: {} combined with policyTypes: [Ingress] and zero rules is easy to misread as "no policy" when it is in fact the strictest possible one (deny everything); always check for this shape first, since it is the single most common cause of "NetworkPolicy is blocking traffic unexpectedly" reports.
• namespaceSelector and podSelector inside one from entry are ANDed together (both must match the same source pod); a common mistake is listing them as two separate entries in the from list, which ORs them instead and unintentionally allows any pod in the web namespace, or any app=frontend pod in any namespace, rather than the intersection actually intended.
• Widening an existing broad policy (adding an exception to default-deny-ingress itself) instead of adding a new, narrowly scoped policy makes the security intent harder to audit later; prefer additive, purpose-named policies over editing a shared deny-all policy in place.
• This fix only affects Ingress on the backend side; if prod or web also enforces Egress policies, the frontend pods' egress must independently allow traffic to backend, or the same symptom recurs from the other direction.

Direct answer
A kill-switch is a feature flag whose sole purpose is an emergency, instant off-switch, evaluated as simply and reliably as possible so it works even under the stressed conditions (a live incident) it's most likely to be needed in; unlike a targeting flag, it typically has no complex rules to evaluate, just a boolean the application checks before executing risky logic.

Structured elaboration and worked example

def is_feature_enabled(flag_name: str, flag_client) -> bool:
    # Checks a kill-switch flag at runtime. Fails SAFE (returns False,
    # i.e. treats the risky feature as disabled) if the flag can't be
    # read, rather than assuming the feature is fine to run.
    try:
        return flag_client.get_bool(flag_name, default=False)
    except Exception:
        return False

def process_payment(order, flag_client):
    if not is_feature_enabled("new_payment_processor", flag_client):
        return legacy_payment_path(order)
    return new_payment_processor_path(order)

The application checks the flag BEFORE entering the risky code path, not as an afterthought inside it, so flipping the flag off takes effect on the very next check, no redeploy, no waiting for a rolling restart. The except fallback to False (fail safe, disable the risky path) rather than propagating the exception or defaulting to True is the specific design choice that makes this a SAFETY mechanism rather than just a toggle: if the flag system itself is having a bad day, that shouldn't mean risky code keeps running unchecked.

Trade-offs and pitfalls
A kill-switch needs to be evaluated with minimal dependencies and minimal latency, ideally from a local in-memory cache rather than a live network call, precisely because the moment it's needed most (an active incident) is also the moment the rest of the system's infrastructure is most likely to be under stress or partially degraded. The common mistake is building a kill-switch with the same rich targeting-rule evaluation logic as an ordinary feature flag; the extra complexity is exactly what you don't want in the one flag type whose entire job is working reliably in a crisis.

A correlation ID is a single identifier generated at the edge of a request and propagated through every hop, so logs, metrics, and traces from otherwise-independent services can be joined into one end-to-end picture.

Design

1. Generate once, at the edge (the first service to see the request), and reject or ignore any incoming ID from an untrusted external source unless you specifically want to correlate with an external partner's identifier.
2. Propagate via headers (e.g. X-Request-Id or the W3C traceparent format) on every downstream call, and via message baggage for async work (queue messages, background jobs) so the ID survives a hop that isn't a synchronous HTTP call.
3. Attach it to every log line and trace span emitted while handling that request, not just at the top-level handler.
4. For third-party/legacy components you cannot instrument, enrich logs at your own ingress/egress boundary around that component (the request going in, the response coming out) so you can at least bound how much time was spent inside the black box, even without visibility inside it.
5. Query time: given an ID, a single query across the log/trace store reconstructs the full path, ordered by timestamp, across every service that touched the request.

Trade-offs and pitfalls

High-cardinality IDs are cheap to log but expensive to index if you try to make every field high-cardinality; keep the correlation ID itself indexed and unique, but avoid also indexing every other high-cardinality field (like raw user input) at the same tier, or storage cost balloons. Sampling (only tracing 1% of requests in detail) is fine for aggregate analysis but means a specific customer's specific failing request may not have full trace data; the mitigation is to force full capture whenever an error is detected on that request, regardless of the sampling decision made at the start.

Brief approach: Treat policies as versioned code, enforce them at multiple gates (shift-left and run-time), and give developers clear, actionable feedback linked to policy source and remediation steps.

Tools and examples:

• Static/CI linting: terraform validate + tflint, conftest (uses OPA/Rego) for IaC; kubeval or kube-score for manifests.
• Policy engines: OPA/Gatekeeper (Kubernetes admission with Rego), HashiCorp Sentinel (Terraform Enterprise), Azure Policy (AKS + resource control).
• Admission/runtime: Gatekeeper or OPA as a validating/admission webhook to block disallowed k8s objects; Azure Policy for cloud resource drift/prevent-create.

Where to enforce:

• Pre-commit: lightweight checks (pre-commit hooks running tflint, conftest) to catch obvious issues early.
• CI: stronger checks (conftest, terraform plan + Sentinel or policy checks) that fail builds with policy reports.
• Cluster/admission: Gatekeeper/OPA enforce runtime constraints and prevent bypass for manual kubectl applies.

Presenting denied changes (actionable feedback):

• Consistent error format: include policy ID, human-friendly reason, failing field path, example remediation snippet, and link to policy doc/PR.
• CI output: fail the job with a concise summary plus downloadable full JSON report (OPA/gatekeeper emits structured results).
• IDE/editor: surface policy lint results via language server or pre-commit so developers see fixes inline.
• Pull request automation: post a comment summarizing violations with remediation steps and a link to the policy repo and a sample patch.
  Example message: "DENY: policy 'no-hostPath' — hostPath used at spec.volumes[2]. Use an emptyDir or PVC. See https://git.company/policies/no-hostPath and example fix: replace hostPath with emptyDir."
  Best practices: keep policies small and testable, version policies with CI, provide exemptions workflow, and collect telemetry on blocked changes to iterate on policy clarity.

Direct answer

Stay neutral by asking, not telling, both sides. Get underneath the stated positions by finding what each person is actually protecting, revenue predictability, technical risk, reputation, past experience, and negotiate from that underlying interest instead of from the positions they walked in with.

Structured elaboration

The technique is separating positions from interests: a position is what someone is demanding, an interest is why they are demanding it.

1. Meet each person separately first if the joint room is too charged for either to speak candidly. Ask direct, non-leading questions: what happens if this goes the other way, what are you actually worried about? People often state a harder position in front of the other person than they hold privately.
2. Listen for the interest underneath the stated position. Someone insisting on a hard deadline may actually be worried about looking unreliable to their own stakeholders, not about the date itself. Someone insisting on a particular technical approach may be protecting against repeating a past incident, not defending the approach for its own sake.
3. Stay neutral by not adopting either position as your own recommendation early. Reflect both interests back to each person to confirm you understood correctly, so neither feels unheard before you move forward.
4. Once the interests are visible, look for an option that satisfies both, rather than a compromise that splits the difference on the original positions, which usually satisfies neither interest fully.
5. Bring both people into the same room only once you understand both sides well enough to keep the conversation anchored on interests instead of letting it slide back into restating positions.
6. Get an explicit, verbal agreement to the resolution from both people in the room, then document it along with the reasoning, not just the outcome. Decisions that skip the why tend to get quietly re-litigated later.

Worked example

A sales leader wants a fully featured launch by end of quarter. Engineering leadership wants a two-quarter delay to avoid instability. The positions look flatly opposed, but talking to each separately surfaces that sales actually needs one specific, demonstrable capability for a committed customer conversation, not literally everything, and engineering is actually worried about one particular unstable subsystem, not the whole feature set. A phased plan that ships the specific capability sales needs now, built on the stable parts of the system, while the risky subsystem gets its rework on its own timeline, satisfies both underlying interests even though neither person's original position was granted in full.

Trade-offs and pitfalls

Interest-based framing is slower than just picking a side. If a decision genuinely has to happen today, spend the time finding interests quickly rather than skipping the step entirely, a decision made purely from positions rarely survives contact with implementation.
Staying neutral does not mean staying silent when one side's underlying interest is simply wrong on the facts, a risk that is not real, a deadline that is not actually fixed. Neutrality is about the process, not about withholding a correction you are confident in.
If you only interview people separately and never get them into the same room to hear each other directly, you risk becoming a relay each side later blames for misrepresenting them. A well-framed joint session is usually still necessary, not optional.

Direct answer
For a global, latency-sensitive user-profile subsystem, a reasonable decomposition splits Profile Storage, Authentication, Preferences, and Avatar/Media Processing into separate services, each with a clearly different access pattern and failure-isolation need, communicating synchronously only where a request genuinely can't proceed without an immediate answer.

Structured elaboration
Authentication is kept separate because it's a security-sensitive, extremely high-frequency, low-latency dependency that nearly every other request needs to check, and isolating it means an issue in less-critical functionality (like avatar processing) can never take down the ability to authenticate a request. Profile Storage owns the core profile fields (name, settings, account state) and needs to be read on nearly every request, so it's optimized for fast, simple reads, kept narrowly scoped to just that core data rather than accumulating every profile-adjacent feature over time. Preferences (notification settings, UI preferences, and similar lower-stakes, less frequently-read data) is split out specifically because it doesn't need the same latency or availability guarantees as core profile and authentication data; if the Preferences service is slow or briefly unavailable, the rest of the profile experience should still work with sensible defaults, whereas Authentication being unavailable is a much more serious failure. Avatar/media processing is naturally asynchronous (resizing, transcoding, or moderating an uploaded image takes real time and doesn't need to block the rest of the profile experience) and has a very different resource profile (CPU/IO-heavy batch-style work) from the fast, simple reads the other services handle, making it a clear candidate for its own service with its own scaling behavior.

On communication choices: reading a user's core profile (needed on most page loads) should be synchronous, since the caller can't meaningfully proceed without it; updating a display avatar can be asynchronous, since the caller doesn't need to wait for image processing to finish, just for the upload to be accepted. Failure isolation follows from the split: if Avatar Processing is degraded, users can still authenticate, read their profile, and use the rest of the product, with a graceful fallback (a default avatar, or the previous one) rather than the whole profile experience failing.

Worked example
For team coordination across these services: the team owning Authentication needs the strictest change-management discipline given its security sensitivity and blast radius if it breaks, and its API contract with the other three services should be the most stable and carefully versioned of the four; Profile Storage's API needs to stay backward-compatible for the many other services and clients that read from it directly; Preferences and Avatar Processing, being lower-stakes and more independently used, can iterate faster with a lighter review process, as long as their optional, non-blocking nature is preserved (nothing else should come to depend on them synchronously in a way that would undo the isolation benefit).

Trade-offs and pitfalls
The most common mistake in a decomposition like this is letting Profile Storage's scope creep to absorb Preferences or other adjacent data over time ("it's just one more field"), which quietly recreates the coupling the split was meant to avoid; keeping the boundary honest requires treating a new field's home as a deliberate decision (does this need Profile Storage's stricter availability and latency guarantees, or can it live in the more relaxed Preferences service) rather than defaulting to wherever's most convenient to add it.

Direct answer
The strongest mentoring story names a concrete starting point (not "they were new," but what specifically they didn't yet know or couldn't yet do), describes what you actually did differently because of that starting point, and points to a real change in what the person could do independently afterward as the evidence of growth, not just that time passed or that they were nice about it.

Structured elaboration

What "starting from" should actually specify

Vague ("they were junior") is weak. Specific ("they could write correct code but always needed help scoping the actual problem before writing it") is strong, because it sets up a real before and after.

What "what you did" should show

The interesting part isn't a list of activities (pairing, reviews, 1:1s); it's the judgment behind them: why you chose that particular intervention for that particular gap, and what you adjusted when the first approach didn't fully work.

What "how you know they grew" should show

This is the part candidates under-answer. Two things separate a senior answer here:

• Independence as the real signal, not sentiment. The strongest evidence isn't "they thanked me," it's a concrete example of them handling something on their own that they previously couldn't, ideally something you didn't have to prompt.
• Reframing your own impact as leverage, not personal output. A senior candidate can articulate that developing someone else who can now independently do the work is a multiplier on team capacity, arguably more valuable than the same hours spent on your own individual output, because it compounds. That's a different, and stronger, claim than "I helped someone and it felt good."

The real tension: mentoring time vs. delivery

Mentoring genuinely competes with your own delivery time, especially early in a relationship when the payoff hasn't materialized yet. A senior answer is honest about this rather than pretending mentoring is free: it names a moment where mentoring time actually cost something (a deadline got tighter, you did more of the work yourself that cycle) and explains the judgment call for when it's right to deliberately scale mentoring back temporarily to protect a real deadline, versus when protecting the mentoring time is the higher-leverage call even under pressure.

Worked example

Situation

I mentored someone who was technically capable but consistently needed help before they'd start: given an ambiguous problem, they'd wait for someone to scope it into clear steps rather than attempting that themselves.

Action

Instead of continuing to scope tasks for them, I deliberately started handing over problems one level more ambiguous than they were comfortable with, then worked through their proposed scoping with them afterward rather than before, so the struggle happened on their side first. Early on this slowed things down, and I redid some of their scoping myself before it went further, which cost real time on a couple of deadlines.

Result

Over time the gap between their first attempt at scoping and a workable plan narrowed, until they were handling genuinely ambiguous problems without needing that step from me at all. The clearest evidence wasn't a compliment, it was a specific instance of them independently scoping and delivering something ambiguous while I was out, without anyone asking them to check with me first.

The trade-off moment

Partway through, we had a hard deadline where I made a deliberate call to scope their next task myself rather than continuing the hands-off approach, because the team couldn't absorb the risk of a slower first pass that cycle. I was explicit with them about why, so it didn't read as a loss of confidence in them, just a temporary trade-off.

Trade-offs & pitfalls

• Confusing activity with growth. Listing pairing sessions and 1:1s isn't evidence of anything; a senior answer points to a specific, observable change in independent capability.
• Never naming the cost. A story where mentoring never competed with anything else usually isn't a very real story. Naming a moment you scaled it back, and why, is more credible than claiming it was free.
• Missing the leverage framing entirely. Describing mentoring purely as "helping a nice person" misses the stronger claim: that growing someone else's independent capability is a real multiplier on what the team can deliver.

Requirements & constraints (clarify): 99.99% availability (~52 min/year downtime), RPO 15m (max 15m data loss), global read/write latency goals, expected traffic pattern (regional vs cross-region), compliance.

High-level recommendation: active-active per-region Kubernetes clusters (one control plane per region) with global traffic steering and cross-region data replication for stateful services. Use GitOps + centralized control tooling for operator consistency.

Control plane topology

• One control plane per region (regional clusters): reduces API-server latency for operators and controllers, isolates failures (control plane outage in region ≠ global outage), simpler networking (no single global control-plane networking).
• Centralized management plane (non-runtime): give a single pane for lifecycle via Cluster API/Crossplane/ArgoCD running in a management region or SaaS. This plane does NOT replace regional control planes — it provisions/configures them.
• Trade-off: more operational overhead (upgrade/DR per-cluster) vs much better isolation and latency.

Data replication for stateful services

• Use database systems that support geo-replication with RPO ≤15m:
  • For strong transactional needs: managed globally-distributed DB (Spanner, Cosmos DB) if available — lowest operational complexity.
  • For relational with primary write region: async WAL shipping + CDC to replicas (Postgres with BDR/pglogical or Patroni + WAL-E). Ensure WAL shipping frequency <15m and monitor lag.
  • For services tolerating eventual consistency: use multi-region replicated stores (Cassandra, CockroachDB) configured for appropriate consistency levels.
• Stateful sets on K8s: keep persistent volumes region-local (no cross-region PVs). Use regional read-replicas and application-level failover logic to redirect writes.
• Backups + point-in-time recovery with frequent snapshots (<<15m WAL retention) and tested restore playbooks.

Minimizing API server latency

• Locally-hosted API servers in each region; point CI/CD and operators to region-local API endpoints.
• Use kube-proxy in IPVS mode, keep controller-runtime caches tuned to reduce API calls.
• For cross-cluster orchestration, use control-plane API aggregators or lightweight agents (GitOps agents) that talk to local API servers rather than a centralized API.
• Use API server horizontal scaling (etcd tuned for size/IO) and read-only API server replicas for heavy read patterns.

Global DNS and traffic steering

• Use a Global Load Balancer / GSLB (Route53 latency-based + health checks, NS1, or Anycast + CDN/GCLB/Azure Front Door) to:
  • Route clients to nearest healthy region (latency-based).
  • Health checks (L7 health, upstream application probes) to remove unhealthy regions quickly.
  • Weighted failover for gradual traffic shifts during degraded capacity.
• Combine with client-side locality hints and regional failover priorities.
• Use TTLs low enough (30–60s) for rapid failover but balanced with DNS query volume.

Automated failover procedures

• Detection: multi-layer health checks (global LB, regional ingress, pod liveness, DB replication lag).
• Automated actions:
  • Traffic steering: GSLB toggles weights -> move traffic away from region automatically when health check fails.
  • Data: if primary DB region fails, automated promotion scripts (or managed DB failover) promote a read-replica to primary; ensure promotion completes within RTO and meets RPO via replication lag thresholds.
  • K8s: bootstrap replacement nodes via IaC + Cluster API; auto-scale regional cluster capacity.
• Orchestration: use runbooks encoded as automation (Argo Workflows/Step Functions) and gated escalation for risky operations (promotions).
• Post-failover validation: smoke tests, canary traffic, monitoring alerts.

Observability & testing

• End-to-end SLOs, synthetic tests from multiple geos, replication lag metrics, control-plane health, DNS failover time measurement.
• Regular chaos exercises (chaos monkey, simulating region loss) and DR drills validating RTO/RPO and runbooks.

Trade-offs

• Per-region clusters increase ops complexity and cost but give isolation, lower latency, and simpler failover.
• Strong consistency across regions increases latency and cost; often RPO 15m indicates async replication acceptable — choose DB technology to meet RPO.
• Faster DNS failover (low TTL) increases DNS load and cache churn; combine with passive health checks and GSLB for faster convergence.
• Fully automated DB promotions risk split-brain if network partitions exist — build safe fencing (quorum, lock service) and observable safeguards.

Example concrete stack

• Regional clusters managed via Cluster API, configured via ArgoCD.
• Global traffic: Cloud GCLB / Anycast + Route53 latency-based with health checks.
• Data: CockroachDB for multi-region SQL or managed Spanner for strong global consistency; Postgres primary in region with WAL streaming + pglogical for apps needing lower cost.
• Automation: Argo Workflows + SRE-runbooks in Git + Prometheus/Thanos + Grafana for metrics; PagerDuty + automated runbook runners.

This design balances isolation, API latency, and operational control while meeting 99.99% availability and RPO 15m; final choices depend on cost, consistency needs, and cloud/provider features.

P0 (Critical / Sev-0)

• Objective criteria: Complete service outage or data loss for enterprise customers; >50% of users or key customers affected; revenue impact immediate and high (e.g., billing failures); material legal/regulatory breach or data exfiltration; SLOs violated catastrophically.
• Response priority: Immediate all-hands paged response, 24/7 war room, engineers and on-call leader triage within 5–15 minutes.
• Communications: Continuous public/customer-facing status updates every 15–30 minutes; executive notif within 30 minutes; postmortem mandatory with timeline, root cause, and remediation plan.

P1 (High / Sev-1)

• Objective criteria: Major feature degraded or partial outage affecting a subset of enterprise customers (e.g., 10–50%); significant performance degradation causing SLA breach risk; revenue impact moderate; potential regulatory impact low.
• Response priority: High-priority response team within 30–60 minutes, focused mitigation and workaround; restore core functionality asap.
• Communications: Status updates every 30–60 minutes to customers and internal stakeholders; escalation to product and account teams; post-incident review within a week.

P2 (Medium / Sev-2)

• Objective criteria: Non-critical degradations, intermittent errors, single-customer impact or smaller cohort (<10%); performance warnings but SLOs mostly met; minimal revenue/legal risk.
• Response priority: Address during business hours or next on-call rotation; create ticket with priority and SLA for resolution (e.g., 24–72 hours).
• Communications: Initial acknowledgement within 1 business hour, status updates at major milestones, inform affected customers and account managers proactively.

P3 (Low / Sev-3)

• Objective criteria: Minor bugs, UI issues, documentation gaps, or enhancements with no customer impact.
• Response priority: Backlog grooming, scheduled fixes.
• Communications: Include in release notes; notify affected users as needed.

Notes on classification: use measurable thresholds (percentage of users, error rates, revenue/day, regulatory flags). If multiple criteria span levels, adopt the highest-severity classification. Ensure runbooks map each level to roles, escalation paths, SLAs, and communication templates.