Meta Staff Site Reliability Engineer Interview Preparation Guide

Site Reliability Engineer (SRE)

Interview Rounds

Recruiter Screening

45 min5 focus topicsbehavioral

What to Expect

The initial conversation with Meta's recruiting team to validate your background, assess role fit, and determine if your experience aligns with the Staff-level SRE position. This combines initial phone screening and recruiter follow-up into a single call. The recruiter will discuss your career trajectory, understanding of the SRE discipline, and motivation for joining Meta. This round covers logistics, compensation expectations, and timeline. For Staff-level candidates, expect deeper questions about your impact in previous roles, your approach to scaling teams and systems, and your vision for reliability engineering.

Tips & Advice

Be prepared to articulate your career story with emphasis on the progression to Staff level and the key technical and leadership milestones. Clearly explain what excites you about the SRE role at Meta and how your background specifically prepares you for this position. Have specific examples ready showing how you've improved reliability at scale, led infrastructure initiatives, and mentored engineers. Ask thoughtful questions about Meta's SRE organization, their current reliability challenges, and how this role contributes to Meta's engineering organization. Be honest about compensation expectations but avoid anchoring too early. Show enthusiasm for Meta's mission and technical challenges.

Focus Topics

Motivation & Fit for Meta

Clearly articulate why you're interested in Meta specifically and why this Staff SRE role aligns with your career goals. Research Meta's scale, their infrastructure challenges, and where your skills can create impact. Mention specific aspects of Meta's technology or mission that attract you.

Practice Interview

Study Questions

Understanding of SRE Discipline & Philosophy

Demonstrate understanding of SRE as a discipline—the philosophy of treating operations as an engineering problem, the importance of reliability versus feature velocity balance, error budgets, and the SRE toolkit. Show familiarity with industry concepts like SLI/SLO/SLA, toil automation, incident response, and postmortems.

Practice Interview

Study Questions

Mentorship & Team Development Approach

Discuss your approach to developing engineers—how you mentor senior engineers, develop talent, create learning opportunities, and contribute to team culture. For Staff level, this is critical to the role.

Practice Interview

Study Questions

Examples of Technical Leadership & Impact

Prepare 3-4 concrete examples of times you've made significant technical decisions at scale, led cross-functional initiatives, influenced architecture direction, or improved reliability metrics substantially. For Staff level, focus on examples that affected multiple teams or had organization-wide impact.

Practice Interview

Study Questions

Career Narrative & Staff-Level Progression

Articulate your career journey emphasizing how you reached Staff level expertise in SRE. Highlight key inflection points where your impact scaled beyond individual contributions to team and organizational level. Prepare to discuss the progression from IC to senior IC roles, growth in technical depth, and increasing scope of responsibility.

Practice Interview

Study Questions

Technical Phone Screen 1: Infrastructure & Systems Knowledge

60 min5 focus topicstechnical

What to Expect

A focused technical discussion with a Meta SRE engineer exploring your deep knowledge of large-scale infrastructure, systems architecture, and reliability practices. This 50-60 minute interview assesses your ability to think about complex infrastructure problems, design scalable systems, and make informed decisions about reliability trade-offs. The interviewer will present scenarios related to Meta's scale and ask you to discuss architectural choices, tradeoffs, and implementation considerations. Expect questions covering distributed systems concepts, infrastructure automation, monitoring, and capacity planning—all grounded in real-world challenges at Meta's scale.

Tips & Advice

Think out loud and explain your reasoning. For infrastructure problems, start by clarifying requirements and constraints. Discuss tradeoffs explicitly—reliability versus cost, consistency versus availability, automation complexity versus manual effort. Draw diagrams if helpful using a whiteboard or shared document. Show familiarity with monitoring and observability—discuss how you'd measure whether your solution works. Be specific about tools and technologies you'd use, such as Kubernetes, service mesh, and monitoring tools. Don't just describe what you would do; explain why. If you're unsure about something, say so and explain how you'd approach learning it. Show that you understand Meta's scale challenges—billions of users, real-time requirements, global infrastructure.

Focus Topics

Reliability Engineering Principles & SLOs

Deep understanding of SRE philosophy—error budgets, the tension between reliability and velocity, SLI/SLO/SLA definitions, and using these concepts to make engineering decisions. How to instrument systems to track reliability, how to use error budget to guide prioritization.

Practice Interview

Study Questions

Capacity Planning & Scalability

Predicting infrastructure needs based on growth, designing systems to scale linearly or better, understanding resource constraints, and planning expansions. Discuss how to forecast demand, identify bottlenecks, plan database scaling, network scaling, etc. Understanding growth metrics and their implications.

Practice Interview

Study Questions

Large-Scale System Architecture Design

Design systems that operate at Meta scale—billions of daily active users, global infrastructure, real-time requirements. Discuss multi-region architecture, failover strategies, consistency models under failure conditions, and architectural patterns for resilience. For Staff level, go deeper into architecture decisions, tradeoffs between consistency and availability, geographic distribution, and handling partial failures.

Practice Interview

Study Questions

Infrastructure Automation Frameworks & Tooling

Deep understanding of infrastructure automation—infrastructure as code using tools like Terraform or CloudFormation, configuration management, deployment automation, and rollback strategies. Discuss how to automate infrastructure changes safely, versioning approaches, testing automation, and handling state.

Practice Interview

Study Questions

Performance Monitoring & Observability Design

Designing comprehensive monitoring systems for large-scale distributed systems. Understanding metrics, logs, traces, and their tradeoffs. SLI/SLO instrumentation, alerting strategies that minimize false positives, and designing dashboards for different audiences. Knowledge of observability tools and how to architect monitoring for reliability.

Practice Interview

Study Questions

Technical Phone Screen 2: Incident Response & Troubleshooting

60 min5 focus topicstechnical

What to Expect

A technical interview focused on incident response, troubleshooting methodology, root cause analysis, and operational decision-making under pressure. The interviewer will present realistic incident scenarios from Meta's scale and ask how you'd respond—diagnostics, decision-making, communication, and resolution. This assesses your ability to remain calm under pressure, apply systematic troubleshooting, identify root causes, and drive incidents to resolution. For a Staff-level candidate, focus is also on how you'd lead response, mentor others during incidents, and conduct effective post-mortems.

Tips & Advice

Treat this like a real incident—ask clarifying questions about what's happening, what symptoms you're seeing, and what services/systems are affected. Work methodically through diagnosis rather than jumping to conclusions. Explain your thought process as you investigate. For each hypothesis, explain how you'd test it and what tools you'd use. Show systematic debugging approaches—understand logs, metrics, traces, customer impact. Don't fixate on one theory; generate and test multiple hypotheses. When you identify the root cause, discuss how to fix it quickly while minimizing risk. Think about rollback options and communication during incidents. For Staff level, discuss how you'd coordinate with other teams, communicate with leadership, and structure the response. Talk about post-incident review—what lessons would you capture, how would you prevent recurrence.

Focus Topics

Post-Incident Review & Learning Process

Designing effective post-incident reviews that extract lessons without blame. Identifying systemic issues versus one-time failures. Creating actionable follow-up work. Understanding how to use incidents as learning opportunities for the team.

Practice Interview

Study Questions

Communication & Coordination During Incidents

Communicating clearly during crisis situations—keeping stakeholders informed, avoiding false updates, coordinating across teams, managing customer communication. For Staff level, this includes leading the response, delegating work, and escalating appropriately.

Practice Interview

Study Questions

Decision Making & Trade-offs Under Pressure

Making sound technical decisions during incidents when under time pressure and with incomplete information. Balancing speed of recovery versus risk of making things worse. Knowing when to rollback versus rollforward, when to apply bandaids versus permanent fixes, and when to escalate.

Practice Interview

Study Questions

Incident Triage & Root Cause Analysis

Systematically diagnosing incidents at scale. Understanding how to triage severity, identify which systems are affected, correlate symptoms, and work toward root cause. Creating effective hypotheses and testing them methodically. Using logs, metrics, traces, and other signals to build understanding.

Practice Interview

Study Questions

Troubleshooting Methodology for Distributed Systems

Applying structured troubleshooting approaches to distributed systems where issues can be subtle and hard to reproduce. Understanding cascading failures, partial failures, and how to isolate problems. Knowledge of debugging tools and how to use them systematically.

Practice Interview

Study Questions

Onsite Round 1: Systems Design Interview

60 min6 focus topicssystem design

What to Expect

An in-depth systems design interview where you architect a large-scale system from scratch. You'll be given a system design problem likely related to Meta's infrastructure or services and asked to design a complete system considering reliability, scalability, performance, monitoring, and operational aspects. The interviewer will probe your design decisions, ask about tradeoffs, and explore how you'd handle edge cases, failures, and operational requirements. For a Staff-level candidate, expect higher bar—deeper discussion of reliability patterns, failure scenarios, operational automation, and how this system would be monitored and maintained at scale.

Tips & Advice

Start by clarifying requirements and constraints—scale, latency requirements, consistency requirements, failure modes you need to handle. Think about both the happy path and failure scenarios. Discuss reliability—what happens when components fail, how do you detect failures, how do you recover. Design monitoring and alerting as part of the system, not an afterthought. Think about operational concerns—how would you deploy this, how would you scale it, how would you debug issues. Use diagrams liberally. For each component, discuss tradeoffs—consistency versus availability, strong versus eventual consistency, synchronous versus asynchronous, etc. Discuss data modeling and storage considerations. Show familiarity with patterns used in large-scale systems—load balancing, caching, replication, sharding. For Staff level, go deeper—discuss how the team would be organized, how you'd approach operational automation, how you'd handle geographic distribution, etc.

Focus Topics

SLO-Driven Design & Error Budgets

Designing systems with explicit SLOs and using error budgets to guide reliability investment. Understanding how to instrument systems to track SLIs, how to use error budget to prioritize work, and how design decisions impact achievable SLOs.

Practice Interview

Study Questions

Observability & Monitoring Architecture

Designing comprehensive observability into the system—metrics that track business and technical health, logs that help with debugging, traces that help understand distributed flow. SLI instrumentation, alerting strategy.

Practice Interview

Study Questions

Multi-region & Geographic Architecture

Designing systems across geographic regions—handling latency, consistency across regions, failover strategies, and traffic routing. Understanding tradeoffs between replication and network costs.

Practice Interview

Study Questions

Operational Automation & Deployment Strategy

Designing systems to be operationally simple—automation of common operational tasks, safe deployment strategies such as canary, rolling, or blue-green, rollback capabilities, and monitoring for operational health.

Practice Interview

Study Questions

Distributed System Trade-offs & Consistency Models

Understanding fundamental tradeoffs in distributed systems—strong versus eventual consistency, availability versus consistency (CAP theorem), synchronous versus asynchronous communication. When to use each pattern and implications for operations.

Practice Interview

Study Questions

Design for Reliability & Failure Handling

Designing systems that continue operating under partial failures. Understanding circuit breakers, bulkheads, graceful degradation, and failure modes. Designing for geographic redundancy, data replication strategies, and consistency models that handle failures. How to design systems so that individual component failures don't cascade.

Practice Interview

Study Questions

Onsite Round 2: Distributed Systems & Architecture

60 min5 focus topicstechnical

What to Expect

A deep-dive technical interview focusing on distributed systems concepts, algorithms, and architecture patterns used in large-scale systems. The interviewer will discuss specific distributed systems challenges—consensus, replication, failure detection, data consistency, load balancing—and explore your understanding at depth. Expect questions about specific algorithms (Raft, Paxos concepts, etc.), when to use them, tradeoffs, and failure scenarios. This round assesses theoretical understanding combined with practical application to real systems.

Tips & Advice

Show deep understanding of distributed systems principles. Be familiar with consensus algorithms, replication strategies, and consistency models—understand the concepts even if you can't write code for them. Discuss specific systems you've worked with—how they handle certain challenges, what tradeoffs they make. When discussing algorithms, explain the intuition and tradeoffs, not just mechanical details. Relate concepts to real operational scenarios—what happens when you lose a majority, what happens during network partitions, how does this impact operations. Be precise with terminology—understand the difference between strong consistency, eventual consistency, causal consistency, etc. Show awareness of practical concerns—latency implications, failure impact, recovery time. For Staff level, discuss how you'd design systems using these concepts, mentor others on these principles, and make architecture decisions based on deep understanding.

Focus Topics

Load Balancing & Traffic Distribution Strategies

Understanding different load balancing approaches, consistent hashing, request routing, handling of uneven load, and stateful load balancing challenges. Tradeoffs between different strategies.

Practice Interview

Study Questions

Network Reliability & Timeout Handling

Understanding network failure modes—packet loss, latency, partitions. How systems should handle timeouts, retries, and idempotency. Understanding failure detection challenges caused by network complexity.

Practice Interview

Study Questions

Failure Detection & Recovery Mechanisms

Understanding how systems detect failures using health checks, heartbeats, etc., challenges in detection such as false positives and slow detection, recovery mechanisms, and how to design systems robust to different failure scenarios.

Practice Interview

Study Questions

Data Consistency Models & Guarantees

Understanding different consistency models—strong consistency, eventual consistency, causal consistency, etc. Understanding what each guarantees and implies for application behavior and operational complexity. Understanding how to maintain consistency under failures.

Practice Interview

Study Questions

Consensus & Replication Algorithms

Understanding consensus algorithms and their role in distributed systems. Knowledge of algorithm families such as Paxos and Raft, their properties, failure modes, and operational implications. Understanding quorum systems, split-brain scenarios, and recovery from failures.

Practice Interview

Study Questions

Onsite Round 3: Coding Interview (Systems-Focused)

60 min4 focus topicstechnical

What to Expect

A coding interview focused on systems programming and infrastructure-related coding challenges. You'll solve 1-2 coding problems, likely involving systems concepts like thread safety, resource management, or performance optimization. Problems may relate to building infrastructure components, optimization challenges, or distributed systems algorithms. For a Staff-level SRE, the bar is higher—clean, optimized code with good design. You should be able to discuss efficiency implications, tradeoffs, and how this code would behave in production.

Tips & Advice

Even at Staff level, coding interviews expect clean, working code. But the bar is higher for design and optimization. Think about edge cases and how your code would behave at scale. Consider memory usage, CPU efficiency, and how code would perform under load. Use appropriate data structures and algorithms. For systems-focused problems, think about concurrency, resource limits, and operational aspects. Be ready to optimize based on feedback. Discuss tradeoffs—is correctness more important than performance, or vice versa? How would you test this code? Write code that others on the team would want to maintain.

Focus Topics

Resource Management & Error Handling

Writing code that manages resources correctly—memory, files, connections, etc. Proper cleanup and error handling. Code that doesn't leak resources or crash ungracefully.

Practice Interview

Study Questions

Infrastructure as Code & Automation Scripts

Writing code for infrastructure automation—scripts for deployment, monitoring, or operational tasks. Clean, maintainable code that others can understand and modify. Proper error handling and logging.

Practice Interview

Study Questions

Performance-Critical Code & Optimization

Writing and optimizing code for performance—understanding algorithmic complexity, data structure choices, memory efficiency, cache efficiency. Profiling and identifying bottlenecks. Code that performs well at scale.

Practice Interview

Study Questions

Systems Programming & Concurrency

Writing code that handles concurrency correctly—thread safety, synchronization primitives, avoiding deadlocks and race conditions. Understanding performance implications of different concurrency approaches. Coding problems may involve multi-threaded systems or concurrent processing.

Practice Interview

Study Questions

Onsite Round 4: Infrastructure Automation & Tooling

60 min5 focus topicstechnical

What to Expect

A technical interview focused on infrastructure automation, tooling, and operational practices. The interviewer will discuss how you design and implement infrastructure automation—tools and frameworks you'd use, how you'd structure automation for maintainability and safety, how you'd approach rolling out infrastructure changes, and how you'd handle operational tooling needs. This may include discussion of container orchestration like Kubernetes, CI/CD systems, configuration management, and monitoring tools. You may be asked to architect an automation solution for a specific operational challenge.

Tips & Advice

Discuss automation in concrete terms—which tools would you use and why? How would you structure the automation to be maintainable? Show familiarity with modern infrastructure tools. For Kubernetes, understand the concepts beyond just deployment—how would you handle upgrades, what about monitoring, how would you handle failures. Discuss how you'd make infrastructure changes safely—testing, gradual rollout, rollback capability. Show that you think about operational concerns—visibility into what's running, debugging when things go wrong, capacity planning. For Staff level, discuss architecture decisions—how you'd organize automation, how you'd scale it, how you'd make it work across teams. Show that you understand tradeoffs—flexibility versus simplicity, automated versus manual, etc.

Focus Topics

Monitoring, Alerting & Observability Tooling

Designing and implementing comprehensive monitoring and alerting—metrics collection, log aggregation, distributed tracing. Understanding observability tools and platforms. How to design monitoring that's actually useful for operations.

Practice Interview

Study Questions

Infrastructure as Code & Configuration Management

Treating infrastructure as code—versioning infrastructure, reviewing changes, testing infrastructure changes, documenting infrastructure. Using IaC tools effectively, handling secrets, managing drift.

Practice Interview

Study Questions

CI/CD Pipeline Design & Implementation

Designing comprehensive CI/CD systems—build automation, testing, deployment automation, rollback capabilities. Understanding different deployment strategies such as canary, rolling, or blue-green, how to make deployments safe, and how to handle failure scenarios.

Practice Interview

Study Questions

Infrastructure Automation Framework & Tooling

Understanding and designing infrastructure automation systems using tools like Terraform, Ansible, or custom solutions. How to make automation maintainable, testable, and safe. How to handle state management, versioning, and rollback in infrastructure automation.

Practice Interview

Study Questions

Container Orchestration & Kubernetes Architecture

Deep understanding of Kubernetes architecture, deployment strategies, resource management, networking, storage, and operational practices. Understanding how to design reliable Kubernetes deployments, handle node failures, perform upgrades safely. Understanding security and isolation in containerized environments.

Practice Interview

Study Questions

Onsite Round 5: Behavioral & Leadership Interview

60 min5 focus topicsbehavioral

What to Expect

A behavioral interview assessing cultural fit, values alignment, and leadership capability. The interviewer will ask about your experience with cross-functional collaboration, how you lead technical initiatives, how you handle ambiguity and conflict, and how you've grown as a leader. At Staff level, focus is on your ability to influence across teams, mentor senior engineers, communicate with stakeholders at different levels, and drive complex technical decisions to completion. You'll be asked about specific situations you've handled and how they demonstrate Meta values—moving fast, being intellectually honest, building impact, etc.

Tips & Advice

Use the STAR method for behavioral questions—Situation, Task, Action, Result. Focus on your role and impact specifically, not just team achievements. Show how you handle ambiguity—when you don't have all information, how do you proceed? Demonstrate learning and growth by sharing examples of challenges you've overcome. Show emotional intelligence—how you handle disagreement, support teammates, navigate organizational dynamics. For Staff level, emphasize leadership—how you've influenced decisions, driven changes across teams, developed junior engineers. Share examples of times you've made unpopular but necessary technical decisions. Show how you balance speed and quality, individual contribution and team development. Discuss how you've handled failure and what you learned. Align your stories with Meta values—moving fast safely, being direct and honest, building things that matter.

Focus Topics

Technical Decision Making & Ownership

How you approach complex technical decisions—gathering data, involving stakeholders, making decisions with incomplete information. Examples of decisions you've made and owned, including decisions that were later changed (showing humility). How you think about tradeoffs.

Practice Interview

Study Questions

Meta Values Alignment (Move Fast, Honesty, Impact, etc.)

Demonstrating alignment with Meta values—examples of moving fast while maintaining quality, being intellectually honest even when it's unpopular, building things that create real impact. Showing how you'd fit into Meta culture.

Practice Interview

Study Questions

Communication & Stakeholder Management

How you communicate complex technical concepts to different audiences—engineers, managers, executives. How you handle misalignment and different perspectives. Examples of times you've communicated through crisis or ambiguity.

Practice Interview

Study Questions

Cross-Functional Leadership & Influence

Demonstrating ability to lead initiatives that span multiple teams without direct authority. How you influence technical decisions, build consensus, and drive changes across organizational boundaries. Examples of successfully navigating complex technical or organizational situations.

Practice Interview

Study Questions

Mentoring & Technical Talent Development

Your approach to developing engineers—how you've mentored junior and senior staff, helped teammates grow, created learning opportunities. Examples of engineers you've developed and their growth trajectory. How you approach coaching on technical and professional development.

Practice Interview

Study Questions

Frequently Asked Site Reliability Engineer (SRE) Interview Questions

Deployment and Release StrategiesHardSystem Design

74 practiced

Design a deployment strategy for a global, active-active service across multiple regions where you must upgrade software with minimal user-facing disruption. Discuss traffic shifting, data consistency, failover considerations, and how to stage upgrades across regions.

Sample Answer

Requirements & constraints:- Active-active across N regions, <X ms user impact, zero data loss, support rolling upgrades with quick rollback, maintain SLOs.

High-level strategy:1. Progressive, region-staggered rolling upgrades using canary → ramp → full, combined with feature flags and traffic-weighted global load balancing (GSLB / DNS + Anycast LB or Cloud LB with weighted routing).

Architecture & components:- Global traffic manager (GSLB/Cloud LB) with per-region weights and health checks.- Per-region control plane: Kubernetes cluster + service mesh (Istio/Linkerd) for fine-grained routing, retries, and circuit breaking.- Data layer: prefer region-local primary for low-latency reads with async replication + global consensus for cross-region writes when strong consistency required (use Raft/Paxos or Spanner/Cloud Spanner). Alternatively, multi-master with CRDTs/conflict resolution for eventual consistency use cases.- CI/CD: automated pipelines that run integration, canary, and chaos tests; runtime feature flags and automated rollback hooks.- Observability: end-to-end tracing, per-region SLOs, synthetic checks, traffic & error dashboards, automated alerting.

Traffic shifting:- Canary in one AZ/region at 1-5% traffic. Observe latency, error budget, business metrics for predefined window.- Ramp by doubling weights (5 → 10 → 25 → 50) with health gates; if metrics breach thresholds, automatically rollback to safe weight and alert.- Use service-mesh splits for intra-cluster canaries (header-based, cookie stickiness) to avoid DNS TTL issues.

Data consistency & schema changes:- Use backward-compatible schema changes: deploy new code that can handle both old+new schema, run data migration asynchronously (dual-write with CDC), then switch reads.- For writes needing strong consistency: route those to a global leader or use quorum writes. During upgrade, ensure leader stays on stable version or run leader election with version-awareness.- If using multi-master, implement deterministic conflict resolution and a reconciliation job; avoid protocol changes that break binary compatibility.

Failover considerations:- Health checks at LB + app + readiness; service mesh circuit breakers to avoid cascading failures.- Automated cross-region failover policy: promote warm standby region or redirect traffic weights; ensure data catch-up via CDC before accepting writes or use last-writer-wins only where acceptable.- Maintain runbooks & automated playbooks for manual intervention.

Staging upgrades across regions (practical plan):1. Pre-flight: run CI, integration, blue/green smoke tests, load tests; verify DBA migration plan.2. Stage 0: Canary region with 1% traffic for 30–60 min. Monitor SLOs, logs, traces, business metrics.3. Stage 1: Ramp in region to 25–50% over controlled intervals (30–60 min windows).4. Stage 2: Expand to secondary region(s) staggered by time zone (minimize blast radius), repeating canary/ramp.5. Stage 3: Global cutover once all regions stable.6. Post-upgrade: run validation jobs, reduce monitoring thresholds back to normal, keep canary monitoring for longer window.

Automation & rollback:- Automate weight adjustments and health gate checks; automatic rollback triggers on SLO breach.- Keep immutable artifacts and blue-green images to switch back instantly.- Maintain automated DB rollback-safe migrations (backfill scripts, tombstones).

Trade-offs:- Strong consistency increases latencies and complexity (consensus), but simplifies correctness.- Eventual consistency simpler/fast but needs conflict handling and careful UX expectations.- Staggered upgrades lengthen total rollout but reduce blast radius.

Key operational practices:- Define clear SLOs + error budgets per region.- Frequent practice drills: simulated rollbacks, failovers, chaos engineering.- Detailed runbooks, postmortems, and incremental improvement after each rollout.

Automation and ScriptingMediumTechnical

81 practiced

Write a Bash wrapper 'safe_cron.sh' suitable for running from crontab that accepts a command and uses flock to ensure only one instance runs at a time across hosts using an NFS-shared lock file. Include timeout behavior, descriptive exit codes, and a --force flag to break stale locks. Explain NFS caveats in comments.

Sample Answer

To ensure only one instance runs (across hosts using an NFS-shared lock file), safe_cron.sh uses flock on a shared lock file descriptor, supports a configurable timeout, returns descriptive exit codes, and exposes --force to remove a stale lock file and retry. NFS caveats are documented in comments.

bash

#!/usr/bin/env bash
# safe_cron.sh - run a command under an NFS-shared flock.
# Usage: safe_cron.sh [--timeout=SECONDS] [--force] -- command args...
# Exit codes:
# 0  = command ran & exited 0
# 100 = couldn't acquire lock within timeout
# 101 = lock broken by --force but still couldn't lock
# 102 = command failed (proxy - command's exit code returned)
# 103 = usage/error
#
# NFS caveats:
# - flock requires a filesystem that supports POSIX locks. Many NFS setups do, but
#   some misconfigured NFS servers/clients or kernel versions may not honor flock()
#   across hosts. Test in your environment. If unreliable, consider a DLM or consul/etcd.
# - Avoid using stale PID semantics; we use flock on the file descriptor, not PID files.

set -euo pipefail

TIMEOUT=60
FORCE=0

# parse args
while [[ $# -gt 0 ]]; do
  case "$1" in
    --timeout=*) TIMEOUT="${1#*=}"; shift ;;
    --timeout) TIMEOUT="$2"; shift 2 ;;
    --force) FORCE=1; shift ;;
    --) shift; break ;;
    -h|--help) echo "Usage: safe_cron.sh [--timeout=SECONDS] [--force] -- command..."; exit 103 ;;
    *) echo "Unknown arg: $1"; exit 103 ;;
  esac
done

if [[ $# -lt 1 ]]; then
  echo "No command provided"; exit 103
fi

# canonicalize lock path on NFS; choose a path writable on all hosts
LOCK_DIR="/var/lock/safe_cron"
mkdir -p "$LOCK_DIR"
LOCK_FILE="$LOCK_DIR/$(basename "$1").lock"

# open FD 200 for lock file
exec 200>"$LOCK_FILE"

# If --force, try to break stale lock by truncating file (not guaranteed safe on NFS)
if [[ $FORCE -eq 1 ]]; then
  # Best-effort: remove file to encourage other hosts to release. This is racy.
  rm -f "$LOCK_FILE" || true
  # re-open
  exec 200>"$LOCK_FILE"
fi

# Try to acquire exclusive flock with timeout (in seconds)
if flock -w "$TIMEOUT" 200; then
  # We have the lock; run the command and capture exit
  "$@"
  STATUS=$?
  # release by closing fd
  exec 200>&-
  if [[ $STATUS -ne 0 ]]; then
    exit 102
  fi
  exit 0
else
  # failed to acquire lock within timeout
  if [[ $FORCE -eq 1 ]]; then
    # even after --force we couldn't lock
    exit 101
  else
    exit 100
  fi
fi

Key points:- Uses flock -w for timeout; returns clear exit codes for monitoring/alerts.- --force does a best-effort remove of lock file, but on NFS this may not always break an active lock — test and prefer a distributed lock service if correctness is critical.

Container Orchestration and Kubernetes OperationsHardSystem Design

72 practiced

You manage upgrades for hundreds of Kubernetes clusters across environments. Propose an automation and governance strategy for cluster version upgrades that ensures safety, compliance, minimal downtime and consistent configurations. Include fleet management tools, canary cluster upgrades, automated testing, rollback strategies, and metrics to monitor upgrade health.

Sample Answer

Requirements:- Safe, auditable, low-downtime upgrades across hundreds of clusters, consistent config, ability to canary and rollback, measurable SLO impact.

High-level approach:1) Fleet lifecycle control plane (Cluster API for self-managed clusters OR managed-provider APIs for cloud clusters) + GitOps for desired state (ArgoCD/Flux).2) Policy and gating (OPA/Gatekeeper or Kyverno) + CI/CD pipelines (GitHub Actions/GitLab/Jenkins) that orchestrate upgrade waves.3) Canary + progressive rollout controller (Argo Rollouts or custom controller) to advance upgrades across cluster groups.4) Automated test harness and observability to validate each stage and trigger rollback.

Architecture & components:- Fleet manager: Cluster API + Cluster Registry (e.g., Cluster API + Crossplane or Rancher/Anthos/GKE Fleet) to group clusters by environment, owner, SLA.- GitOps repos: cluster-profiles and kustomize/helm overlays per group; upgrade PRs automated by pipeline.- Orchestrator: Pipeline triggers rolling upgrade job that targets cluster groups: canary -> small wave -> full.- Gate/policy: OPA/Gatekeeper enforces security, allowed images, API deprecations before merge.- Testing: automated preflight (kube-conformance, CIS checks), smoke tests (app endpoints), integration tests (service mesh checks), chaos/soak as applicable.- Observability: Prometheus + Thanos (federation) + Loki + Grafana; tracing with Jaeger.

Upgrade flow:1. Create upgrade proposal (bump control-plane & node image) in Git repo; PR triggers preflight CI (lint, policy check).2. If pass, pipeline applies to a canary cluster/group (non-critical staging) via GitOps; Cluster API executes upgrade.3. Automated verification runs: node readiness, kube-apiserver latency, kubelet restarts, PodDisruptionBudget adherence, app smoke tests, e2e tests. Wait window for SLOs.4. If metrics within thresholds, advance to next wave; otherwise automatic rollback of cluster to previous version (Cluster API machine rollout revert or restore from machine image snapshot) and open incident with diagnostics.

Rollback strategy:- Pre-upgrade snapshots: node image/instance templates and etcd snapshots for self-managed; managed clusters use provider rollback API/snapshot.- Automated rollback triggers: failing health checks, increased error budget consumption, pod crash loops, API errors. Rollback is executed via Cluster API or provider reprovision using previous machine image + applying previous GitOps state.- Manual override with RBAC approvals for emergency cases.

Automated testing details:- Preflight: kubeconformance, CIS, schema checks (kubernetes version compatibility for CRDs), admission controller compatibility.- Post-upgrade smoke: kube-system pods health, control plane metrics, app readiness probes, integration tests hitting critical endpoints.- Canary app-level traffic shifting using service mesh (Istio/Linkerd) or LB weights to validate app behavior before upgrading app clusters.

Governance & compliance:- Policy as code (OPA/Gatekeeper) in pipeline; upgrade windows and maintenance calendar enforced.- Approval workflow: staged approvals for prod clusters (SRE on-call + app owners).- Audit: all upgrade PRs, runs, rollbacks logged in centralized audit (ELK/Loki + immutable storage).- RBAC and least privilege for upgrade orchestration.

Metrics & SLOs to monitor upgrade health:- Upgrade success rate (per version, per week)- Time-to-upgrade per cluster and fleet- API server request latency and error rate (p95/p99)- Node readiness and node reboot rate- Pod restart rate and CrashLoopBackOff counts- Service error rate and latency for critical endpoints- SLO/error budget consumption during/after upgrade- Mean time to detect/rollback- Number of manual interventions

Scalability & trade-offs:- Use Cluster API + GitOps for scale and consistency; managed providers simplify control plane upgrades but reduce low-level control.- Conservative waves reduce blast radius but increase total upgrade duration.- Automated rollback improves safety but requires reliable snapshots and tested rollback paths.

This strategy provides automated, auditable progressive upgrades with policy gates, automated testing, and clear rollback triggers while exposing metrics SREs need to ensure minimal downtime and compliance.

Failure Detection and Automated ResponseEasyTechnical

122 practiced

Explain heartbeat-based failure detection (periodic pings/heartbeats). For a distributed worker fleet of 10k workers, what heartbeat interval and timeout would you choose and why? Discuss trade-offs regarding network noise, detection latency, and scale considerations.

Capacity Planning and Resource OptimizationHardSystem Design

20 practiced

You must size capacity for a distributed NoSQL store that must handle 1M reads/sec and 200k writes/sec. Describe how you would dimension nodes for CPU, memory, disk IOPS, and network, and how to account for background operations such as compaction, repair, and garbage collection. Explain how you would plan for read tail latency spikes and required replication bandwidth.

Sample Answer

Requirements & assumptions (clarify up-front):- 1,000,000 reads/sec, 200,000 writes/sec steady-state.- Replication factor RF=3, synchronous replicate to 2 followers.- Average item size = 1 KB payload + 200B metadata = 1.2 KB.- Read: mostly single-key point reads; 95% hits in cache assumed possible.- Use SSD NVMe; nodes homogeneous.

Capacity math:- Raw request bandwidth (payload only): (1,000,000+200,000) * 1.2KB ≈ 1.44 GB/s inbound+outbound combined.- Replication network: each write must be sent to RF-1 = 2 replicas => write replication traffic = 200k *1.2KB*2 ≈ 480 MB/s.- Read traffic served from nodes: assume 50% reads served from replicas; worst-case network egress ≈ 1,000,000 *1.2KB ≈ 1.2 GB/s. Plan for 2–3 GB/s total egress across cluster.

Node sizing (example baseline):- Network: 100 Gbps per node (~12.5 GB/s) gives headroom for spikes; can host multiple rack placements. With 100 Gbps, cluster-level egress demand (≈1.2–3 GB/s) divided across N nodes.- CPU: assume ~200–400 CPU cores total needed for request processing, compaction and GC. Per node: 32–64 vCPUs (continuous low-latency processing + background threads).- Memory: allocate enough cache + JVM heap (if Java store). Example: 256–512 GB RAM per node (256 GB heap reserved carefully; leave host memory for page cache).- Disk/IOPS: writes = 200k/sec (durable writes). With 1.2KB per write => 240 MB/s sequential write plus compaction rewrite amplification (x2–3) -> plan 600 MB/s sustained per cluster. Per node NVMe: 3–6 GB/s and 300k IOPS (4KB) comfortable. Use many drives; target <70% utilization.

Accounting background ops:- Compaction/repairs: model write amplification factor (WAF). If WAF=2.5, effective write bandwidth = 200k*1.2KB*2.5 = 600 MB/s. Provision IOPS/throughput for that plus headroom. Schedule major repairs/anti-entropy during low-traffic windows or throttle via token bucket; prioritize read/write latency by throttling compaction when percentiles exceed SLO.- GC: minimize stop-the-world pauses: run with G1/ZGC or native runtimes. Keep heap per JVM modest (≤32–64GB) and use many service threads across containers to limit GC impact. Monitor and set alerts for GC pause tail.- Background I/O isolation: use cgroups/blkio or QoS on disk to reserve IOPS for foreground operations.

Planning for read tail latency spikes:- Multi-tier cache: use local node memory cache + global edge caches (CDN/redis) to absorb spikes; aim to serve 80–95% reads from cache.- Client-side retries with jitter and hedged reads: issue secondary read after 5–10ms to another replica for high-percentile mitigation.- Prioritized scheduling: prioritize small reads over compaction I/O; use io_uring or native async IO.- Autoscaling: add nodes when request rate sustained; use rolling restarts to avoid correlated GC/compaction.

Replication bandwidth & durability:- With RF=3 plan for ~480 MB/s replication + overhead (protocol, TLS) -> provision ~600 MB/s margin.- For cluster elasticity, ensure leaderless or configurable quorum to avoid single-node bottlenecks. Use batching for replication (group writes into 10–20KB batches) to reduce per-write overhead.

Operational practices:- SLOs: set read/write p99/p999 targets; monitor CPU, GC, disk latency, IOPS, network.- Load testing: synthetic load that exercises compaction and repair simultaneously to validate tail behavior.- Capacity formula summary: provision per-node network/IO such that (cluster throughput)/(nodes) ≤ 50% of a node’s network/disk capacity to allow headroom for background ops and failure scenarios.

Example final sizing (rough):- Nodes: 50 nodes, each: 100 Gbps NIC, 48 vCPUs, 384 GB RAM, NVMe 8TB (3–5 GB/s, 300k IOPS). This yields cluster headroom for replication, compaction, failover — adjust after benchmarking with real data and WAF.

Blameless Postmortem and Organizational LearningEasyTechnical

40 practiced

What constitutes a 'near-miss' incident in production, and how would you run a lightweight blameless postmortem to capture actionable learnings without overburdening teams? Describe the minimum artifact you would expect from such a review.

Deployment and Release StrategiesMediumSystem Design

67 practiced

Design a canary release process for a payment-processing microservice handling 1M transactions per day. Describe stages, metrics to monitor (latency, error rate, business metrics), safety gates, and an automated rollback policy. Include how you would test fraud detection or other critical downstream integrations during the canary.

Sample Answer

Requirements & constraints (clarify up front):- 1M tx/day (~11.6 tx/s avg, peaks higher); payments are high-risk (fraud, money movement).- Zero/low customer impact target; quick detection & automated rollback required.- Environment: Kubernetes + service mesh (Istio/Linkerd) and CI/CD pipeline.

High-level canary stages1. Pre-canary (validation)- Build verification, unit/integration tests, DB migrations dry-run, vulnerability scan.- Run synthetic end-to-end payment flows against staging with representative data.2. Canary rollout (0% → 1% → 5% → 25% → 100%)- Use traffic shaping (service mesh or load balancer weights) + feature flag to divert real traffic.- Hold at each stage for a defined observation window (typ. 10–30 minutes for early ramps, 60–120 minutes at larger ramps), longer during peak hours.3. Post-canary verification- Extended monitoring for 24–72 hours; compare long-tail metrics and business metrics.4. Full promotion or rollback

Metrics to monitor (real-time + aggregated)- Reliability & performance: - Error rate (5xx/4xx per endpoint) — alert if relative increase > X% absolute threshold (e.g., 0.1% → 0.5%). - Latency percentiles (p50/p95/p99) — alert if p99 increases > 50% or absolute > SLAs. - Throughput (tx/s) per pod & overall. - Resource utilization (CPU, memory, GC pauses).- Business metrics: - Transaction success rate / approval rate. - Decline reason distribution (auth failures, insufficient funds, fraud flags). - Revenue-per-minute, avg transaction value. - Fraud metrics: suspected fraud rate, false positives/negatives compared against baseline.- Observability: - Traces for tail-latency analysis, structured logs correlated by transaction-id, metrics tagged by canary=true/false so canary-vs-stable comparison is trivial.

Safety gates (automated & manual)- Automated gates at each ramp: require all canary metrics to be within safe bounds vs baseline for a sustained window (e.g., 10m sliding window). - Example thresholds: error rate not above baseline + 3σ and < 0.5% absolute; p99 latency within 1.5x baseline.- Business gate: approval rate drop < 0.2% absolute; suspicious-fraud-rate not increased > 20%.- Manual gate before >25% ramp for high-risk releases (product owner or SRE sign-off).- Circuit breaker: per-endpoint rate-limits and per-pod failure budgets to stop forwarding traffic to unhealthy instances.

Automated rollback policy- Rollback triggers (any sustained condition for T minutes, default T=5–15 min depending on severity): - Error rate spike: canary error rate > threshold for >5 minutes. - Latency spike: p99 > threshold for >10 minutes. - Business-impacting: approval-rate decrease beyond threshold or fraud-signal increase for >5 minutes. - Resource exhaustion or OOMs on canary pods.- Rollback action: - CI/CD automation reverts traffic weights to 0% for new version, scale down canary pods, restore stable routing. - Create incident with automated runbook: attach recent traces/logs, snapshot of metric diffs, promote hotfix pipeline if needed. - If rollback fails, emergency kill-switch: disable payments path (fallback to read-only / queue processing) and switch to degraded mode (e.g., enqueue and process async) to protect users.- Post-rollback: automated postmortem data capture and block further promotion until root cause resolved.

Testing fraud detection & downstream integrations during canary- Shadow/mirroring: mirror 100% of production traffic to the new service instance or fraud service in “observe-only” mode so it sees real events without affecting decisions. Compare decisions offline.- Sampled live checks: for a small percentage (e.g., 1%) run canary-fraud-decisions in-path but mark transactions as “do-not-apply” unless both stable and canary agree. Use canary flags to prevent money movement on transactions flagged only by canary.- Synthetic fraud scenarios: inject crafted transactions that should trigger specific fraud rules (velocity, geo anomalies); validate rule triggers and downstream webhook behaviors.- End-to-end test accounts: route low-risk test accounts with small amounts through canary and confirm settlement, notifications, clearing, and downstream systems (clearing house, ledger).- Integration contract tests: execute automated contract tests against downstream partners (auth, acquiring bank) in sandbox mode during ramp.- Compare outputs: define statistical checks (confusion matrix, drift) between stable vs canary fraud decisions; set thresholds for allowable divergence (e.g., false positive rate delta < 1%).- Data retention: keep detailed audit logs of all canary-decisions for forensic analysis.

Operational practices- Tag metrics and traces with deployment metadata (git sha, canary=true) for easy comparison.- Chaos / failure testing in staging: degrade downstream (slow auth, DB latency) and ensure canary behaves gracefully (timeouts, retries, circuit breakers).- Runbook & playbooks: clear steps to diagnose, rollback, and mitigate (failover to async processing, manual approval).- SLO & error budget awareness: avoid canaries during exhausted error budget or during business-critical peak unless emergency hotfix.

Reasoning & trade-offs- Gradual ramp balances risk vs speed; small initial windows catch severe regressions quickly.- Shadow testing captures fraud model differences without user impact.- Automated gates reduce MTTR; manual checkpoints before larger ramps add human judgement for business risk.- Conservative thresholds minimize financial risk; longer observation windows decrease false positives but slow rollout.

This process yields safe, observable, and automated canaries for a high-risk payments service while ensuring fraud detection and downstream integrations are validated before full promotion.

Automation and ScriptingHardTechnical

85 practiced

Provide concise Python pseudocode implementing a saga-style multi-step orchestration coordinator. Each step must implement 'execute' and 'compensate' methods; the coordinator should persist progress, retry idempotently on transient failures, and execute compensating actions if a fatal step fails. Show how you would store state and resume after restarts.

Sample Answer

Approach: orchestrator persists per-saga state (steps with statuses, results, idempotency tokens) in a durable store (e.g., KV DB / Redis / RDS). Coordinator loads state on start, resumes from first non-complete step, retries transient errors idempotently, and on a fatal failure runs compensations for completed steps in reverse order.

python

# python pseudocode

import time, json
from enum import Enum

class Status(Enum): PENDING = "PENDING"; DONE = "DONE"; FAILED = "FAILED"; COMPENSATING = "COMPENSATING"; COMPENSATED = "COMPENSATED"

class Step:
    def __init__(self, name):
        self.name = name
    def execute(self, ctx): raise NotImplementedError
    def compensate(self, ctx): raise NotImplementedError

# simple durable store interface (wrap Redis/RDB)
class Store:
    def save(self, saga_id, state_dict): ...
    def load(self, saga_id): ...

class Orchestrator:
    def __init__(self, steps, store, max_retries=5):
        self.steps = steps
        self.store = store
        self.max_retries = max_retries

    def start(self, saga_id, initial_ctx):
        state = self.store.load(saga_id) or self._init_state(saga_id, initial_ctx)
        try:
            self._run(state)
        finally:
            self.store.save(saga_id, state)

    def _init_state(self, saga_id, ctx):
        state = {
            "saga_id": saga_id,
            "ctx": ctx,
            "steps": [{ "name": s.name, "status": Status.PENDING.value, "retries":0 } for s in self.steps]
        }
        self.store.save(saga_id, state)
        return state

    def _run(self, state):
        for idx, step_meta in enumerate(state["steps"]):
            step = self.steps[idx]
            if step_meta["status"] == Status.DONE.value: continue
            if step_meta["status"] in (Status.FAILED.value, Status.PENDING.value):
                success = self._execute_with_retries(step, state, idx)
                if not success:
                    # fatal -> compensate completed steps in reverse
                    self._compensate_completed(state, idx-1)
                    raise Exception(f"Fatal step failure: {step.name}")
        # all done
        self.store.save(state["saga_id"], state)

    def _execute_with_retries(self, step, state, idx):
        meta = state["steps"][idx]
        while meta["retries"] < self.max_retries:
            try:
                # idempotency: step.execute should be safe to call multiple times using state["ctx"]
                result = step.execute(state["ctx"])
                meta["status"] = Status.DONE.value
                meta["result"] = result
                self.store.save(state["saga_id"], state)
                return True
            except TransientError:
                meta["retries"] += 1
                self.store.save(state["saga_id"], state)
                time.sleep(2 ** meta["retries"])
                continue
            except FatalError:
                meta["status"] = Status.FAILED.value
                self.store.save(state["saga_id"], state)
                return False
        # exceeded retries -> treat as fatal
        meta["status"] = Status.FAILED.value
        self.store.save(state["saga_id"], state)
        return False

    def _compensate_completed(self, state, last_completed_idx):
        for idx in range(last_completed_idx, -1, -1):
            meta = state["steps"][idx]
            if meta["status"] != Status.DONE.value: continue
            step = self.steps[idx]
            try:
                meta["status"] = Status.COMPENSATING.value
                self.store.save(state["saga_id"], state)
                step.compensate(state["ctx"])
                meta["status"] = Status.COMPENSATED.value
                self.store.save(state["saga_id"], state)
            except Exception:
                # log and continue – compensations should be retriable and idempotent
                meta["status"] = Status.FAILED.value
                self.store.save(state["saga_id"], state)

Key points:- Persist full saga state after each change so restarts resume exactly.- Steps must be idempotent (use unique tokens or check-before-create).- Retry transient failures with backoff; classify errors as transient vs fatal.- Compensate completed steps in reverse order on fatal failures.Complexity: orchestration control is O(n) steps; storage and retries dominate runtime. Edge cases: partial compensations, store outages (use durable leader election / at-least-once persistence), concurrent coordinators (use optimistic locking or single-owner lease).

Container Orchestration and Kubernetes OperationsHardSystem Design

66 practiced

Create a disaster recovery plan for stateful applications running on Kubernetes across two regions that meets an RTO of 1 hour and RPO of 5 minutes. Include backup strategies, cross-region replication approaches, automated recovery playbooks, DNS failover, and how you'd test and validate recovery objectives regularly.

Sample Answer

Requirements & constraints:- RTO ≤ 1 hour, RPO ≤ 5 minutes for stateful workloads across two regions (primary and secondary). Expect network partitions, region failover, and consistent client experience. Use Kubernetes-native and cloud-managed primitives where possible.

High-level architecture:- Active-passive primary region (writes → primary, secondary replicates up-to-5-min lag). Stateful workloads run on StatefulSets with PersistentVolumes backed by cloud block storage. Use cross-region object storage for backups and replication streams.

Core components & strategies:1. Continuous replication (RPO ≤5m)- Database-level replication: native async streaming (e.g., PostgreSQL logical replication, MySQL GTID, MongoDB replica set with delayed sync) configured to replicate WAL/oplog to secondary. Tune sync to force flush every 1–2s and network batching to bound lag <5m.- PV replication for non-db state: use application-level change capture to write events to replicated object store or message bus (Kafka MirrorMaker/Confluent Replicator) to replicate writes to secondary.

2. Backups (for point-in-time & corruption recovery)- Frequent incremental backups to cross-region object storage (every 5 minutes using WAL archiving or incremental snapshots). Full nightly backups. Retention, encryption, and immutable storage (object lock) for compliance.

3. Kubernetes & infra replication- GitOps for manifests (ArgoCD). Push same manifests to secondary cluster. Use cluster-autoscaler, managed control plane in each region.- Store secrets in cross-region secret store (e.g., Vault with multi-region replication).

Automated recovery playbooks (executable runbooks)- Orchestrated runbook (Argo Workflows / Terraform + scripts) that: 1. Promote replica DB to primary (ensure write-safe cutoff; apply last WALs). 2. Reconfigure application config map/secret to point to promoted DB. 3. Scale StatefulSets/Deployments in secondary cluster from zero to desired replicas. 4. Apply any required DB migrations in controlled manner. 5. Run health checks and smoke tests.- All steps automated with idempotent scripts, with human-in-the-loop checkpoints for irreversible steps (promotion).

DNS failover & traffic shift- Use low-TTL DNS (30s) + health-checked global load balancer (Cloud DNS + Global LB) or Route53 health checks with weighted routing. During failover, flip weight to secondary; background scripts update DNS and invalidate CDN. For clients requiring faster cutover, use Anycast + global LB to adjust route via health checks.

Testing & validation- Weekly DR drills: automated chaos test that simulates region failure, runs recovery playbook, measures RTO/RPO (time to full traffic + data loss). Maintain runbook metrics in SLO dashboard.- Quarterly full-scale restore from backups to staging cluster, validate data integrity and app function.- Continuous monitoring: replicate lag, backup success, restoration dry-run logs, and alert if replication lag >2min or backups fail.- Postmortem and playbook improvement after each test.

Trade-offs & notes- Active-active reduces RTO but increases complexity (conflict resolution). Chosen active-passive to meet strict RPO cheaply.- Accept small replication lag risk; frequent incremental backups mitigate corruption.- Invest in automation and frequent tests to keep RTO ≤1h and RPO ≤5m.

Failure Detection and Automated ResponseEasyTechnical

89 practiced

Compare active (synthetic/blackbox) monitoring vs passive (instrumentation/whitebox) monitoring. Provide concrete examples of signals each provides, strengths and weaknesses, and when you'd choose one over the other for detecting failures across a global e-commerce stack (CDN, API gateway, backend services). Discuss cadence, cost, and blind spots.

Sample Answer

Active (synthetic/blackbox) vs Passive (instrumentation/whitebox) monitoring

Quick definition- Active (blackbox): external probes simulate user traffic from many locations (HTTP checks, synthetic transactions, RUM is a hybrid). Treats system as a black box.- Passive (whitebox): collects internal telemetry (metrics, logs, traces, health endpoints, profilers) from services themselves.

Concrete signals- Active: global HTTP status codes, end-to-end latency, DNS resolution time, TLS handshake time, CDN cache hit/miss as seen by a probe, payment checkout flow success rates from probes.- Passive: per-service CPU/memory, internal error counters, request latencies by span, DB connection pool saturation, application logs with stack traces, deployment/version tags.

Strengths & weaknesses- Active strengths: detects customer-visible outages (global reach), validates routing/edge config, catches problems even when telemetry pipeline is down. Weakness: limited internal visibility, synthetic load may not hit all code paths, cost grows with cadence × locations.- Passive strengths: rich root-cause data, fine-grained SLOs, resource and error context. Weakness: blind to CDN/DNS/ISP issues and to telemetry pipeline failures; requires instrumentation and access.

When to choose (global e‑commerce stack)- CDN: active probes from multiple ISPs/regions to detect cache misconfig, TLS/DNS problems, or POP-level outages.- API gateway: combine both—active health checks for availability and end-to-end latency; passive metrics/traces for routing/middleware errors and per-backend error rates.- Backend services: prioritize passive (metrics, traces, logs) to diagnose CPU spikes, DB latency, or code exceptions; add synthetic tests for critical user journeys (checkout) that exercise backend integration.

Cadence, cost, blind spots- Cadence: critical paths (checkout, login) run synthetic checks every 15–60s globally; less critical every 5–15m. Passive is continuous but watch cardinality of metrics/logs to control cost.- Cost: active cost = probes × locations × frequency (can be substantial at high cadence globally). Passive cost = ingest volume (high-cardinality traces/logs).- Blind spots: active misses internal state and low-volume edge-cases; passive misses network/DNS/CDN reachability and failures of the monitoring pipeline itself.

Recommended strategy- Use both: deploy global synthetic probes for customer-facing availability and edge issues; rely on rich instrumentation for root-cause analysis and SLOs. Alerting: trigger high-level page- or checkout-service pager from active outage; attach passive-context playbooks that surface traces/metrics automatically when paging.

Practice Site Reliability Engineer (SRE) questions across all topics

Additional Information

Want to create your own tailored preparation guide using our deep research?

Get Started for Free

Interview-Ready Courses

Visual-first, interactive, structured learning paths

Browse Site Reliability Engineer (SRE) jobs

AI-enriched listings across hundreds of company career pages

Explore Jobs

Meta Staff Site Reliability Engineer Interview Preparation Guide

Interview Process Overview

Interview Rounds

Recruiter Screening

What to Expect

Tips & Advice

Focus Topics

Motivation & Fit for Meta

Practice Interview

Study Questions

Understanding of SRE Discipline & Philosophy

Practice Interview

Study Questions

Mentorship & Team Development Approach

Practice Interview

Study Questions

Examples of Technical Leadership & Impact

Practice Interview

Study Questions

Career Narrative & Staff-Level Progression

Practice Interview

Study Questions

Technical Phone Screen 1: Infrastructure & Systems Knowledge

What to Expect

Tips & Advice

Focus Topics

Reliability Engineering Principles & SLOs

Practice Interview

Study Questions

Capacity Planning & Scalability

Practice Interview

Study Questions

Large-Scale System Architecture Design

Practice Interview

Study Questions

Infrastructure Automation Frameworks & Tooling

Practice Interview

Study Questions

Performance Monitoring & Observability Design

Practice Interview

Study Questions

Technical Phone Screen 2: Incident Response & Troubleshooting

What to Expect

Tips & Advice

Focus Topics

Post-Incident Review & Learning Process

Practice Interview

Study Questions

Communication & Coordination During Incidents

Practice Interview

Study Questions

Decision Making & Trade-offs Under Pressure

Practice Interview

Study Questions

Incident Triage & Root Cause Analysis

Practice Interview

Study Questions

Troubleshooting Methodology for Distributed Systems

Practice Interview

Study Questions

Onsite Round 1: Systems Design Interview

What to Expect

Tips & Advice

Focus Topics

SLO-Driven Design & Error Budgets

Practice Interview

Study Questions

Observability & Monitoring Architecture

Practice Interview

Study Questions

Multi-region & Geographic Architecture

Practice Interview

Study Questions

Operational Automation & Deployment Strategy

Practice Interview

Study Questions

Distributed System Trade-offs & Consistency Models

Practice Interview

Study Questions

Design for Reliability & Failure Handling