Operational Excellence and Resilience Questions

Design and operationalize systems, teams, and processes that deliver efficiency, cost effectiveness, and resilient service delivery. Cover cost optimization and right sizing, automation and self healing processes, monitoring and observability (or the equivalent operational visibility for non-technical workflows), service level objectives and agreements, incident response and disaster recovery planning, resilience testing (including chaos engineering for technical systems), capacity planning, and continuous improvement practices such as postmortems and operational maturity models. Candidates should be able to explain trade offs between cost and reliability, how they instrument and alert on the health of a system or process, and how they measure and improve operational maturity for their function, whether that function is a software platform, an IT organization, or a business operations team.

HardTechnical

34 practiced

As incident commander during a major outage you must decide whether to rollback a recent deployment even though telemetry is incomplete. Describe a decision framework you would use: what signals to consider, how to assess rollback safety, rollback execution safeguards, stakeholder communications, and how to capture the decision and rationale for the post-incident review.

Sample Answer

Situation: I’m incident commander during a major outage that started shortly after a deployment. Telemetry is incomplete (metrics delayed, traces sampled). I must decide whether to rollback.

Decision framework (quick checklist):- Timebox initial analysis (e.g., 10–15 minutes) to avoid paralysis.- Ask: Is the deployment the most likely cause? (correlation vs causation)- Decide threshold: rollback if risk of continued harm > risk of rollback.

Signals to consider:- Deployment metadata: scope, services changed, config flags, new DB migrations.- User-facing signals: error rate, latency, traffic drops, feature-specific failures.- System signals: CPU/memory, thread pools, queue/backlog growth, downstream errors.- Recent logs and traces (even sampled), CI/CD test results, canary/health-check failures.- External reports: customer incident reports, on-call messages.- Blast radius: percentage of users / regions affected.

Assessing rollback safety:- Check if deployment included irreversible changes (migrations, schema, data writes). If yes, rollback may be unsafe.- Validate backward compatibility of schema and APIs; check migration rollback scripts and their tested status.- Confirm dependencies: if other services already adapted to new behavior, rollback could cascade.- Evaluate rate of data divergence — can you reconcile later?

Rollback execution safeguards:- Prefer feature-flag disable over full rollback when possible (safer, faster, less data risk).- If full rollback required: - Use staged rollback (canary -> region -> global). - Pre-run a rollback dry-run on staging or low-traffic region if time permits. - Ensure runbook and runbook owner present; require two-person approval for destructive steps. - Implement traffic steering (load balancer or fleet control) and monitor healthchecks. - Preserve artifacts: keep current deployment images/tags and record commit hashes.

Stakeholder communications:- Immediate: brief on-call, SRE, product lead, PM with one-line status: impact, hypothesis, planned action, ETA for next update.- Regular cadence (every 10–15 minutes) with short updates; use single source (incident channel + status page).- Be explicit about uncertainty and decision criteria (e.g., “Telemetry incomplete; will rollback if errors > X% or latency > Yms for 10m”).- If rollback approved, notify customer-facing teams to prepare messaging and support scripts.

Capturing decision and rationale:- Log decision in incident timeline with who decided, time, evidence reviewed, chosen action and fallback plan.- Record signals that would trigger reversal (success/fail criteria) and exact rollback steps taken (commands, artifacts).- Tag relevant commits/releases and attach dashboards/snapshots to post-incident doc.- Post-incident: run a blameless review focusing on telemetry gaps, rollback safety, and improving runbooks, then add actions (improve metrics, test rollback automation, require feature flags).

Example short phrasing during incident:“Hypothesis: deployment 2025.11 is likely cause. Telemetry incomplete. Plan: disable feature-flag A in canary within 10m; if errors persist or backlog grows after 5m, proceed staged rollback (canary → us-east → global). Two-person approval required for destructive DB rollback. Next update in 10m.”

This balances speed with caution: act quickly to protect users, but avoid unsafe rollbacks when data loss or incompatibility risks are high.

EasyTechnical

29 practiced

In Python (Flask or FastAPI), design a /health HTTP endpoint that: 1) checks two dependencies (a SQL DB and a Redis cache) with individual timeouts (DB 200ms, cache 50ms); 2) returns JSON containing each dependency's status and latency; 3) returns overall HTTP 200 if all checks pass otherwise 503. Describe how you would avoid blocking worker threads for slow checks and what readiness vs liveness semantics you would expose in Kubernetes.

Sample Answer

Approach: implement an async endpoint (FastAPI example) that runs dependency checks concurrently with per-check timeouts using asyncio.wait_for. Use async DB/cache clients (asyncpg, aioredis) where possible; if only sync clients exist, run them in a threadpool via asyncio.to_thread to avoid blocking worker threads. Collect per-dependency status, latency, and return 200 only if all are healthy; otherwise 503.

python

from fastapi import FastAPI, Response, status
import asyncio
import time
import asyncpg
import aioredis

app = FastAPI()

DB_DSN = "postgresql://user:pass@db:5432/mydb"
REDIS_URL = "redis://redis:6379/0"

async def check_db(timeout=0.2):
    start = time.perf_counter()
    try:
        async def probe():
            conn = await asyncpg.connect(DB_DSN)
            await conn.execute("SELECT 1")
            await conn.close()
        await asyncio.wait_for(probe(), timeout)
        return {"status": "ok", "latency_ms": int((time.perf_counter()-start)*1000)}
    except Exception as e:
        return {"status": "fail", "latency_ms": int((time.perf_counter()-start)*1000), "error": str(e)}

async def check_redis(timeout=0.05):
    start = time.perf_counter()
    try:
        async def probe():
            r = await aioredis.from_url(REDIS_URL)
            await r.ping()
            await r.close()
        await asyncio.wait_for(probe(), timeout)
        return {"status": "ok", "latency_ms": int((time.perf_counter()-start)*1000)}
    except Exception as e:
        return {"status": "fail", "latency_ms": int((time.perf_counter()-start)*1000), "error": str(e)}

@app.get("/health")
async def health():
    # run checks concurrently and enforce individual timeouts inside each check
    db_task = asyncio.create_task(check_db())
    redis_task = asyncio.create_task(check_redis())
    results = await asyncio.gather(db_task, redis_task)
    response = {"dependencies": {"db": results[0], "redis": results[1]}}
    overall_ok = all(d["status"] == "ok" for d in response["dependencies"].values())
    return Response(content=app.json_encoder().encode(response),
                    status_code=status.HTTP_200_OK if overall_ok else status.HTTP_503_SERVICE_UNAVAILABLE,
                    media_type="application/json")

Key points:- Use async clients or asyncio.to_thread to run blocking calls off the event loop.- Use asyncio.wait_for for per-check timeouts (200ms DB, 50ms cache).- Return per-dependency status and latency_ms; include error messages for diagnostics.- Avoid blocking worker threads so server stays responsive under slow dependencies.

Readiness vs Liveness:- Liveness endpoint: simple, very fast check (e.g., always 200 or minimal self-check) to ensure process isn't dead; avoid heavy external checks to prevent restarts due to transient downstream issues.- Readiness endpoint: perform full dependency checks (DB, Redis) to indicate whether traffic should be routed here. Use the /health/readiness path mapped to the logic above. Kubernetes: liveness probe -> lightweight local check; readiness probe -> dependency-aware health that returns non-ready (503) so the pod is removed from service until dependencies recover.

MediumTechnical

32 practiced

Write a concise runbook that an on-call engineer can execute to restore service when a read replica becomes read-only or when replication lag exceeds acceptable thresholds. Include detection thresholds, commands to inspect replication state, immediate mitigations (e.g., redirecting reads), steps to promote or reconfigure replicas, and post-recovery verification checks.

Sample Answer

Situation: You are on-call and notice a read-replica is read-only or replication lag > threshold. Follow this concise runbook to restore service safely.

Detection thresholds- Alert when replica replication_lag > 5s for OLTP-critical, > 30s for analytical; or replica reports read_only = ON unexpectedly; or >5% of app reads fail.

Immediate inspection (run as DBA/ops user)- Check replica status (MySQL/MariaDB):

bash

mysql -e "SHOW SLAVE STATUS\G"

- Check replication lag (Postgres):

bash

psql -c "SELECT now()-pg_last_xact_replay_timestamp() AS lag;"

- Check read-only flag:

bash

mysql -e "SELECT @@read_only;"
psql -c "SHOW default_transaction_read_only;"

- Check disk, IO, CPU, network:

bash

top; iostat -x 1 3; df -h

Immediate mitigations (fast, reversible)1. Redirect new read traffic away from the unhealthy replica to other replicas or primary via load balancer / service discovery. Example: disable node in HAProxy/ProxySQL or mark instance OUT in service registry.2. If app can tolerate stale reads, lower SLA by switching to other replicas; notify stakeholders.

Short-term fixes (non-destructive)- If replication thread stopped, try restart (MySQL):

bash

mysql -e "START SLAVE;"

- For Postgres:

bash

pg_ctl reload && repmgr standby clone/check; -- use cluster tooling

- If IO or disk pressure, free space (rotate logs), kill long queries causing bloat.

When to promote a replica- Promote when master unreachable or replica consistently healthy and up-to-date (lag < 1s) and failover authorized.- Verify replica has all transactions: MySQL SHOW SLAVE STATUS Slave_IO_State = 'Waiting for master to send event' and Seconds_Behind_Master = 0; Postgres replay timestamp ~ master.

Promote steps (MySQL example)1. Stop writes to old master (if alive): put it in maintenance.2. On chosen replica:

bash

mysql -e "STOP SLAVE; RESET SLAVE ALL; SET GLOBAL read_only=OFF;"

3. Point app write endpoint to new master and update DNS/HAProxy/ProxySQL.4. Reconfigure other replicas to replicate from new master:

bash

CHANGE MASTER TO MASTER_HOST='new-master', MASTER_USER='repl', MASTER_PASSWORD='pw', MASTER_LOG_FILE='file', MASTER_LOG_POS=pos;
START SLAVE;

Promote steps (Postgres example)- Use repmgr/Patroni/etcd workflow; manual:1. Promote:

bash

pg_ctl promote -D /var/lib/postgresql/data

2. Reconfigure old primary (when recovered) as replica: update recovery.conf / primary_conninfo and start.

Post-recovery verification- Confirm replication healthy: Seconds_Behind_Master ≈ 0 or lag < threshold.- Run smoke tests: read queries, write to master, consistency checks (row counts, checksums).- Monitor metrics and alerts for 30–60 minutes.- Run integrity: compare high-watermark txn IDs / binlog positions.- Document incident: root cause, actions taken, timeline, and postmortem.

Notes and safety- Prefer redirecting reads over promoting unless master is dead.- Always snapshot binlog/LSN before destructive ops.- Communicate with team and stakeholders before failover if SLA allows.

EasyTechnical

37 practiced

Define and differentiate SLI, SLO, and SLA in the context of a public HTTP JSON API used by external customers. For each term provide: a concise definition, one concrete example metric (name, measurement window and denominator), and explain how each is used operationally during incident response and (for SLAs) contractual enforcement.

EasyTechnical

36 practiced

Compare horizontal autoscaling and vertical autoscaling. Describe typical implementations (e.g., Kubernetes HPA vs changing VM flavor), pros and cons of each, and one concrete scenario where vertical scaling is preferable over horizontal scaling despite its limitations.

Unlock Full Question Bank

Get access to hundreds of Operational Excellence and Resilience interview questions and detailed answers.

Join thousands of developers preparing for their dream job.