On Call Culture & Runbook Development Questions

Understand on-call responsibilities: on-call engineer is responsible for incident response for their services. Discuss runbooks and playbooks: step-by-step procedures for common incidents allowing quick diagnosis and mitigation. Know how to structure on-call rotations, define escalation paths, and support on-call engineers with good runbooks and documentation.

HardTechnical

83 practiced

Implement a Python function dedupe_alerts(alerts: List[Dict], window_seconds: int) -> List[Dict] that groups incoming alert events by (service, signature) within a sliding time window and returns deduplicated events with fields: service, signature, earliest_timestamp, count. Input example: [{'service':'auth','signature':'db-timeout','ts':1609459200}, ...]. Aim for O(n log n) or better.

Sample Answer

Approach: sort events by timestamp (to handle out-of-order arrival) and perform a single linear sweep grouping contiguous events with the same (service, signature) where each group's events lie within window_seconds from that group's earliest_timestamp. Sorting gives O(n log n), sweep O(n) — overall O(n log n) time, O(k) extra space for output and current-group state.

python

from typing import List, Dict, Tuple

def dedupe_alerts(alerts: List[Dict], window_seconds: int) -> List[Dict]:
    """
    Group alerts by (service, signature) into sliding windows of size window_seconds.
    Returns list of dicts: {service, signature, earliest_timestamp, count}
    """
    if not alerts:
        return []

    # Sort by timestamp ascending
    alerts_sorted = sorted(alerts, key=lambda a: a['ts'])

    result = []
    # key -> current group: (earliest_ts, count)
    current: Dict[Tuple[str, str], Tuple[int, int]] = {}

    for a in alerts_sorted:
        key = (a['service'], a['signature'])
        ts = a['ts']

        if key not in current:
            # start new group
            current[key] = (ts, 1)
        else:
            earliest_ts, count = current[key]
            if ts - earliest_ts <= window_seconds:
                # inside same window, increment
                current[key] = (earliest_ts, count + 1)
            else:
                # flush existing group
                service, signature = key
                result.append({
                    'service': service,
                    'signature': signature,
                    'earliest_timestamp': earliest_ts,
                    'count': count
                })
                # start new group with this event
                current[key] = (ts, 1)

    # flush remaining groups
    for (service, signature), (earliest_ts, count) in current.items():
        result.append({
            'service': service,
            'signature': signature,
            'earliest_timestamp': earliest_ts,
            'count': count
        })

    # Optional: stable ordering by earliest_timestamp
    result.sort(key=lambda r: (r['earliest_timestamp'], r['service'], r['signature']))
    return result

Key points:- Sorting handles out-of-order arrivals; sweep groups contiguous events per (service,signature).- This uses a per-key rolling window anchored at each group's earliest_timestamp (commonly desired for dedupe: group events within window of first occurrence).Time complexity: O(n log n) due to sort. Space: O(n) worst-case for output and O(k) for active groups.Edge cases:- Very large number of distinct keys (memory for `current` grows) — consider flushing inactive keys if timestamps advance far.- If you prefer fixed buckets (e.g., align to epoch windows) or tumbling windows, use floor(ts / window) as key instead.- For streaming high-throughput: replace sort with a time-ordered input or use a streaming buffer + eviction (deque) to maintain O(n) overall.

MediumTechnical

94 practiced

Create a runbook template for performing a primary relational database failover to a replica (e.g., Postgres). Include preconditions, replication lag thresholds, failover steps, verification queries, rollback steps, and stakeholder communication steps.

Sample Answer

Runbook: Primary → Replica Failover (Postgres)

Preconditions- Confirm replica exists and is configured for synchronous/asynchronous replication.- Backups: latest base backup + WAL archived; snapshot available.- Access: SSH to DB hosts, DB superuser credentials, orchestration/HA tool creds (repmgr/patroni/pg_auto_failover).- Maintenance window or incident declared; change control/approvals if required.

Replication lag thresholds- Safe failover: replication lag < 5s (recommended) or < 1000 WAL bytes for critical writes.- Caution: 5–30s — accept partial loss risk; >30s — do NOT failover without application impact assessment.

Immediate checks- Host reachable:

bash

ssh db-primary
pg_isready -h localhost -p 5432

- Replication status:

sql

-- on primary
SELECT client_addr, state, sent_lsn, write_lsn, flush_lsn, replay_lsn,
       pg_wal_lsn_diff(sent_lsn, replay_lsn) AS lag_bytes
FROM pg_stat_replication;

Failover steps (manual)1. Notify stakeholders (see comms section).2. Stop application writes/load-balancer → read-only or drain connections.3. Ensure replica is caught up: replay_lsn ≈ sent_lsn and lag within threshold.4. Promote replica:

bash

# repmgr:
repmgr standby promote -f /etc/repmgr.conf
# or Patroni:
patronictl promote cluster-name
# or:
pg_ctl -D /var/lib/postgresql/data promote

5. Update configuration: point primary DNS/virtual IP/load-balancer to new primary.6. Reconfigure old primary as replica (after ensuring it's safe): stop Postgres, reset recovery.conf/standby.signal, base backup from new primary.

Verification queries (on new primary)

sql

SELECT pg_is_in_recovery(); -- should be false
SELECT count(*) FROM critical_table WHERE updated_at > now() - interval '1 minute';
SELECT pg_stat_activity WHERE state='active';

Check application health and error rates.

Rollback steps- If promotion incorrect and old primary healthy: demote new primary (if tool supports) or promote old primary after re-synchronizing.- Re-sync by taking base backup from desired primary, restore to replica, start replication.- Validate data consistency (row counts, checksums) before redirecting traffic.

Stakeholder communication- Initial: Incident declared, impact, ETA, who is performing failover.- During: “Failover in progress” updates every 5–15 minutes.- Completion: New primary endpoint, validation results, ETA for full recovery.- Post-incident: RCA plan, expected corrective actions, timeline.

Notes & safety- Prefer automated HA tooling (Patroni/repmgr) for production.- Always record timestamps and commands executed in incident log.- Test this runbook in staging quarterly.

EasyTechnical

100 practiced

Outline a 4-week onboarding plan to make a new engineer ready for on-call. Include shadowing sessions, runbook study, simulated incidents (tabletop and live), access provisioning, and who should mentor them during their first solo shift.

EasyTechnical

91 practiced

Describe best practices for runbook storage, access permissions, and discoverability so an on-call engineer can reliably find and use the correct runbook during a major incident. Include considerations for offline access and sensitive information.

Sample Answer

Store runbooks in a single, authoritative, versioned, and searchable location (e.g., a docs repo + static site generator, Confluence with page versions, or a git-backed docs site). Use a standard template (purpose, symptoms, pre-reqs, steps, rollback, verification, owners, links) so engineers can scan quickly.

Access & permissions- Apply least-privilege RBAC: broad read access for on-call, write access for maintainers/owners.- Separate edit rights from emergency “exec” rights to avoid accidental changes during incidents.- Log and audit all changes and accesses.- For break-glass or escalated actions, require temporary elevated access with approval and audit trail.

Discoverability- Tag pages with service, severity, and alert keys; maintain a centralized index or runbook map.- Integrate runbook links into alert payloads, incident pages, and the on-call rotation tool so the alert directly points to the relevant runbook.- Provide quick-summary sections and step numbers; include a “when to use this” checklist at top.- Keep a curated “most-used” or “critical-path” shortlist for major incident responders.

Offline access & resilience- Provide an easily downloadable cached bundle (HTML/PDF) on call devices or a lightweight CLI tool that syncs runbooks; store a signed, read-only copy on on-call laptops.- For completely offline sites, keep a printed/emergency binder or local VM image with runbooks.

Sensitive information- Never embed credentials, keys, or private tokens directly. Reference a secrets store (Vault, AWS Secrets Manager) with explicit steps to retrieve secrets and required approvals.- Redact or encrypt any sensitive snippets; use role-bound access to vault secrets and require multi-person approval for highly sensitive actions.- Document escalation contacts and out-of-band verification methods in case systems for secrets or auth are unavailable.

Why this works: single source + templates + integrations reduce time-to-action; RBAC + audit protect safety; secret references prevent leakage; offline copies and clear tagging ensure availability and fast discovery during high-pressure incidents.

EasyTechnical

88 practiced

Design a runbook template for common incidents. For each section include purpose and a short example: symptom summary, prerequisites, immediate mitigation steps (1-3 actions), verification checks, rollback steps, escalation contacts, links to dashboards, and post-incident actions.

Unlock Full Question Bank

Get access to hundreds of On Call Culture & Runbook Development interview questions and detailed answers.

Join thousands of developers preparing for their dream job.