On Call and Work Availability Questions

Candidate availability expectations and flexibility for operational responsibilities. Topics include on call commitments, shift schedules, time zone constraints, responsiveness during urgent incidents, ability to participate in drills and on demand mitigation, and honesty about personal constraints. Interviewers may probe for preferred schedules, limits on availability, and willingness to handle urgent infrastructure issues.

EasyTechnical

67 practiced

Explain what being on-call as a Site Reliability Engineer entails at an enterprise scale. Include immediate responsibilities during an alert (acknowledgement, triage, mitigation), expectations for response time and escalation, typical non-incident duties while on-call (monitoring, runbook updates, automation), and how on-call links to SLOs and error budgets.

EasyBehavioral

77 practiced

Tell me about a time you needed to decline or limit on-call availability due to a personal constraint (travel, family, medical). Describe how you communicated the constraint, what mitigations you proposed, and how the team responded. Use the STAR structure.

Sample Answer

Situation: Two weeks before a planned international medical appointment for a close family member, I realized I wouldn’t be reliably reachable during our normal on-call window (significant timezone differences and limited phone access).

Task: I needed to decline full on-call for the two-week period while ensuring our SLOs and on-call SLAs stayed intact and incidents would still be handled quickly.

Action:- Notified my manager and the on-call rotation owner immediately (10 days before travel) with written details: exact dates, timezone gaps, and expected availability windows.- Proposed a mitigation plan: - Swapped my on-call shifts with two teammates who agreed to cover my primary week and on-call errands; documented the swap in the rota. - Prepared an updated runbook for my services: step-by-step playbooks, key dashboards, top-5 failure modes, and remediation commands. - Implemented two small automations before leaving: a health-check script that restarts a flaky worker and an alert threshold tweak to reduce noisy warnings during my travel window. - Added explicit escalation paths and an “incident owner” for my services so PagerDuty would route to the correct secondary if needed. - Scheduled a 30-minute handoff meeting and shadowing session with the covering engineers; made myself available for one scheduled daily 30-minute check-in during my limited connectivity windows.- Communicated all changes to the broader on-call rotation via Slack and our runbook repository, and left a clear note in the incident timeline for auditors.

Result: The two-week period passed with no SLO breaches. There were three incidents affecting my services; the covering engineers resolved two and escalated one to me during an agreed window — response times remained within our SLA (median time to acknowledge < 10 minutes). The team appreciated the detailed runbook and automation; after returning I ran a quick retro and incorporated a checklist for future temporary coverage. I learned that early, concrete communication plus practical mitigations (runbooks, automation, clear escalation) preserves reliability while respecting personal boundaries.

MediumTechnical

63 practiced

Case study: During a global outage you must coordinate five teams across regions. Describe the communication channels, incident roles (IC, scribe, liaisons), structure of incident calls, and techniques to reduce communication overhead while keeping stakeholders informed.

Sample Answer

Situation: A global outage impacts a critical customer-facing service; five regional teams (NA, EU, APAC, Infra, and DB) must coordinate to restore service quickly.

Approach: Establish a clear incident communications plan, assign roles, run time-boxed calls, and minimize overhead while keeping executives informed.

Incident roles:- Incident Commander (IC): single decision-maker for tactical priorities and mitigations; rotates only if IC becomes unavailable.- Scribe/Coordinator: documents timeline, actions, decisions, and owner assignments in a shared incident log.- Regional Liaisons (one per region/team): surface region-specific telemetry, blockers, and execute tasks.- Communications Lead (Comms): crafts external and stakeholder messages; approves with IC.- Subject Matter Experts (SMEs): on-call engineers invited as needed.

Communication channels:- Primary: dedicated incident bridge (voice/video) for core responders.- Chat: a single incident room (e.g., #inc/SEV1) for async updates, links, and quick handoffs.- Docs: shared incident timeline (Google Doc/Confluence) updated by scribe.- Stakeholder updates: email/Slack #execs or status page managed by Comms.- Escalation: PagerDuty/ops tool for paging specific SMEs.

Structure of incident calls:- Cadence: initial full bridge (15–30 min) to align; then fixed short syncs (every 10–15 min) or ad-hoc for major events. Time-box each call (max 15 min).- Agenda template: 1. Timestamp & current status (IC, 1 min) 2. New facts/alerts (Liaisons, 2–3 min) 3. Actions in flight & blockers (Owners, 5 min) 4. Decisions/next steps + owners and ETA (IC/Scribe, 3 min) 5. Communications to stakeholders (Comms, 1–2 min)- After critical mitigation, transition to recovery call cadence (30–60 min) and then to postmortem.

Techniques to reduce communication overhead:- One-source-of-truth: scribe maintains canonical timeline; everyone references it.- Use templated status updates and action item format (Owner | Action | ETA).- Triage in chat: non-critical technical discussion happens in side channels; only escalated summaries go to main bridge.- Silent observers: limit full-bridge attendees to decision-makers and active owners; others follow via chat/timeline.- Automation: scripted runbooks and playbooks for common actions to reduce verbal coordination.- Clear escalation rules and “no-surprise” policy for cross-region changes (announce 2–3 min before execution).- Post-incident: rapid postmortem with clear RCA, action owners and follow-up deadlines.

Result: This structure reduces duplicate communication, speeds decision-making, keeps stakeholders informed with concise updates, and creates an auditable timeline for post-incident learning.

HardTechnical

62 practiced

An error budget policy shows slow degradation over three weeks that does not trigger immediate alerts but is steadily consuming the budget. As the on-call lead, how would you detect this trend earlier using on-call signals and what automated or manual actions would you trigger to prevent full budget burn?

Sample Answer

Situation: The error budget shows a slow, steady consumption over three weeks that hasn’t crossed alert thresholds but will exhaust if trend continues.

Approach — detect earlier:- Instrument more sensitive trend signals in addition to hard SLO alerts: - Sliding-window burn rate (e.g., 7d and 28d) and a shorter EWMA of daily burn to surface smoothing trends. - Change-point / anomaly detection (CUSUM or simple linear regression slope) on SLO error rate. - Composite indicators: rising request latency P95/P99 + increasing retry/error counts even if abs. error rate is below threshold. - Business-impact signal: estimated customer-minutes-at-risk = current burn-rate × remaining budget.- Signal design: warn alerts (Severity: P3) when slope > threshold or EWMA exceeds baseline+Xσ; ensure these go to on-call Slack + ticket system, not full-page.

Example EWMA detector (Python):

python

# alpha chosen for sensitivity
def ewma(prev, value, alpha=0.2):
    return alpha*value + (1-alpha)*prev

Preventative actions (automated then manual):- Automated, low-risk mitigations triggered by warn signals: - Auto-scale to absorb load if errors are capacity related. - Apply rate-limits / throttles to non-critical traffic (batch jobs, analytics) using feature flags or API gateway rules. - Shift traffic away from suspect region via load-balancer or routing policy. - Canary roll back last deploys if error signature matches deployment timeframe. - Enable circuit breakers for noisy downstream services. - Auto-create a "situation ticket" with metrics snapshot and runbook link.- Manual actions (on-call lead evaluates): - Page responsible service owners if warnings persist > X hours or slope increases. - Run targeted mitigations: temporary feature kills, rolling restart, DB connection pool tuning. - Convene quick incident review (15–30 min) to agree on action and communication to product/ops. - If burn continues, initiate throttling of lower-priority customers and coordinate with product for customer notices.

Why this works:- Trend detectors catch gradual regressions before hard thresholds.- Tiered responses avoid alert fatigue: automated safe mitigations first, human escalation second.- Business-impact metric prioritizes actions that reduce customer risk per unit error budget consumed.

Key runbook items:- Define slope/EWMA thresholds, blackout windows, rollback criteria, and ownership for automated playbooks.- Regularly backtest detectors against historical incidents to tune sensitivity and false positive rates.

MediumTechnical

77 practiced

You are on-call and receive an alert showing a spike in payment failures during peak hours with high customer impact. You have 30 minutes to mitigate impact without corrupting data. Describe your triage checklist, immediate mitigation steps, what data to inspect, and safe rollback/feature-flag options.

Sample Answer

Situation: Peak-hour alert — sudden spike in payment failures with high customer impact. With 30 minutes to act, my goal is to mitigate user-visible failures quickly while preserving data integrity for safe recovery and postmortem.

Triage checklist (first 5–10 minutes)- Confirm alert fidelity: check monitoring dashboards (payment failure rate, SLO burn, pager details).- Scope blast radius: percentage of transactions failing, geographic/PI/plan segments, percent of users affected.- Identify change window: recent deploys, config changes, third‑party gateway deployments, certificate/credentials rotations.- Check upstream/downstream: payment gateway health pages, DNS, network, message queues, DB availability.- Communicate: open incident channel, notify stakeholders, set 30‑minute update cadence.

Immediate mitigation steps (next 10–15 minutes)- Fail fast and degrade gracefully: - Enable circuit breaker / rate limit to reduce repeated requests to failing gateway. - Route traffic to backup payment gateway or fallback flow if available. - Disable non-essential payment features (promo validation, non-critical fraud checks) behind feature flags to reduce failure surface.- Stop automated harmful actions: - Pause retry workers/consumers that could cause duplicate or partial writes. - Throttle client/partner retries via API gateways.- Protect data: - Switch critical payment writes to a safe queue (durable, idempotent) or put DB in controlled mode if suspect. - Take quick backups/snapshots of affected DB tables before any rollback that touches data schema.

What data to inspect (concurrent)- Metrics: per-minute failure rate, HTTP status codes (4xx vs 5xx), latency percentiles, retries count.- Logs/Traces: recent transaction traces (request ID, correlation IDs), stack traces, gateway error payloads.- Third‑party health: gateway response bodies, error codes (insufficient funds vs auth errors vs rate limit), provider incident pages.- Queues/Workers: depth, processing rate, error queues, poison messages.- Database: pending transactions, open transactions, idempotency-key collisions, ledger consistency checks.- Config/Secrets: recent config changes, rotated credentials, cert expiry.

Safe rollback / feature-flag options- Prefer non-data-migrating rollbacks: - Flip feature flags to disable the new payment flow and revert clients to the last known-good path. - If a deploy caused it, rollback the application code but keep DB writes paused until validated.- If rollback touches DB migrations, avoid automatic revert — instead: - Maintain schema compatibility and use read-only modes or shadow writes to avoid corrupting ledger. - Replay failed transactions from durable queue after root cause fixed (idempotent replay).- Use canary/unwind: shift a small % of traffic to fallback and validate end-to-end before full cutover.- Post-mitigation: re-enable retries gradually, monitor for duplication, run reconciliation job to verify ledger correctness.

Why this approach- Prioritize reducing customer impact quickly (circuit breakers, routing) while preventing any irreversible data corruption (pause retries, snapshots, idempotent queues). Communication and controlled, reversible changes (feature flags, canary traffic) ensure safe recovery and clear post-incident analysis.

Unlock Full Question Bank

Get access to hundreds of On Call and Work Availability interview questions and detailed answers.

Join thousands of developers preparing for their dream job.