Technical Depth Verification Questions
Tests genuine mastery in one or two technical domains claimed by the candidate. Involves deep dives into real world problems the candidate has worked on, the tradeoffs they encountered, architecture and implementation choices, performance and scalability considerations, debugging and failure modes, and lessons learned. The goal is to verify that claimed expertise is substantive rather than superficial by asking follow up questions about specific decisions, alternatives considered, and measurable outcomes.
MediumTechnical
65 practiced
You need to verify that a StatefulSet on Kubernetes fails over correctly when the leader pod is killed. Describe how you would implement automated leader-failover tests including readiness/liveness probes, pod disruption budgets, anti-affinity rules, and steps to measure failover correctness and time-to-restore. Include safe teardown steps to avoid production impact.
Sample Answer
Situation: We're validating that a StatefulSet-backed service elects a new leader and resumes traffic when the current leader pod is killed. The test must be automated, non-destructive to prod, and measure correctness and time-to-restore.Approach / test plan:1. Test environment: run tests against a staging/canary cluster with same config as prod. Use a separate namespace and test-specific labels to avoid affecting production.2. Preconditions: ensure StatefulSet has: - Readiness probe that reflects application-level readiness (e.g., RPC health endpoint that returns leader/non-leader readiness). Liveness probe to detect crash-loop and allow kubelet restart. - PodDisruptionBudget (PDB) set so planned evictions don't reduce availability below acceptable threshold. - Pod anti-affinity rules (preferred or required depending on capacity) to spread pods across nodes to avoid single-node failures.3. Automation steps: - Deploy StatefulSet test instance with leader-election enabled (same algorithm as prod). - Start synthetic traffic / health-checker client that continuously queries the application and records successful requests and which pod served them (via response header or metrics). - Assert which pod is leader via /metrics or leader-status endpoint; record leader pod name and timestamp. - Kill the leader pod: issue kubectl delete pod <leader-pod> --grace-period=0 --force (or use API to simulate crash). Capture timestamp t0. - Continue synthetic traffic; detect when responses resume from the new leader and when readiness endpoint reports leader-ready. Capture timestamp t1 (first successful leader-acknowledged request) and t2 (pod becomes Ready per Kubernetes).4. Measurements & correctness checks: - Time-to-detect: time between t0 and kubelet/k8s marking pod NotReady / Terminated. - Time-to-elect: time between t0 and the new leader log/metrics change. - Time-to-restore (service-level): time between t0 and first successful request served by new leader (t1). - Correctness: verify exactly one leader at any time (query all pods' leader-status). Ensure no split-brain: only one pod reports leader=true. Verify no request loss beyond SLO thresholds and no violation of PDB constraints. - Repeat under various conditions: node drain, network partition (using tc/netem), high CPU load.5. Safety & teardown: - Run only in non-prod namespace; use ResourceQuota and admission labels to guard. - After test, restore any modified PDB/affinity to original values via git-ops; delete test namespace with kubectl delete namespace <test-ns> and verify no leftover PVCs or Services. - If running against a shared cluster, throttle test frequency and coordinate with on-call via alerts suppression windows.6. Observability & CI integration: - Emit metrics (Prometheus) for t0,t1,t2 and test outcome; store logs and traces for the leader election window. - Fail CI job if time-to-restore > SLO or if multiple leaders detected.Key reasoning: probes must reflect application semantics so Kubernetes routing aligns with app readiness; PDB and anti-affinity protect availability but tests must ensure they don't mask real failover issues; measuring both k8s readiness transitions and application-level recovery gives meaningful SLO-aligned metrics. This approach automates repeatable, safe, and measurable leader-failover validation.
HardSystem Design
123 practiced
Given a microservice architecture currently using synchronous RPC, propose a migration plan to asynchronous, event-driven messaging to improve resilience. Cover idempotency, message ordering, schema evolution, exactly-once vs at-least-once tradeoffs, dead-letter handling, and a rollout strategy that minimizes customer impact.
Sample Answer
Requirements & constraints:- Maintain current SLAs while introducing async events to improve resilience and scalability.- Minimize customer-visible regressions; allow gradual rollback.- Support schema evolution, ordering where needed, and robust failure handling.High-level migration approach:1. Introduce a durable message backbone (e.g., Kafka for ordered topics; or a cloud pub/sub + FIFO/ordering keys). Deploy as a sidecar/bridge so RPC paths remain functional.2. Implement event adapters: producers publish events in addition to responding to RPC (dual-write pattern) — initially synchronous RPC still authoritative.3. Incrementally convert consumers to subscribe to events; once consumed and validated, flip producer to event-first and phase out RPC.Idempotency:- Require each event carry a globally unique event_id and source_id + version.- Consumers maintain per-entity dedup store (compactable): Redis with TTL, or Kafka compacted topic of processed event_ids.- Design handlers to be idempotent: use upserts, compare-and-set, or store last-processed-sequence per aggregate.Message ordering:- Use partitioning keys (user_id, aggregate_id) to guarantee per-key ordering on Kafka or FIFO queues.- For cross-aggregate ops requiring global order, keep RPC or implement causal sequencing with vector clocks or orchestration service to avoid blocking hot partitions.Schema evolution:- Use schema registry (Avro/Protobuf/JSON Schema) and backward/forward compatible schemas.- Consumers should be tolerant (defaults for new fields); producers version events (event_type, schema_version).- When incompatible change needed, use new topic or event_type and orchestrate parallel consumers during migration.Exactly-once vs at-least-once:- Default to at-least-once (simpler, higher throughput). Compensate with idempotent handlers.- Use transactional producers + Kafka exactly-once semantics only where necessary (high-value financial flows), aware of complexity and operational cost.Dead-letter and retries:- Implement exponential backoff retry queues and a dead-letter topic/queue with metadata (error, attempts).- Build tooling to inspect, reprocess, and replay DLQ messages after fixes; track DLQ metrics and alert.Rollout strategy (minimize customer impact):- Canary per service: route small % of traffic/events to new consumer; compare outcomes to RPC baseline.- Dual-write then dual-read: producers publish events while still serving RPC; consumers read both RPC and events during verification.- Feature flags to quickly turn off event publishing.- Automated end-to-end tests and contract testing; run load tests to validate capacity.- Monitor SLOs, consumer lag, DLQ rates, and business KPIs. Gradually increase traffic slices and deprecate RPC only after stability windows.Operational considerations:- Capacity planning for broker (retention, partitions), monitoring (consumer lag, throughput), alerting thresholds.- Runbooks for broker failure, consumer backpressure, and DLQ handling.- Post-mortems and iterative improvements.This plan balances safety (dual-write/read, canaries), reliability (idempotency, retries, DLQ), and operational control (schema registry, monitoring) to migrate synchronously-coupled services to resilient event-driven architecture.
HardTechnical
79 practiced
A cloud provider region experienced a total network partition and many services experienced split-brain writes. Draft a recovery playbook: diagnostics to identify divergent replicas, strategies for data reconciliation (last-writer-wins, changelog merge, manual reconciliation), safe resume order for services, verification steps, and customer communication considerations.
Sample Answer
Situation: A full region network partition caused split‑brain writes across many services; we need a safe, auditable recovery to reconcile data, restore services, and communicate to customers.Playbook — High level steps1) Triage & containment (0–30m)- Freeze writes where safe: enable global write‑halt flags / traffic‑shaping for affected region(s).- Activate incident channel, incident commander (IC), SRE leads, data owners, Legal/Comms.2) Diagnostics — identify divergent replicas- Inventory services/datastores impacted (from topology + BGP/SDN logs).- For each datastore: collect replica state metadata (last-applied-index, vector clocks, LSNs, timestamps), checksums/hashes of recent segments, and changelogs.- Use cross-region health checks and operation logs to detect writes that occurred during partition; produce a per-shard divergence report listing conflicting keys/objects and metadata (timestamps, origin region, operation id).- Prioritize by blast radius: control plane, auth, billing, payment, customer data.3) Reconciliation strategies (per data-type)- Last-writer-wins (LWW): acceptable for idempotent state or where timestamp authority exists. Use only if clocks trusted and data loss risk acceptable.- Changelog / CRDT-merge: for append-only logs, queues, or CRDT-backed stores – perform deterministic merges from operation logs, replaying in causally consistent order.- Application-aware merge (recommended for complex entities): call service owners to supply merge rules (e.g., for user profiles, merge fields with conflict resolution policies).- Manual reconciliation: for financial/billing/transactions or other high-integrity data, extract conflict sets and route to SMEs for manual review with audit trail.- Always stage merges in a sandboxed environment and run consistency checks.4) Safe resume order (dependencies first)- Identity/auth and control plane (so access & coordination work)- Storage systems: metadata services (catalog, config), then primary key‑value stores- Messaging/brokers (to resume event flows)- Business-critical services (payments, billing) — only after reconciliation and verification- Frontends and secondary services last- Resume in read-only mode, enable writes progressively with canary traffic and autoscaling.5) Verification & validation- Automated checks: checksum comparison, read-after-write for sample keys, end-to-end test suites, SLO probes.- Invariant checks: no negative balances, referential integrity, schema validations.- Audit logs preserved and immutable; produce reconciliation reports.- Rollback plan: snapshot/backup checkpoint before each major merge, ability to revert to pre-merge state.6) Communication- Internal: concise incident timeline, recovery plan, roles/responsibilities, and decision log.- External/customers: initial acknowledgement, affected services, ETA for next update; regular updates (e.g., hourly) with clear scope and mitigation steps, and post-incident RCA and remediation plan.- Regulatory: notify compliance/legal for critical data exposures or transactional inconsistencies.Lessons & safeguards post-recovery- Harden split‑brain prevention: stronger quorum policies, anti‑entropy, automated fencing, regional write throttles.- Improve observability: divergence detectors, per-key vector clocks, automated reconciliation tooling, runbooks tested in chaos exercises.This playbook emphasizes safety (freeze → diagnose → reconcile in sandboxes → validate → resume progressively) with clear owner responsibilities and customer transparency.
EasyTechnical
75 practiced
Compare eventual consistency and strong consistency with concrete examples relevant to SRE: DNS updates, distributed cache invalidation, and user profile updates. For each example discuss operational tradeoffs in latency, availability, complexity of rollbacks, and how you would test for correctness under network partitions.
Sample Answer
Eventual vs strong consistency — short: strong consistency guarantees reads reflect the latest write (linearizability), while eventual consistency allows stale reads but promises convergence without further writes. For SREs the tradeoffs show in latency, availability, rollback complexity, and partition testing.1) DNS updates- Strong: synchronous global propagation (e.g., service updates requiring immediate routing) gives correct reads but requires coordination and higher latency/possible unavailability during propagation.- Eventual: standard DNS TTL-based propagation is low-latency on writes and highly available, but clients see stale records until TTL expires.- Rollbacks: eventual easier (just reissue record and wait TTL); strong requires careful coordination to avoid split-brain.- Test under partitions: simulate region partition and varying resolver caches; verify clients hit old vs new IPs, measure time-to-convergence.2) Distributed cache invalidation- Strong: use synchronous invalidation or write-through with consensus so reads reflect updates — higher write latency and potential unavailability if cache nodes or consensus fail.- Eventual: asynchronous invalidation/pub-sub (invalidate message) gives fast writes and availability but stale reads window.- Rollbacks: eventual harder if competing updates occur; strong rollbacks require coordinated invalidation and possible cache warmup.- Test: partition cache vs app nodes, inject writes, then assert read-after-write behavior and convergence times; include message-loss scenarios.3) User profile updates- Strong: single-master or consensus-writes ensure immediate correctness (critical for billing/permissions) — higher latency and reduced availability on master failure.- Eventual: multi-region replicated stores with async replication improve availability and read latency, but users may see stale profiles.- Rollbacks: eventual needs conflict resolution (last-write-wins or CRDTs); strong rollbacks require transactional compensation.- Test: create concurrent conflicting updates while partitioning regions; verify conflict resolution, monotonic reads where required, and SLOs for staleness.Operational guidance: pick strong for correctness-critical paths (auth, billing), eventual for high-read, geo-distributed features (profile display, DNS caching). Instrument metrics: staleness window, convergence time, error rates; run chaos tests (network partition, packet loss) and assert invariants (no split-brain for strong; bounded staleness for eventual).
MediumTechnical
69 practiced
Describe a CI/CD deployment pipeline that minimizes downtime for a microservice: include steps for build, automated tests, canary promotion, health gating, automated rollback, and how you validate the rollback path in advance. What specific metrics and gates would you use to allow or abort a rollout?
Sample Answer
Situation: As an SRE designing a CI/CD pipeline for a microservice, the goal is zero-to-minimal downtime with safe canary promotion and fast automated rollback.Pipeline steps (end-to-end):1. Build: CI (e.g., GitHub Actions/Jenkins) builds container image, runs static analysis, signs artifact, pushes to registry.2. Automated tests: unit → integration → contract tests → e2e smoke tests in ephemeral environment. All must pass before deploy.3. Canary deployment: CD (ArgoCD/Spinnaker/Flux) deploys canary to a small subset (e.g., 1–5% of traffic or 1 of N pods) with traffic routing via service mesh/ingress.4. Health gating & metrics collection: observe canary for a warm-up window (e.g., 5–15 minutes) collecting metrics: error rate, success rate, p95/p99 latency, request throughput, CPU/memory, saturation, and business KPIs (e.g., checkout conversion).5. Promotion: If gates pass, incrementally increase canary (5% → 25% → 50% → 100%) with waits and reevaluation at each step.6. Automated rollback: If any gate breaches, orchestrator triggers rollback to previous stable version and re-routes traffic immediately.Health gates & thresholds (examples):- HTTP error rate (5xx) for canary must be <= baseline + 0.5% absolute OR <= 2x baseline for small baselines.- Success rate >= 99.5% for critical flows.- p95 latency must not increase > 30% vs baseline and p99 not > 50%.- CPU/memory per-pod > no more than 20% above baseline.- Business KPIs: conversion drop < 1% absolute.- Alert if logs show new exception class count > 5 occurrences/min or new unique error fingerprint.Why these metrics:- Error rate & success rate detect correctness regressions.- Latency captures performance regressions affecting UX.- Resource metrics catch scaling regressions and OOMs.- Business KPIs guard user impact.Automated rollback mechanics:- Rollback is an automated pipeline action that redeploys the previous image, rebalances traffic, and marks the failed release. It also creates an incident with diagnostics and links to related traces/logs.- Use circuit-breakers and traffic-shaping in the mesh to minimize blast radius while rollback completes.Validating rollback path in advance:- Rehearsals: run automated “rollback drills” in staging and scheduled canary rehearsals in production on low-traffic windows to verify rollback scripts work and latency of switching is acceptable.- Chaos tests: periodically inject failures (fault injection, terminate canary pods, network latency) and assert rollback triggers correctly.- Backwards-compatible DB migrations: prefer expand-then-migrate pattern; validate backwards path by deploying older version against migrated schema in staging.- Feature flags: decouple code deploy from feature activation so you can rapidly toggle behavior without redeploying.- Canary health test suite: include synthetic transactions and tracing checks that validate both promotion and rollback behavior.Observability + automation:- Centralize metrics in Prometheus/Grafana, traces in Jaeger, logs in ELK, and set alerting rules in Alertmanager/SLO-based alerts.- Use a policy engine (e.g., Kayenta, Flagger) to evaluate gates automatically and trigger rollout/rollback actions.- Keep human-in-the-loop for high-impact releases: optional manual approve at 50% step or when business KPI gates are used.Post-failure:- Capture diagnostics (traces, flamegraphs, logs, heap dumps) automatically on rollback.- Create a postmortem runbook: analyze root cause, adjust tests/gates, and add regression tests to CI.This approach minimizes downtime by small incremental exposure, automated health gating, rapid rollback, and pre-validated rollback procedures so rollbacks are reliable and fast.
Unlock Full Question Bank
Get access to hundreds of Technical Depth Verification interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.