Covers the end to end practices and trade offs involved in releasing, running, and operating software in production environments. Topics include deployment strategies such as blue green deployment, canary releases, and rolling updates, and how each approach affects reliability, rollback complexity, recovery time, and release velocity. Includes feature flagging and release gating to separate deployment from feature exposure. Addresses continuous integration and continuous deployment pipeline design, automated testing and validation in pipelines, artifact management, environment promotion, and release automation. Covers infrastructure as code and environment provisioning, containerization fundamentals including container images and runtimes, container registries, and orchestration fundamentals such as scheduling, health checks, autoscaling, service discovery, and the role of Kubernetes for scheduling and orchestration. Discusses database migration patterns for large data sets, strategies for online schema changes, and safe rollback techniques.
Explores monitoring and observability including metrics, logs, and traces, distributed tracing and error tracking, performance monitoring, instrumentation strategies, and how to design systems for effective troubleshooting. Includes alerting strategy and runbook design, on call and incident response processes, postmortem practice, and how to set meaningful service level objectives and service level indicators to balance reliability and velocity. Covers scalability and high availability patterns, multi region deployment trade offs, cost versus reliability considerations, operational complexity versus operational velocity trade offs, security and compliance concerns in production, and debugging and troubleshooting practices for distributed systems with partial information. Candidates should be able to justify trade offs, explain when a simple deployment model is preferable to a more complex architecture, and give concrete examples of operational choices and their impact.
HardTechnical
46 practiced
Design an instrumentation plan for a newly onboarded critical service so it is triageable in production from day one. Specify which metrics (business and system), logs and structure, trace spans, labeling/tagging strategies for high-cardinality identifiers, SLI calculation points, and how to validate instrumentation correctness in staging before production rollout.
Sample Answer
Situation: You're onboarding a new critical service and need full triageability from day one.Plan (high level):- Goal: enable fast detection, triage, and RCA with low alert noise and privacy-safe identifiers.Metrics (collect at 1s–1m resolution):- Business: request rate (RPS) by endpoint, success rate (200s vs error classes), revenue-impacting transactions, queue depth, items processed.- System: CPU%, memory%, disk I/O, network I/O, thread pool saturation, GC pause time, file descriptors, connection pool usage.- App infra: request latency histogram (p50/p95/p99/99.9), error counts by type, retries, circuit-breaker state, downstream latency.Logs and structure:- Structured JSON logs with fields: timestamp, service, env, hostname, pod, request_id, user_id_hash (hashed), tenant_id_tag, level, msg, component, span_ids, error_code, stack.- Log sampling: full logs for errors; 1% for INFO high-volume endpoints; ensure redaction for PII.Tracing:- Instrument entry span (HTTP/gRPC) and key internal spans: auth, validation, DB calls, cache, downstream HTTP, background jobs.- Capture attributes: request_id, route, user_tenant_hash, db_statement_summary, cache_hit(bool), retry_count.- Set span durations and status codes. Ensure 100% sampling for error traces, 1-5% for normal.Labeling/tagging high-cardinality:- Never tag metrics with raw user_id, email, or order_id. Use low-cardinality labels: service, env, region, endpoint, error_code.- For traces/logs include hashed/salted identifiers and separate lookup service accessible to SREs for authorized investigations.- For business rollups use tenant_id (if small cardinality) or bucketed segments (top-100 tenants + "others").SLIs and SLOs:- Primary SLI: successful request rate (>=200 and not user-cancelled) measured at ingress after retries; secondary SLI: p99 latency at ingress; availability = (successful requests)/(total requests) over rolling 28 days.- SLI calculation points: compute on ingress gateway for whole request path; for downstream dependencies compute separate SLIs (db/cache) to attribute faults.Validation in staging:- Unit tests for instrumentation (metrics emitted on events), integration tests with test metrics backend (assert labels, types).- Load tests to verify histograms, cardinality, and alert thresholds; synthetic error injection to confirm error traces sampled and alerts trigger.- End-to-end trace validation: generate synthetic transaction and follow trace across services; check span names, tag presence, and propagation of request_id.- Run log ingestion checks: JSON parseability, schema validation, and retention.Operational:- Alerts: page on SLO breaches (error budget burn), high p99 latency, big drop in revenue transactions; use tiered alerts with runbook links.- Dashboards: overview (traffic, errors, latency), dependency maps (traces), per-endpoint drilldowns.- Rollout: feature-flagged instrumentation, flag off sampling increases; monitor cardinality and cost first 48 hours, adjust.This plan balances observability, privacy, and cost while enabling immediate triage and measurable SLOs.
MediumTechnical
45 practiced
Describe strategies for performing schema migrations on very large tables (hundreds of GB to TB) with minimal downtime: expand-contract pattern, online schema change tools (gh-ost, pt-online-schema-change), batched backfills, zero-downtime deploy patterns, and how to measure progress and mitigate timeouts and failures.
Sample Answer
Start with a clear plan: define acceptable downtime, SLOs, and rollback criteria. For very large tables use the expand–contract pattern and online tools so writes stay live.Expand–Contract pattern- Expand: add new nullable columns or new tables, build required indexes/materialized data incrementally (no blocking DDL). Use backfill jobs to populate new columns.- Swap/contract: switch application to read/write the new schema behind a feature flag, then drop old columns once safe.Online schema-change tools- gh-ost and pt-online-schema-change create a shadow table and replay row changes via triggers/replication to avoid long metadata locks. gh-ost uses the binary log and is safer for production; pt may be simpler but can be heavier.- Best practices: run in replica, use throttling, test on a copy of production, and validate checksums between old/new tables.Batched backfills & zero-downtime deploys- Break backfill into idempotent, small chunks (by primary key ranges or time windows); run parallel workers with a global coordinator to avoid hotspots.- Use feature flags to route traffic to new columns gradually; make reads tolerant (fallback to old column) during transition.- Use connection pooling and retry logic to tolerate transient errors.Measuring progress and mitigating failures- Track rows/sec, bytes/sec, % complete, ETA; monitor binlog/replication lag, lock waits, CPU/IO, and tail latency.- Implement throttling based on replication lag or system load; auto-pause on lag spikes or high lock/wait metrics.- Handle timeouts: exponential backoff per chunk, idempotent retries, and circuit breakers to stop work if SLOs are endangered.- Prepare rollback: feature-flag revert, stop backfills, and if needed rebuild in maintenance window. Always take schema-aware backups or point-in-time recovery snapshots before large operations.- Post-migration: run consistency checks (checksums), remove triggers, and clean up shadow artifacts.Operationalise: automate orchestration, emit metrics/alerts for progress and safety thresholds, and rehearse the procedure on staging with production-sized data.
MediumSystem Design
41 practiced
Describe how to design a release automation engine that can deploy a set of interdependent services in a defined order, handle partial failures with safe rollback or compensation, and integrate seamlessly with feature flags. Explain how you'd model dependencies, persist deployment state, and implement retry and compensating actions for transient failures.
Sample Answer
Requirements & constraints:- Deploy interdependent services in order, support partial failures with safe rollback/compensation, integrate feature flags (canary/gradual rollout), be resilient and observable.High-level design:- Controller service that orchestrates a release plan. Worker executors run deployment steps (K8s helm, terraform, scripts). Persistent store holds release state and history.Modeling dependencies:- Represent services as nodes in a directed acyclic graph (DAG). Edges = “A must be healthy before B”. Build deployment plan via topological sort; allow parallelism for independent subgraphs and priority/weighting for ordering.- Each node contains: service id, target version, pre-checks, post-checks, rollback/compensation action, feature-flag toggles, timeout/policy.Persisting deployment state:- Use a transactional durable store (Postgres or etcd + write-ahead log). Schemas: - releases: id, status (PENDING, IN_PROGRESS, PARTIAL_FAILED, ROLLBACK, SUCCESS), created_by, timestamps - steps: release_id, node_id, status, attempts, last_error, start_time, end_time - events/logs: deterministic audit trail for replay/rollback- State machine per step persisted so restart/resume is safe. Store idempotency tokens for operations.Retry and transient failure handling:- Steps are idempotent. Implement retry policy per step: exponential backoff + jitter, max attempts. Use circuit-breaker for flapping services.- On transient failures: retry automatically; emit metrics and alerts if thresholds exceeded.- If step exceeds retries or a non-transient failure occurs, mark release PARTIAL_FAILED and trigger compensation according to strategy.Compensating actions and rollback:- Two classes: automatic rollback and compensating action. - Rollback: revert to previous version using stored previous artifact reference; only if safe (no destructive DB migration). - Compensating action: domain-specific undo (e.g., reverse config change, disable feature flag, run cleanup job).- Define rollback policy per node (NONE, AUTOMATIC, MANUAL). Rollback plan is generated from logs and DAG: revert nodes in reverse topological order.- Use transactional migration patterns: prefer backward-compatible DB changes, use feature flags to decouple deploy and enablement so rollback is lighter (toggle flag off).Feature-flag integration:- Orchestrator integrates with flag service (LaunchDarkly, Unleash) via API.- Steps include explicit flag actions: create toggle, set initial state (off), perform deploy, progressively enable (canary percent), monitor metrics, continue or rollback.- Support progressive rollout strategies: percentage-based ramp, user-segmented, or rollout by region/cluster.Safety, observability, and testing:- Pre/post health checks: smoke tests, readiness/liveness, SLO checks, business metrics (error rate, latency).- Metrics/alerts for attempts, durations, rollback frequency. Tracing for causal debugging.- Dry-run and simulation mode using persisted plan and validations.- Chaos-ready: test compensations in staging and simulate partial failures.Trade-offs:- Using DAG + persisted state favors correctness and resumeability; more complex than linear scripts.- Storing full rollback artifacts increases storage but improves safety.- Feature flags add operational complexity but greatly reduce rollback blast radius.This design gives safe, resumable deployments with clear rollback/compensation semantics and tight feature-flag control to minimize user impact.
MediumTechnical
58 practiced
Explain the expand-contract (forward/backward compatibility) pattern for online schema changes. Provide a step-by-step plan (with commands or pseudocode) for adding a column used by new code: add nullable column, backfill, switch readers/writers, make column non-null, and cleanup. Discuss ORM compatibility, triggers, and foreign-key implications.
Sample Answer
Approach summary: use the expand-contract (forward/backward compatibility) pattern: first “expand” the schema in a way old and new code both tolerate (nullable/new columns), perform safe backfill and dual-write, switch readers to new behavior, then “contract” by making the column NOT NULL and finally cleanup. Use online-schema-change tools (gh-ost / pt-online-schema-change) or safe chunked updates to avoid long locks.Step-by-step plan (MySQL-flavored; adapt for Postgres):1) Add nullable column (fast, metadata-only in many engines)
ALTER TABLE users ADD COLUMN new_flag TINYINT NULL;
Use gh-ost/pt-online-schema-change if ALTER is blocking:
2) Backfill in chunks (avoid long transactions / replication lag)
-- pseudocode
WHILE True:
rows = SELECT id FROM users WHERE new_flag IS NULL LIMIT 10000;
if rows.empty: break
UPDATE users SET new_flag = <derived_value> WHERE id IN (rows);
Or use an UPDATE ... WHERE new_flag IS NULL LIMIT ... loop. Monitor replication lag, I/O.3) Dual-write / dual-read feature rollout - Deploy new service version that writes new_flag (still reads old behavior if absent). - Keep old instances writing too until all traffic moves. - Optionally add a short-lived trigger or application-side middleware to set new_flag for writes missed during backfill:
CREATE TRIGGER users_before_update BEFORE UPDATE ON users
FOR EACH ROW
SET NEW.new_flag = COALESCE(NEW.new_flag, compute_value(NEW.*));
- Prefer application dual-write over DB triggers if possible (easier to reason & test). If using triggers, ensure idempotence and performance.4) Switch readers to prefer new column - Flip feature flag / deploy code that reads new_flag; keep fallback to old computation if NULL. - Monitor errors, latency, correctness.5) Make column NOT NULL (contract) - Ensure all rows have non-null values. - For Postgres: add constraint NOT NULL or use VALIDATE CONSTRAINT pattern.
ALTER TABLE users ALTER COLUMN new_flag SET NOT NULL;
- For MySQL large table: use gh-ost/pt to alter or set a DEFAULT then alter in online-safe way:
ALTER TABLE users MODIFY new_flag TINYINT NOT NULL DEFAULT 0;
6) Cleanup - Remove fallback code, old computed columns/logic. - Drop triggers if added. - Drop indexes or columns no longer needed using online tools.ORM compatibility notes:- ORMs may auto-send NULLs/omit columns — verify generated SQL. Add migration files so ORM schema matches DB.- Some ORMs infer defaults client-side; prefer DB-level defaults for safety.- Ensure models accept nullable field until migration completes; update model field to non-null in a later deploy after DB constraint.Triggers:- Useful to capture writes missed during backfill, but add complexity: hidden behavior, harder to test, and can increase row write cost.- Prefer application dual-write or idempotent background jobs. If using triggers, measure CPU and replication impact.Foreign-key implications:- Adding a FK column that references another table can be expensive: - In Postgres, can ADD CONSTRAINT NOT VALID then VALIDATE CONSTRAINT later to avoid long locks:
ALTER TABLE child ADD COLUMN parent_id BIGINT NULL;
ALTER TABLE child ADD CONSTRAINT fk_parent FOREIGN KEY (parent_id) REFERENCES parent(id) NOT VALID;
-- backfill parent_id...
ALTER TABLE child VALIDATE CONSTRAINT fk_parent;
- In MySQL, adding FK on large tables may require a table rebuild — use online change tools or create FK during a maintenance window.- Ensure referenced rows exist before validating FK; backfill must set correct referents.Monitoring & safety:- Monitor replication lag, CPU, I/O, and errors during backfill.- Use feature flags and gradual traffic shifting.- Have rollback plan: stop writing new column and keep fallback behavior until resolved.This pattern ensures zero-downtime, reversible schema changes while keeping both old and new code compatible during rollout.
HardTechnical
56 practiced
You are given partial logs and traces: service A’s request latency to service B has increased, but B’s outbound calls and metrics look normal. Describe a systematic debugging approach with partial information: how you'd prioritize signals, formulate hypotheses, add temporary instrumentation or sampling, and pursue mitigations that avoid making the outage worse.
Sample Answer
Clarify scope and goals quickly- Confirm user impact (which clients, regions, p50/p95/p99 increase), timeframe, and whether this is production impacting SLOs. Ask if change control/rollouts happened.Prioritize signals1. Client-side telemetry from A: latency histogram, retries, error rates, connection setup times.2. Network layer: TCP retransmits, TLS handshakes, SYN/ACK timing between A→B.3. B’s ingress metrics (request rate, queue length, accept latency) and host-level (CPU, runqueue, cgroups).4. Cross-service traces: span durations at A and B, including DB/ downstream calls.5. Infra (LB, sidecar/proxy, Kubernetes CNI) logs.Formulate hypotheses (ordered least-invasive → more invasive)- Hypothesis 1: Network/transit (packet loss, routing, MTU) causing retransmits — explains A-only latency.- Hypothesis 2: Client-side issue in A (threadpool exhaustion, bad SDK, blocking GC).- Hypothesis 3: Intermediate proxy/sidecar or LB misconfiguration adding latency.- Hypothesis 4: Rare B behavior on specific request shapes (cold caches) not visible in aggregate B metrics.Temporary instrumentation & sampling (low-risk)- Increase trace sampling rate for A→B for affected timeframe/clients (not global) and capture full spans including TCP/TLS timings.- Add lightweight metrics on A: thread pool queue length, blocking GC pauses, socket backlog, socket RTT (tcp_info via getsockopt).- Enable eBPF/tcpdump sampling on a subset of A instances to measure retransmits and RTT (sample 1% traffic).- Add histogram logging for connect, DNS, TLS, first-byte, and last-byte latencies at A and sidecars.- Tag traces with route/feature flags to find request-shape correlation.Pursue mitigations that avoid worsening outage- Apply read-only or reduced feature mode for affected clients if possible.- Gradually shift traffic away (canary/weighted) from suspect A instances or a particular AZ while monitoring SLOs.- Avoid global restarts; instead restart one pod/instance at a time if investigating client-side resource exhaustion.- If network packet loss suspected, isolate by moving a small traffic slice to a different path or node to test impact.- Implement circuit-breaker or local retry with jitter on A to prevent head-of-line blocking; keep retry budget conservative.Investigation workflow & decision points- Start with quick wins: compare client-side vs B ingress histograms; if B ingress shows no increased service time but A shows long tcp connect or TLS, focus on network/client.- If traces show long span before reaching B (e.g., in proxy), instrument that component next.- Use sampling and small-scale traffic shifts to validate hypotheses; only scale changes after verifying improvements on samples.- Record every action, rollback plan, and guard rails (alerts if error rate rises).Post-incident- Capture minimal permanent telemetry discovered useful (connect/TCP histograms, threadpool metrics).- Run RCA with timeline, root cause, mitigations, and SLO/engineering follow-ups (e.g., better tracing fidelity, synthetic clients, improved circuit-breakers).
Unlock Full Question Bank
Get access to hundreds of Production Deployments and Operations interview questions and detailed answers.
