Covers the end to end practices and trade offs involved in releasing, running, and operating software in production environments. Topics include deployment strategies such as blue green deployment, canary releases, and rolling updates, and how each approach affects reliability, rollback complexity, recovery time, and release velocity. Includes feature flagging and release gating to separate deployment from feature exposure. Addresses continuous integration and continuous deployment pipeline design, automated testing and validation in pipelines, artifact management, environment promotion, and release automation. Covers infrastructure as code and environment provisioning, containerization fundamentals including container images and runtimes, container registries, and orchestration fundamentals such as scheduling, health checks, autoscaling, service discovery, and the role of Kubernetes for scheduling and orchestration. Discusses database migration patterns for large data sets, strategies for online schema changes, and safe rollback techniques.
Explores monitoring and observability including metrics, logs, and traces, distributed tracing and error tracking, performance monitoring, instrumentation strategies, and how to design systems for effective troubleshooting. Includes alerting strategy and runbook design, on call and incident response processes, postmortem practice, and how to set meaningful service level objectives and service level indicators to balance reliability and velocity. Covers scalability and high availability patterns, multi region deployment trade offs, cost versus reliability considerations, operational complexity versus operational velocity trade offs, security and compliance concerns in production, and debugging and troubleshooting practices for distributed systems with partial information. Candidates should be able to justify trade offs, explain when a simple deployment model is preferable to a more complex architecture, and give concrete examples of operational choices and their impact.
MediumTechnical
47 practiced
Case study: after adopting CI/CD pipelines the company increased deployments from weekly to daily, but incident and rollback rates rose. Analyze likely root causes across people, process, and technology. Propose a prioritized remediation plan with measurable KPIs (e.g., MTTR, rollback rate, change failure rate) to restore reliability while keeping improved velocity.
Sample Answer
Analysis — likely root causes (people / process / technology)- People: teams may lack deployment discipline, insufficient testing ownership, or inadequate training on new pipelines. Pressure to ship daily can cause shortcuts.- Process: missing gating (code review SLAs, release checklist), poor change batching, no feature flag strategy, weak rollback/runbook procedures, and immature post-incident learning.- Technology: pipelines may run insufficient integration/e2e tests, no canary/blue-green deployments, limited observability (poor SLOs/alerts), slow or brittle rollback mechanisms.Prioritized remediation plan (with timeline & KPIs)1) Immediate (0–2 weeks) — Quick wins- Enable feature flags for risky changes; require toggles for user-facing features.- Add mandatory simple pre-merge checks: lint, unit tests, basic security scan.KPIs: rollback rate ↓ by 20% in 4 weeks; change failure rate measured baseline.2) Short term (2–8 weeks) — Safety & visibility- Implement canary or blue-green deployments for production releases.- Improve observability: instrument SLOs, error rates, latency, and dashboards; set automated alerts.- Define MTTR runbooks and run one tabletop drill/week.KPIs: MTTR target < 60 minutes within 8 weeks; detection-to-fix time reduced 30%; % of deploys using canary = 80%.3) Medium term (8–16 weeks) — Pipeline hardening- Add integration and smoke tests in pipeline; shift-left testing in CI.- Enforce trunk-based development, small PR size limits, and code review SLAs.- Automate rollback and health checks; add deployment gating on SLO breaches.KPIs: change failure rate target < 5% (industry benchmark), rollback rate < 3% per deploy, lead time for changes stable or improved.4) Long term (3–6 months) — Culture & continuous improvement- Instituting blameless postmortems, RCA tracking, and a reliability backlog with SRE/rotations.- Training on testing, observability, and feature-flag best practices.KPIs: number of postmortems with action items closed within 30 days = 90%; developer confidence surveys improved; deployment frequency retained (daily) while reliability KPIs met.Trade-offs and governance- Balance speed vs safety by gating high-risk services more strictly.- Track both velocity and reliability metrics together (deploys/day, lead time, change failure rate, MTTR) in a single dashboard to avoid local optimizations.Expected outcomeWithin 3 months expect rollback and change-failure rates to fall substantially while preserving daily deployments; MTTR and detection times will improve via better observability and runbooks, enabling sustained velocity with reduced incidents.
HardSystem Design
49 practiced
Design a CI pipeline that validates database migrations by running them against a reproducible test environment seeded with synthetic data representative of production (schema shapes, distribution, and hotspots). Describe how you would generate representative synthetic datasets safely, run migrations at scale in CI, and detect performance regressions or data correctness issues automatically.
Sample Answer
Requirements & constraints:- Reproducible, automated CI runs that exercise real migration code against data shaped like prod: schema, value distributions, hotspots, foreign keys, null patterns.- Safety: no PII leakage.- Detect correctness (data integrity) and performance regressions (latency, IO, plan changes).High-level pipeline (CI job per PR / nightly job for full scale):1. Provision reproducible environment - Use immutable images (Docker) + IaC (Terraform/CloudFormation) to spin ephemeral DB instances (same engine/version as prod) and app container. - Record exact DB image, migration branch, and RNG seed for reproducibility.2. Generate representative synthetic dataset - Profile production metadata (schema, histograms, column cardinalities, foreign-key graph, indexes, common predicates, heavy queries). Only collect metadata/stats, not raw values. Use anonymized telemetry: pg_stats, EXPLAIN samples, slow-query logs. - Create a statistical model per column: distribution (categorical, gaussian, zipfian), null-rate, length distribution. Identify hotspots (high-write partitions, hot rows) and temporal patterns. - Synthesize data with a seeded generator (e.g., Python scripts using Faker + numpy/scipy + custom zipfian samplers; or open-source tools like Synthpop, Mockaroo, or in-house generator). Respect constraints: FKs, uniqueness, constraints, check constraints. - Enforce safety: never copy PII; if sampling necessary, apply irreversible hashing + tokenization or differential privacy techniques for aggregates. - Output: deterministic dump or load script parameterized by RNG seed and scale factor.3. Load data at scale - Use parallel loading (COPY, multi-threaded loaders like pg_bulkload or loader jobs on k8s) with transactional boundaries to mimic production insert patterns (batch sizes, concurrency). - Optionally run workload generator (pgbench, HammerDB, custom load) to create hot rows and timing patterns before migrations.4. Run migrations - Apply current repo migrations (current HEAD) and then target migrations in isolation. For testing rolling migrations, run forward/backward cycles (apply migration → run app queries → rollback → apply patched migration). - For long-running migrations (schema change with backfill), simulate production-like load during migration to exercise lock/contention behavior.5. Automated correctness checks - Schema-level: validate constraints, FK integrity, unique constraints. - Row-level sampling: deterministic checksums (e.g., per-table hash partitions) before/after migrations where applicable. - Business logic tests: run application-level integration tests and critical queries; verify results vs expected golden queries on small subsets. - Data drift checks: compare distributions (histogram buckets) pre/post migration to detect corruption.6. Performance/regression detection - Baseline metrics: maintain baseline run(s) for the same synthetic dataset & query suite: wall time, p99/p95 latencies, throughput, explain plan fingerprints, index scan vs seq scan counts, CPU, IO, lock waits. - During CI run collect: query latencies, EXPLAIN ANALYZE for hotspot queries, pg_stat_statements, planner metrics, memory usage, IO, and migration duration. - Automated detectors: - Thresholds: compare key metrics to baseline (e.g., >10% p95 regression triggers failure). - Plan-diff: fingerprint plans (node types, index usage); flag unexpected plan changes. - Regression scoring: weighted score across correctness and performance; fail if score below threshold. - For non-deterministic variance, run multiple iterations and use statistical tests (e.g., t-test or bootstrap) to avoid false positives.7. Scale & parallelization in CI - Use a test matrix for scale factors (smoke small, medium, full-ish nightly). Parallelize across workers/nodes; cache synthetic dumps for repeated runs keyed by seed & scale. - Use incremental testing: quick PR-level run with small but representative dataset and full-scale nightly runs.8. Reporting & remediation - Produce machine-readable report and human-friendly summary: diffs, failing checks, top slow queries, EXPLAIN highlights, sample rows showing corruption. - Link to reproducible artifact: container image + seed + load scripts so engineers can reproduce locally. - Auto-create rollback PRs or block merge when critical correctness/perf regressions detected.Trade-offs & considerations:- Fidelity vs cost: full prod-scale runs are expensive; mitigate with statistically representative smaller datasets, targeted hotspots, and periodic full-scale runs.- Safety: never store PII; only collect metadata and sanitized aggregates.- Flakiness: mitigate via seeding, fixed workloads, multiple iterations, and baselining.Example technologies:- DB: PostgreSQL/MySQL- Profiling: pg_stats, pg_stat_statements, slow-query logs- Generators: Python (Faker, numpy), Synthpop-like tools- Loaders: COPY, pg_bulkload, k8s jobs- Orchestration: GitHub Actions / GitLab CI / Jenkins + Terraform + k8s- Workload: pgbench/HammerDB/custom scenarios- Observability: Prometheus, Grafana, ELK, and CI artifacts (HTML reports)This pipeline ensures migrations are validated for correctness and performance in a reproducible, safe, and automated way while balancing cost and coverage.
EasyTechnical
39 practiced
Explain canary releases and contrast them with blue‑green deployment. Describe the operational steps to run a canary for 5% of traffic: how to route traffic, collect and compare metrics, decide success/failure, and handle rollback. Mention limitations when dealing with low traffic services or non‑deterministic bugs.
Sample Answer
Canary releases: gradually roll a new version to a small subset of real users to validate behavior in production before full rollout. Blue‑green deploys cut over all traffic from one identical environment (blue) to another (green) instantly, enabling fast rollback by switching back; canary is incremental and safer for subtle issues, while blue‑green is simpler for quick atomic switchovers.Operational steps to run a 5% canary:- Route traffic: use weighted routing in your load balancer/ingress (e.g., ALB, Istio, Envoy, GCLB) or feature flags to send 5% of sessions/users to canary instances. Prefer sticky session and consistent hashing if sessions matter.- Collect & compare metrics: capture key KPIs (error rate, latency p95/p99, throughput, business metrics like conversion) from both canary and baseline. Use distributed tracing and logging to correlate traces. Aggregate over an appropriate window (e.g., 30–60 min) and compute relative deltas.- Decide success/failure: define SLO-based thresholds and statistical tests (e.g., compare error rates with confidence intervals or use A/B testing metrics). Example rule: if error rate increases > 2x or latency p95 worsens > 20% with p<0.05, mark as failed.- Handle rollback: automate rollback via the same routing control—reduce canary weight to 0% or flip feature flag; drain and terminate canary instances. Ensure rollback also reverts any DB schema or feature-flagged behavior or run compensating migrations.Limitations:- Low-traffic services: 5% may produce insufficient sample size to detect regressions; need longer observation, synthetic traffic load tests, or increase percentage temporarily for detection.- Non-deterministic or rare bugs: canaries might miss race conditions or memory leaks that manifest only under sustained load—combine canaries with chaos testing, longer soak tests, and observability for resource metrics.- Stateful changes and migrations: schema or data changes may not be safely canaried without feature toggles or backward-compatible migrations.This approach balances risk and velocity; automate routing, alerting, and rollback to make canaries reliable and fast.
MediumSystem Design
42 practiced
Design a CI/CD pipeline for a git monorepo containing 30 microservices in different languages. Requirements: fast developer feedback, isolated builds per service, shared base images, artifact storage, environment promotion, PR validation, and automated rollback. Describe triggers, build isolation, caching strategies, artifact tagging and promotion flow, and how to avoid unnecessary builds.
Sample Answer
Requirements (clarify): fast dev feedback, isolated per-service builds in a git monorepo (30 services, multi-language), shared base images, artifact storage, environment promotion (dev→staging→prod), PR validation, automated rollback, avoid unnecessary builds.High-level architecture:- Git monorepo + path-based change detection- CI system (e.g., GitHub Actions / GitLab CI / Tekton) orchestrating per-service pipelines- Build runners in k8s or autoscaling pool- Container registry (ECR/GCR/Harbor) + artifact repo (Nexus/Artifactory) for non-container artifacts- CD (Argo CD/Spinnaker/Flux) for promotion and rollback- Cache service (registry for layers, remote build cache like Bazel/Gradle/ccache)Triggers and avoiding unnecessary builds:- On PR or push: detect changed paths via a fast script mapping file globs → service(s). Trigger pipelines only for affected services + shared infra if shared files changed (e.g., /libs/, /Dockerfile.base).- On merge to main: same selective build for affected services.- Scheduled nightly full build for integration smoke tests.- Label-based overrides: force-build-all when needed.Build isolation:- Each service has its own pipeline job running in an isolated container build environment; use ephemeral runners per job.- Use language-specific builders (maven, npm, pip) inside the job; store build metadata linking commit SHA → artifact.Shared base images & caching strategies:- Maintain a versioned base image repo (e.g., base:java-11-2025-06). CI pipeline rebuilds base image when base dependencies change; services rebase if base image tag changed.- Use layered Dockerfiles: COPY only app code late to maximize layer cache reuse.- Enable registry-backed build cache: buildkit/kaniko with a remote cache (container registry) to reuse layers across services and runs.- Use artifact cache (maven/ npm proxy) and per-repo remote caches (Gradle/Bazel remote cache) for dependency reuse.- For compiled languages, use incremental build caches persisted between jobs keyed by service + commit hash.Artifact tagging and promotion flow:- Build artifacts and images tagged with: service:commit-SHORT, service:branch-<name>, and semantic tags on promotion (service:staging, service:prod) plus immutable digest-pinned tags.- After CI tests succeed on main, publish image with tag service:commit-SHORT and also service:canary-<build-id>. Deploy canary to staging via CD.- Promotion: a promotion pipeline promotes the immutable digest to environment tags (staging→prod) after passing integration/e2e and manual or automated gate (SLOs, approval). Promotion writes metadata (artifact SHA, build id, tested environments) to artifact registry.PR validation:- On PR open/update, run fast checks: linters, unit tests, build (container build with cache), and smoke integration (if lightweight). Report status checks; only merge when green.Automated rollback:- CD records deployed image digest per environment. On alert or failed health check, CD triggers rollback to last known good digest automatically (with throttles/approval for prod). Keep immutable artifacts forever or for retention period.Additional optimizations:- Parallelize per-service jobs; limit concurrency based on infra.- Use dependency graph so if service A depends on lib X and lib X changed, build A too.- Use incremental tests: run full test suite only on nightly or before promotion; run focused tests on PRs.- Monitor pipeline metrics (build time, cache hit rate) and tune.Trade-offs:- Selective builds speed feedback but require accurate change detection and dependency mapping.- Aggressive caching reduces time but increases operational complexity (cache invalidation).
MediumTechnical
42 practiced
Describe an automated canary promotion algorithm: how to initially route a small percent of traffic, the metrics to collect (error rate, p95 latency, saturations), how to statistically compare canary vs baseline, promotion rules, hold/rollback criteria, and how to handle timeouts or insufficient samples.
Sample Answer
Approach (overview)Start with a small routed fraction, collect key SLO-relevant metrics for both baseline and canary, run statistical tests each evaluation window, and apply deterministic promotion/hold/rollback rules with safeguards for timeouts and low-sample situations.Routing and cadence- Start at 1% traffic for 10–15 minutes, then 5% for 15–30 minutes, 25% for 30–60 minutes, then 100% if all checks pass.- Use weighted routing (service mesh or load balancer) with sticky sessions if needed.Metrics to collect- Reliability: error rate (5xx, application errors), request success rate- Latency: p50/p95/p99, tail behavior- Resource saturation: CPU, memory, thread pools, queue lengths, DB connection pool usage- Business metrics: conversion, throughput, user-perceived errors- Traffic context: request count, user segments, regionsStatistical comparison- Define null: canary metric >= baseline (worse or equal). Use one-sided test.- For rates (error rate): use two-proportion z-test or Bayesian Beta test.- For latencies: compare distributions with Mann–Whitney U or bootstrap CIs on p95.- Use effect size + p-value: require e.g. p < 0.01 AND relative degradation < threshold (or > for improvement).- Account for multiple comparisons with Bonferroni or require majority of critical metrics pass.Promotion rules- Promotion if during last N windows (e.g., 3 consecutive) critical metrics show no regression beyond tolerances: - error rate: increase <= +0.1% absolute OR relative <= 10% - p95 latency: increase <= +10% - saturation: no increase beyond preset safe thresholds- Require minimum sample size: e.g., ≥1000 requests per window or >30 errors for meaningful rate comparison.Hold / rollback criteria- Immediate rollback if any critical metric crosses an absolute safety threshold (e.g., error rate >2% or p95 > 2x baseline).- Soft hold if statistical test flags regression but below safety threshold—keep at current traffic and notify on-call; run extended sampling.- Auto-rollback on sustained failures (e.g., 2 consecutive windows failing statistical test and safety thresholds).Handling timeouts / insufficient samples- If samples insufficient: hold and extend window up to a max (e.g., 2× window). If still insufficient, abort promotion and require manual review.- For timeouts (downstream slowness causing few responses): detect via request count drop and treat as safety signal—rollback or hold.- Use Bayesian priors to make more robust in low-sample regimes, but bias toward safety (conservative priors favor baseline).Operational considerations- Guardrails: kill-switch, canary isolation (circuit breakers, rate limits), feature flags.- Observability: dashboards, automated alerts, and audit logs for each decision.- Continuous learning: record promotions/rollbacks, refine thresholds and tests based on historical outcomes.Example: after 1%→5%→25% windows each passing 3 consecutive 10-min checks with p < 0.01 on error rate and delta p95 <=10%, promote; if any window shows error rate >2% or a statistically significant increase with p < 0.01 and effect >10%, rollback immediately.
Unlock Full Question Bank
Get access to hundreds of Production Deployments and Operations interview questions and detailed answers.
