Covers the end to end practices and trade offs involved in releasing, running, and operating software in production environments. Topics include deployment strategies such as blue green deployment, canary releases, and rolling updates, and how each approach affects reliability, rollback complexity, recovery time, and release velocity. Includes feature flagging and release gating to separate deployment from feature exposure. Addresses continuous integration and continuous deployment pipeline design, automated testing and validation in pipelines, artifact management, environment promotion, and release automation. Covers infrastructure as code and environment provisioning, containerization fundamentals including container images and runtimes, container registries, and orchestration fundamentals such as scheduling, health checks, autoscaling, service discovery, and the role of Kubernetes for scheduling and orchestration. Discusses database migration patterns for large data sets, strategies for online schema changes, and safe rollback techniques.
Explores monitoring and observability including metrics, logs, and traces, distributed tracing and error tracking, performance monitoring, instrumentation strategies, and how to design systems for effective troubleshooting. Includes alerting strategy and runbook design, on call and incident response processes, postmortem practice, and how to set meaningful service level objectives and service level indicators to balance reliability and velocity. Covers scalability and high availability patterns, multi region deployment trade offs, cost versus reliability considerations, operational complexity versus operational velocity trade offs, security and compliance concerns in production, and debugging and troubleshooting practices for distributed systems with partial information. Candidates should be able to justify trade offs, explain when a simple deployment model is preferable to a more complex architecture, and give concrete examples of operational choices and their impact.
HardTechnical
82 practiced
An intermittent partial outage affects about 0.5% of users in a region. Logs are sampled, traces are incomplete, and several services lack metrics. Describe a pragmatic troubleshooting approach to isolate root cause with partial observability: which temporary instrumentation to add, how to increase sampling safely, and how to avoid causing further production impact.
Sample Answer
Start by scoping and forming hypotheses quickly- Confirm user-reported symptoms, affected region, client versions, and exact 0.5% cohort characteristics (ISP, client build, feature flags, endpoints). That narrows instrumentation targets.- Prioritize hypotheses (network, auth, specific service, dependency) and test them one at a time.Pragmatic instrumentation to add (temporary, low-overhead)- Add structured, sampled debug logs around the suspected code paths with a correlation ID (request-id). Log only essential fields: request-id, service, latency, error code, upstream host, feature flags, client version.- Emit lightweight business/technical metrics (counters + histograms): request count, error count by error type, p95/p99 latency, and dependency error rates. Use service tags: region, shard, client-version.- Add short traces or span-openers at service boundaries (no full payloads) to correlate distributed failures.- Add health-check style synthetic monitors from the region to the critical path.How to increase sampling safely- Use targeted sampling: increase trace/log sampling only for the affected region, client versions, or request paths, not globally.- Implement adaptive capture: when an error threshold is hit (e.g., error rate > baseline + Xσ), temporarily bump sampling for that request-id or client-session for N minutes.- Use rate limits: cap extra logs/traces to a small absolute rate (e.g., +1000 traces/min) to protect downstream systems.- Prefer adding coarse-grained metrics first (cheap), then increase trace/logging only for requests that fail or exceed latency thresholds.Avoid causing further production impact- Deploy via config flags / feature toggles and use fast rollbacks.- Canary changes to a tiny percentage of instances or run in read-only/sidecar mode to avoid altering request flows.- Ensure instrumentation is non-blocking and asynchronous (buffered writers, backpressure protections).- Monitor observability pipeline health (queue sizes, ingestion errors) and set alarms to auto-disable high-volume capture if backpressure appears.- Limit PII — sample payloads and never log sensitive fields.Triage loop and validation- Correlate new metrics/traces with user reports to confirm root cause. If identified, implement targeted fix or mitigation (e.g., retry logic, circuit breaker, routing change).- After fix, revert temporary instrumentation and document learnings and permanent observability improvements (missing metrics, too-coarse sampling policies).Example quick action: enable a region-specific config that logs structured errors + p99 latency and increases trace sampling 10x only for failed requests, capped at 500 traces/min, deployed to 5% canary hosts — observe for 15 minutes, then expand or rollback based on telemetry.
EasyTechnical
56 practiced
List and explain the key metrics you would monitor during a canary release for a web service. Include system metrics (e.g., latency percentiles, CPU), business metrics (e.g., checkout conversion), and explain why each metric matters for canary health judgments.
Sample Answer
When running a canary release I monitor a mix of system SLIs, user-facing SLIs, and business KPIs so I can judge both technical stability and customer impact. For each metric I compare canary → baseline (control) with thresholds and statistical significance.System / infrastructure:- Latency percentiles (P50/P95/P99): shows user-perceived responsiveness and tail latency regressions that affect UX. P95/P99 are critical for identifying outliers introduced by the change.- Error rate (4xx/5xx, RPC/gRPC errors): direct indicator of functional failures; even small increases warrant investigation.- Throughput / requests per second: ensures canary can handle load; sudden drops may indicate failures or throttling.- CPU, memory, GC pause times: detect resource regressions or memory leaks introduced by the canary.- Threadpool / connection pool saturation, queue lengths: reveal contention or blocking behavior.User-facing SLIs:- Successful transactions / availability (percent successful requests): combines latency and errors into availability SLI.- Time-to-first-byte / DOM load (if front-end involved): measures perceived performance.Business metrics:- Checkout conversion rate, add-to-cart rate, retention/engagement, revenue per user: detect subtle degradations that technical metrics may miss. If conversions drop in canary group vs control, rollback even if system metrics look OK.Operational practice:- Use rolling windows, confidence intervals, and minimum sample sizes before decision.- Define automated guardrails (e.g., >2x baseline error rate or P99 increase >50% for 5m) and a clear rollback/hold policy.- Correlate metrics (spike in P99 + increase in CPU + error rate) to accelerate diagnosis.These metrics together let you balance technical health with business impact when deciding to promote or rollback a canary.
MediumSystem Design
57 practiced
Design a feature-flagging system for a global SaaS product that supports tenant targeting, percentage rollouts, kill switches, audit logs, and low-latency evaluation. Describe APIs, storage model, synchronization (server vs client-side), cache strategy, and governance (who can create/cleanup flags).
Sample Answer
Requirements & constraints:- Global low-latency eval (99th pct <10ms) for millions of tenants- Supports per-tenant targeting, % rollouts, kill switches, audit logs, admin governance, client & server evaluationHigh-level architecture:- Control Plane (web UI + REST/gRPC APIs + admin service + audit logger + policy engine)- Distributed Config Store (primary: strongly-consistent DB for control plane: PostgreSQL/CockroachDB; materialized to a geo-distributed KV store CDN-like layer: Redis clusters + object store for large bundles)- Data Plane (SDKs for server & client, local cache, polling/streaming via SSE/gRPC or CDN + realtime via message bus)APIs:- Admin API (REST/gRPC, authenticated, RBAC): - POST /flags {key, type, rules, default, metadata} - PATCH /flags/{key}/rules - POST /flags/{key}/kill (immediate global off/on) - POST /flags/{key}/percent-rollout {percentage, seed} - DELETE /flags/{key}- Evaluation API: - GET /eval/{tenant}/{flagKey}?context={...} -> {variation, reason, cacheControl}- Audit API: - GET /audit?flag=&actor=&since=Storage model:- Canonical config in relational DB with normalized tables: flags, rules, conditions, audiences, tenants, change_history.- Precomputed compiled bundles (JSON protobuf) per environment/tenant stored in object store and cached in Redis/CDN edge for fast retrieval.- Audit logs appended to an immutable event store (Kafka -> long-term storage in S3 + indexed in Elasticsearch for queries).Sync and evaluation:- Server-side SDKs: - Prefer local evaluation: SDK loads bundle (per tenant/environment) from CDN/Redis; maintains local copy; subscribes to change stream (Kafka via regional bridge -> push via SSE/gRPC) for near-real-time updates (<1s). - Fallback: on-miss call Evaluation API with rate limiting.- Client-side (browser/mobile): - Use opt-in client SDKs with limited rules (no sensitive logic). Fetch signed bundles from CDN; use local evaluation. Small bundles targeted to tenant and audience segments. Use streaming or short-poll (30s) for updates.- Percentage rollouts: - Deterministic hashing on tenant+identifier+seed performed in SDKs ensuring consistent bucketing; control plane sets seed and percentage.- Kill switch: - Immediate kill via control plane writes to DB + publish to change stream; Data plane applies on first event (SSE) or next short-poll; for emergency, Evaluation API checks a fast Redis override key for immediate global override.Cache strategy:- Multi-layer: 1. CDN / edge caching for bundle retrieval (TTL short, invalidated on change). 2. Regional Redis for hot tenant bundles. 3. Local in-process SDK cache for zero-RPC eval. 4. Audit/event streams for incremental updates.- Cache invalidation via publish/subscribe: control plane publishes change event -> regional brokers -> invalidate Redis & CDN (via cache-control/purge) and push to SDKs.Security, consistency & latency trade-offs:- Strong consistency for control plane (who wrote what); eventual consistency for distribution (acceptable TTL < 1s for normal updates). Emergency kill supports stronger immediacy via Redis override keyed to flag.- Sensitive decisions can be kept server-side to avoid exposing logic in client bundles.Audit & observability:- Immutable audit records for every create/update/delete with actor, IP, timestamp, diff stored in event store + indexed search.- Evaluation logs: sample of eval decisions streamed (not full traffic to avoid cost) for observability and debugging.- Alerting on abnormal kill events or mass changes.Governance:- RBAC integrated with SSO (SCIM): roles: Admin, Owner (per-tenant), Developer, Auditor, Read-only.- Policy: - Who can create flags: Tenant Owners + Platform Admins by default. - Who can release to production: Requires two approvals for production-targeted flags (configurable). - Cleanup: Flags auto-expire if unused for X months (configurable) — deletion requires owner approval; system sends warnings and creates ticket in workflow tool.- Compliance: retention policies for audit logs; flag metadata includes business justification, owner, expiry, and tags for billing.Operational considerations:- Rate-limit Evaluation API; encourage SDK local eval for scale.- Chaos-tested kill switch and rollout reversal.- Provide migration tooling, CLI and IaC integration to manage flags in CI.This design balances low-latency local evaluation with centralized governance, immediate emergency control, auditable changes, and global distribution.
MediumTechnical
50 practiced
Describe autoscaling strategies for stateless frontends and stateful databases. Explain Kubernetes HPA, VPA, and Cluster Autoscaler, and when to prefer horizontal vs vertical scaling. Recommend safe scaling policies (cooldowns, max/min replicas, surge limits) for a service with highly variable traffic.
Sample Answer
Start by separating concerns: stateless frontends scale horizontally; stateful databases require cautious vertical scaling, sharding, or controlled horizontal approaches (read replicas, leader-follower) with attention to data consistency.Kubernetes primitives:- HPA (Horizontal Pod Autoscaler): scales pods based on CPU/memory or custom metrics (QPS, latency). Good for stateless services where additional pods reduce load linearly.- VPA (Vertical Pod Autoscaler): adjusts pod resource requests/limits to fit observed usage. Useful for single-replica or stateful workloads where adding CPU/memory avoids OOMs, but can trigger restarts — use with caution and not alone for high-availability frontends.- Cluster Autoscaler: adds/removes nodes based on pending pods and pod resource requests. Works with cloud provider APIs and is essential when HPA+VPA need extra node capacity.Horizontal vs Vertical:- Prefer horizontal scaling for stateless frontends: better fault tolerance, rolling updates, and linear capacity increase.- Prefer vertical scaling for monolithic stateful DBs or when scaling out is complex. Consider read replicas or sharding as horizontal strategies for databases.Safe scaling policy for highly variable traffic:- Min replicas: set to survive baseline traffic and node failures (e.g., 2–3 across AZs).- Max replicas: cap to control cost and cascading downstream load (calculate based on backend capacity).- Cooldown/ stabilization: 60–180s scale-up evaluation, longer scale-down delay (5–15 minutes) to avoid thrashing.- Surge limits: use PodDisruptionBudget and max surge settings for deployments; limit burst scale-up rate (e.g., no more than 2x within cooldown).- Metrics: combine CPU + request rate + p95 latency; use SLO-based autoscaling (KEDA or custom metrics) for event-driven bursts.- Safety: enable lifecycle probes, graceful shutdown, circuit breakers and queueing (rate limit / buffer) to protect stateful backends.These choices balance responsiveness, cost, and safety for production-grade solutions.
HardSystem Design
51 practiced
Design an observability strategy for a distributed microservices platform to enable SLO monitoring, root-cause analysis, and anomaly detection while controlling telemetry costs. Specify tracing sampling, metric cardinality policies, log retention tiers, and how traces propagate context across services.
Sample Answer
Requirements and constraints:- SLO monitoring (latency/availability/error budget), fast root-cause analysis (RCA), anomaly detection (real-time), and bounded telemetry cost (ingest, storage, processing).- High cardinality user IDs exist; high throughput microservices; multi-region.High-level architecture:- Instrumentation: OpenTelemetry (OTel) SDKs across services for traces, metrics, logs; centralized ingest via collector agents → processing pipeline → storage & analysis (time-series DB for metrics, trace store, log archive). Anomaly detection and SLO evaluation run on processed metrics + sampled traces + indexable logs.Tracing sampling:- Use dynamic, multi-stage sampling: - Always-sample: requests affecting SLOs or error traces (status >= 5xx, latency > p95 threshold), and a small percentage (0.5–1%) of all traces for baseline. - Head-based probabilistic sampling at ingress/span with sticky sampling decision propagated in trace context. - Tail-based adaptive sampling in collector: keep traces with rare/error patterns or high resource usage, drop low-value ones while preserving representative traces per service/endpoint. - Store full traces for N% failed requests and for traces flagged by anomaly detectors.Metric cardinality policy:- Enforce low-cardinality core metrics: service, endpoint, region, status_code, env.- High-cardinality tags (user_id, session_id, request_id) must never be metrics; instead: - Capture those in logs as indexed fields with sparse indexing. - Use aggregation keys (user_bucket, client_type) for per-user patterns when needed.- Apply cardinality limits per metric (e.g., max 1000 label values); emit dimensional rollups (per-minute aggregate) at source via SDK or collector transforms.Log retention tiers:- Hot (7–14 days): full-text logs indexed for recent RCA and tracing correlation.- Warm (30–90 days): compressed, sampled, or limited-index logs (store metadata + pointers to cold archive).- Cold (1–5 years): archive raw logs (S3/Glacier) with event-level encryption; retrieval via async restore.- Implement tiered indexing: only selected fields are indexed long-term (error codes, trace_id, user_bucket).Trace context propagation:- Use W3C Trace Context (traceparent) and baggage for correlation; enforce consistent propagation in HTTP/gRPC headers and message brokers (Kafka: headers).- Ensure services propagate trace_id and span_id on all downstream calls and include trace_id in logs and metrics as non-cardinality tag (only used for on-demand joins).- Collector enriches spans with deployment metadata (git sha, service version, region) for RCA.Cost control and policy enforcement:- Deploy OTel collector with transform processors: - Metric rollups and histogram aggregation at edge. - Rate-limiting and per-service sampling policies. - Enforce export filters to prevent high-cardinality metrics from leaving cluster.- Use budgets per team/service and monitor telemetry spend; set automated throttles when budget exceeded.- Retain SLI-focused high-resolution metrics; downsample others (e.g., 1s → 1m) after hot window.SLO and anomaly pipeline:- Compute SLI from low-cardinality high-resolution metrics (availability, latency buckets).- Alerting on error budget burn rate + anomaly detector (statistical / ML) on residuals of expected traffic patterns; when anomalous, trigger automated tail-sampling to capture more traces for RCA.Scalability & trade-offs:- Tail-sampling and dynamic policies add processing cost but reduce long-term storage.- Aggressive cardinality limits reduce per-incident visibility for specific users; mitigate with logs + ad-hoc trace captures.- Prefer collector-side aggregation to minimize network/storage costs.Operational recommendations:- Start with default policies (1% baseline traces, 7-day hot logs) and iterate using telemetry spend dashboards.- Provide SDK templates and linting checks to prevent anti-patterns (high-card tags).- Run periodic audits of emitted metrics and trace rates; automate alerts when cardinality or volume spikes.
Unlock Full Question Bank
Get access to hundreds of Production Deployments and Operations interview questions and detailed answers.
