Production Readiness and Professional Standards Questions

Addresses the engineering expectations and practices that make software safe and reliable in production and reflect professional craftsmanship. Topics include writing production suitable code with robust error handling and graceful degradation, attention to performance and resource usage, secure and defensive coding practices, observability and logging strategies, release and rollback procedures, designing modular and testable components, selecting appropriate design patterns, ensuring maintainability and ease of review, deployment safety and automation, and mentoring others by modeling professional standards. At senior levels this also includes advocating for long term quality, reviewing designs, and establishing practices for low risk change in production.

HardSystem Design

46 practiced

Design a scalable monitoring architecture for 10,000 microservice instances producing high-cardinality metrics and distributed traces. Address metric ingestion, storage, downsampling, trace sampling, alerting, and how to ensure SLO calculation remains accurate over time and across retention policies.

Sample Answer

Requirements and constraints:- 10k microservice instances producing high-cardinality metrics (many labels) and distributed traces.- Low-latency alerting for SLOs (minutes), long-term retention for trends (months/years).- Cost control while preserving SLO accuracy and debugability.

High-level architecture:ingest layer → streaming processing (real-time) → short-term hot TSDB + trace cache → downsample/rollup pipeline → long-term cold storage → query & alerting layer → SLO calculator

Core components and responsibilities:1. Ingestion- Use a push gateway/agent per node (Prometheus-compatible push/proxy or OpenTelemetry collector) that buffers and batches.- Fronted by an autoscaling ingestion API (Kafka / Pulsar) for backpressure and durability.- Enforce and validate metric schemas and cardinality limits at collector (deny unbounded label cardinality).

2. Real-time processing & storage- Stream processors (Flink/Beam) consume ingestion stream: - Enrich, validate, dedupe, drop noisy labels, and compute per-bucket aggregations (counters, histograms). - Compute exemplars that link metrics to traces.- Hot TSDB: horizontally sharded, high-cardinality-aware store (e.g., Cortex/Thanos/M3DB) for high-resolution (1s-15s) retention window (e.g., 30d).- Tracing: ingest traces into a scalable trace backend (Tempo/Jaeger backend on object store indexless or with light indexing). Use trace ID in exemplars.

3. Downsampling & long-term storage- Periodic rollup service creates fixed-resolution aggregates (1m, 5m, 1h) stored in cold TSDB and object storage.- Retention policy: keep full resolution for SLO windows (e.g., 28d), store lower resolution beyond that (monthly/yearly).- Maintain lossless aggregates needed for SLOs (see SLO calc).

4. Trace sampling- Use adaptive sampling: - Head-based minimal sampling (1-2%) to keep overhead low. - Tail or dependent sampling triggered by signals: error rates, latency p95/p99, or important endpoints -> sample more for affected traces. - Use probabilistic and reservoir sampling; persist sampling decisions for trace continuity.- Keep full traces linked via exemplars for high-signal metrics.

5. Alerting and anomaly detection- Real-time alerting uses hot TSDB and streaming analytics for threshold and anomaly rules.- Alert deduplication, grouping and suppression (based on service & topology).- Integrate machine-learning outlier detection for unknown failure modes; always have fallbacks to deterministic SLO rules.

6. SLO calculation & accuracy across retention- Define SLO windows and SLI semantics first (e.g., success rate per request).- For each SLI ensure raw event counters (success, total) are retained at sufficient resolution for the longest SLO window where high accuracy is required. If 30d 99th-percentile SLOs are required, keep raw or sufficiently granular histograms for 30d.- Store pre-aggregated lossless counters and histograms for SLO-relevant labels (service, endpoint). Use compressed append-only time-series of counters (cheap).- For longer retention beyond that window, compute and store downsampled aggregates plus metadata that allows unbiased reconstitution of SLOs (e.g., weighted aggregates, bucketed histograms).- When downsampling, maintain error bounds (confidence intervals) and propagate them into SLO evaluation. If uncertainty exceeds threshold, mark SLO result as “inconclusive” and fetch cold raw data if available.

Data flow and SLO guarantees:- Ingest events → stream processor increments SLI counters and emits exemplars.- SLO calculator reads counters from hot TSDB for real-time alerts. For historical SLO reports, it queries rollups; if rollup error margin is unacceptable, escalate to raw cold data retrieval or mark as approximate.- Ensure at ingest time that SLI-related metrics are whitelisted to bypass aggressive cardinality reductions.

Scalability and operational considerations:- Shard TSDB by tenant/service + time to handle cardinality. Use compaction and index pruning.- Enforce cardinality quotas per team; provide tooling to detect high-cardinality label sources.- Autoscale ingestion and streaming workers; use backpressure to shed non-critical metrics under overload.- Cost trade-off: keep high-resolution data only for SLO-relevant slices. Use sampling & rollups for everything else.- Test evictions and disaster recovery by simulating retention compaction and verifying SLO reconstruction.

Trade-offs and alternatives:- Full-indexed trace store (easy querying) vs indexless object-store (cheaper, needs exemplars). We prefer indexless + exemplars for cost, with optional indexing for critical services.- Aggressive sampling reduces cost but risks missing rare issues — mitigate with adaptive/tail sampling and exemplar linkage.- Stronger cardinality enforcement upfront simplifies storage but requires dev cooperation and tooling.

Key metrics to monitor the monitoring stack:- Ingestion rate, cardinality per service, latency from ingest→hot-store, sampling rates, SLO calculation error margin, cost per ingest.

This design balances scalability and cost while ensuring SLO accuracy by protecting SLI data at ingest, keeping lossless aggregates for SLO windows, and propagating uncertainty when using downsampled data.

MediumTechnical

34 practiced

Describe how you would validate environment parity across developer laptops, staging, and production to reduce 'works on my machine' issues. Consider containerization, config-by-env, secrets handling, and data dependencies in your answer.

Sample Answer

Situation: As an SRE, I’d treat environment parity as a program — not a one-off — ensuring dev laptops, staging, and prod run the same artifacts, configs, and data shapes so “works on my machine” disappears.

Approach:- Containerization: Build a single immutable image per service (CI builds, tags by digest). Developers run the same image locally (docker compose / kind / minikube). Deploy that exact image to staging and prod — no rebuilds per environment.- Config-by-env: Keep config externalized (12-factor): environment variables, config maps, and feature flags. Use templating (Helm, kustomize) driven from the same git repo with overlays per env. Validate rendered manifests in CI.- Secrets handling: Never store secrets in repo. Use Vault/Secrets Manager + short-lived credentials, or sealed/sealed-secrets for GitOps. Enforce access policies and audit logging. Provide local secret injection tooling (vault-agent, direnv) for devs.- Data dependencies: Provide sanitized, versioned snapshots or synthetic datasets and lightweight DB fixtures. Run migrations in CI against schema checks and use contract tests to ensure API/DB compatibility.- Validation & automation: - CI checks: image reproducibility, config rendering, linter, schema/migration checks. - Integration & smoke tests in ephemeral staging clusters using same image and configs. - Runtime parity checks: health/probe parity, metrics/alerts mirrored in staging, and automated chaos or load tests. - Drift detection: compare container digests, rendered manifests, and environment variables between staging/prod; alert on divergence.

Example: CI builds image:sha256, pushes to registry; deployment manifests reference that digest. Helm values are generated from same secrets source (Vault). A pipeline renders manifests, runs kubetest in kind with seeded synthetic data and runs end-to-end smoke tests before promoting to prod.

Result: Immutable artifacts + single source of truth for config and secrets, automated validation, and realistic data reduce environment-specific bugs and speed safe rollouts.

EasyBehavioral

37 practiced

As an SRE interviewing a candidate, pose a behavioral question: describe a time you pushed back on a fast schedule because of a production risk. Ask the candidate to explain their communication strategy, how they measured and explained the risk, and the outcome including any lessons learned. What would you look for in a strong answer?

Sample Answer

Question to pose:"Describe a time you pushed back on a fast schedule because of a production risk. Explain your communication strategy, how you measured and explained the risk, and the outcome including any lessons learned."

Sample strong answer (STAR):Situation: On a quarterly release we were asked to deploy a database schema change and associated API changes on a tight timeline that conflicted with ongoing load testing and a partial rollback test failure.Task: I had to decide whether to proceed with the scheduled deploy and, if not, persuade stakeholders to delay until risks were mitigated.Action:- I gathered evidence: recent failure rates from staging, rollback test logs, error budget consumption, and projected user impact (estimated % of requests affected and potential downtime minutes).- I made a concise risk brief (one-page) showing likelihood × impact scenarios and alternatives (canary rollout, feature flag, staged deploy).- I convened a 20-minute decision call with release owner, PM, and lead engineers, presented the brief, proposed a 24–48 hour delay with a mitigation plan (complete rollback test, add a canary and monitor SLOs), and offered a contingency: rapid rollback playbook if we proceed.- I communicated transparently in Slack and the release channel with next steps and owners.Result: The team agreed to delay 36 hours, completed additional tests, and deployed via canary. The change rolled out with zero customer-impacting incidents. Stakeholders appreciated the clear data-driven rationale. Lesson: present quantified risk, offer concrete mitigations and timelines, and keep communication concise and action-oriented.

What I'd look for in a strong answer:- Specific metrics and evidence used to quantify risk (error rates, SLOs, rollback test results).- Clear communication strategy: audience-tailored brief, decision call, and written follow-up.- Concrete mitigation options (canary, feature flags, staged rollouts) and contingency plans.- Ownership and collaboration: who you involved and how you delegated follow-ups.- Outcome with measurable impact and a learning loop (what changed in process afterward).- Calm, data-driven decision-making and empathy for schedule/ business trade-offs.

HardSystem Design

35 practiced

You are asked to design a service-level 'test harness' for simulating 10,000 concurrent users performing complex transactions on a financial service. Define how you would model user behavior, test data lifecycle, environment sizing, result validation, and how to integrate these tests into pre-production validation without impacting production data.

Sample Answer

Requirements & constraints:- Simulate 10,000 concurrent users executing complex, stateful financial transactions (auth, quote, place order, settle).- No impact to production data; realistic latency, failure modes, and data variety.- Integrate into pre-prod CI/CD pipeline and run as scheduled/regression tests.

High-level approach:1. Model user behavior- Create probabilistic user personas (e.g., heavy trader 10%, retail 70%, occasional 20%) with state machines per persona that encode sequences (login -> view -> modify -> submit -> confirm -> notify). Use weighted distributions for think-times (exponential), retry/backoff, concurrency bursts, and session stickiness (cookies/tokens).- Implement flows in an extensible DSL (e.g., Gatling/Scala scenarios or k6 JS) so scenarios map to real API calls and maintain session context.

2. Test data lifecycle & isolation- Provision tenant-like isolated namespaces in pre-prod (dedicated DB schemas, test accounts with synthetic KYC) via infrastructure-as-code. Seed data deterministically using fixtures and idempotent scripts that create accounts, ledgers, and reference rates. Tag all test-created records with a test-run UUID and short TTL.- Use a write-path that routes to test partitions or a shadow DB cluster. Ensure downstream services consuming events read from test topics (namespaced Kafka topics) or from a synthetic event bus.

3. Environment sizing & orchestration- Use containerized load generators (k6/Gatling) scaled on Kubernetes with autoscaling based on virtual user targets. Estimate resource: if one generator can sustain 200 virtual users (measured in dry runs), run 50 pods. Include headroom for spikes (+30%). Size pre-prod service cluster to expected QPS + safety factors; use production-like network topology, latency injection via netem for realism.

4. Result validation & observability- Assertions at multiple levels: functional (correct response codes, business invariants like balance consistency), performance (p95/p99 latency, error rate < SLO), and system health (CPU, GC, thread pools).- Correlate transactions using the test-run UUID; build a validation pipeline that reconciles ledger deltas asynchronously to detect lost/duplicated transactions.- Instrument with distributed tracing (OpenTelemetry), metrics (Prometheus), and logs (ELK) and create dashboards/alerts for threshold breaches.

5. Integration into pre-production- Add tests as staged pipeline jobs: quick smoke run on every PR (small user count); full-scale run nightly/on-demand in an isolated pre-prod cluster. For risk mitigation, use blue/green or entirely separate pre-prod clusters; never point tests at production databases.- Implement “traffic shadowing” for non-destructive read-only flows by duplicating live traffic to a shadow cluster for behavior comparison; avoid writes to production.- Gate deployments: fail fast if test harness detects invariant breaches or SLO violations. Store artifacts (traces, metrics, reconciliation reports) for postmortems.

Trade-offs & best practices:- Synthetic data increases realism vs. complexity of maintaining full parity with prod. Use chaos injection and variable latencies to exercise resilience.- Automate tear-down (TTL) and cleanup to avoid resource leaks.- Run periodic tabletop scenarios using harness outputs to tune autoscaling and capacity planning.

This design provides realistic, scalable, and safe validation of 10k concurrent complex transactions while preserving production integrity and enabling SRE-driven observability and gating.

MediumTechnical

47 practiced

You are reviewing a service that emits millions of log lines per hour and costs the company a large amount in log storage. Propose a strategy to reduce logging cost without sacrificing incident investigatability. Cover log sampling, structured logs, dynamic logging levels, and archiving.

Sample Answer

Start by clarifying goals and constraints: keep high-fidelity data for incidents and retain aggregated/rare-event info cheaply. Then implement a layered strategy combining sampling, structured logs, dynamic levels, and archiving.

1) Structured logs first- Switch to JSON or protobuf fields (timestamp, request_id, trace_id, service, severity, user_id, error_code, msg, extras). - Benefits: enable lightweight indexing, targeted queries, and downstream filtering without full-text scan.

2) Smart sampling- Deterministic head sampling for high-volume normal traffic (e.g., 1–5% of INFO/DEBUG) using request_id hashing so traces stay consistent.- Reservoir sampling for tail/error events: always ingest ERROR/WARN and exceptions; sample INFO/DEBUG but keep all traces that link to sampled errors.- Adaptive sampling: increase sample rate when error rate or latency SLO breaches are detected.

3) Dynamic logging levels & feature toggles- Expose runtime flags (via config service or feature flags) to raise verbosity per service, host, or trace-id on demand.- Tie to alerting: when an alert triggers, automatically enable DEBUG for affected component for a short window to collect richer context.

4) Archiving & tiered storage- Keep hot logs (last 7–30 days) in full fidelity in primary cheaper index (e.g., Elasticsearch/managed logs) for active investigation.- Move older logs to compressed, queryable archive (parquet/optimized object store on S3/BigQuery) with inexpensive indexes for forensic needs; restore on-demand.- Retain full-error event store indefinitely (or per compliance), but compress/aggregate routine INFO.

Operational considerations- Measure baseline: current ingestion rate, cost per GB, and incident query needs. Target 5–10x cost reduction with minimal impact.- Add observability: monitor sampling effectiveness, missing traces, and query latencies; run A/B for sampling params.- Rollout safely: start with non-critical services, provide easy rollback, and document investigators’ playbook (how to bump verbosity and retrieve archives).

Trade-offs- Slightly reduced immediate fidelity for sampled flows vs. big cost savings. Deterministic sampling + always-capture errors preserves investigatability.

Unlock Full Question Bank

Get access to hundreds of Production Readiness and Professional Standards interview questions and detailed answers.

Join thousands of developers preparing for their dream job.