Understanding the Company's Infrastructure Context Questions
Research the company's public infrastructure information (engineering blog, tech talks, published case studies, job description). Understand what systems they operate at scale, what problems they likely face, and what your role would contribute to.
HardSystem Design
20 practiced
Design a cross-region strategy for a shopping-cart service that requires low-latency reads from local region and eventual consistency for cart items. Propose data model changes, replication strategy, conflict resolution rules, and how the SRE team would monitor correctness and performance.
Sample Answer
Requirements & constraints:- Local reads must be low-latency (serve reads from local region).- Eventual consistency for cart items is acceptable; avoid lost updates.- Cross-region partitions, intermittent connectivity, and failover support.High-level approach:- Active-active per region with asynchronous replication and CRDT-based conflict resolution (observed-remove set or OR-Set per cart).- Serve reads from local region; accept local writes immediately (optimistic, low latency), replicate updates to other regions via change stream/replication log.Data model changes:- Cart as a map: cart_id -> {items: OR-Set(item_id, quantity, metadata), ttl, last_touch_ts}- Each update is a small operation (add item, remove item, set qty) expressed as CRDT operations with unique operation IDs (timestamp + replica id) and causal metadata (per-cart vector clock or dotted version vectors for compaction).- Tombstones: record removals in CRDT semantics and compact older tombstones after safe TTL.Replication strategy:- Each region writes to local storage (e.g., Dynamo-style key-value store or Cassandra/DynamoDB with streams).- Emit ops to a durable replication stream (Kafka or cloud change-stream) per-cart shard.- Asynchronous subscribers in each region apply remote ops to local CRDT state using idempotent, commutative merge logic (CRDT merge).- Anti-entropy: periodic Merkle-tree or hash sync per cart shard to detect missed ops and repair.Conflict resolution rules:- Use CRDT OR-Set semantics: - Add operations add unique element ids; remove operations remove by element id (observed-remove). - Quantity updates represented as a last-writer-wins on per-item version within the OR-Set or as a monotonic CRDT (PN-counter) depending on business rules.- For business-sensitive conflicts (e.g., promotions): resolve server-side by applying deterministic policy (e.g., higher quantity, or latest timestamp if acceptable) and emit compensating ops if necessary.SRE operational responsibilities (monitoring, correctness, performance):- SLOs & SLIs: - P99 read latency per region <= X ms - Write latency (local persist) P95 <= Y ms - Replication convergence time SLI: % of carts converged within T seconds - Error budget for replication failures, merge errors.- Metrics to emit: - Per-region: read/write latency histograms, request rates, error rates. - Replication: out-queue length, consumer lag, ops/sec replicated, replication API error rates. - Consistency: divergence rate (fraction of carts with different hashes between regions), number of conflicts detected, conflict resolution ops emitted. - Storage: tombstone count, CRDT state size per cart, compaction throughput.- Alerts: - Replication consumer lag > threshold - Divergence rate spike or consistent non-zero divergence beyond grace period - Merge errors or high conflict rate indicating upstream bug - Tombstone growth or storage size per cart exceeds projected bounds - SLO breaches and error budget burn rate- Correctness checks: - Periodic sampling job: compare canonical cart state across regions for random cart sample; verify idempotence and eventual convergence within time window. - Full anti-entropy rounds (Merkle tree checks) nightly for critical shards. - End-to-end tests: simulate concurrent multi-region updates with chaos testing (network partitions, region failovers). - Property-based tests in CI for CRDT merges and compaction logic.- Observability & tooling: - Trace write -> replicate path (distributed tracing) to measure per-op replication latency. - Dashboards: per-region latency, replication pipeline health, convergence percentiles. - Runbooks: steps for replication backlog clearance, manual reconcile (replay streams or run anti-entropy), failover (mark region read/write), and compaction tuning.- Reliability practices: - Backpressure: reject writes or apply degraded mode if replication queue critical. - Safe TTLs & compaction windows to bound state growth. - Canaries for schema/CRDT change: automatic migration with dual-write simulation, verification before rollout.Trade-offs & rationale:- CRDTs give automatic, deterministic merges avoiding coordination and maintaining low-latency local writes. They add storage and implementation complexity (tombstones, vector metadata).- Asynchronous replication favors availability and low-latency but requires robust monitoring and anti-entropy to ensure convergence.- For operations that are strongly business-critical (checkout), perform a synchronous canonicalization step at checkout time (read merged state or reconcile across regions) to ensure correctness for charge fulfillment.This design gives the SRE team clear operational signals to ensure low-latency local reads, bounded convergence time, and tools/alerts to detect and repair divergence while keeping the system highly available and performant.
MediumTechnical
27 practiced
Given a distributed Java backend, globally distributed teams, and compliance requirements, propose a practical automation and tooling stack (IaC, CI, config management, secrets) that balances developer productivity, security, and reliability. Justify your choices and note migration considerations from ad-hoc scripts.
Sample Answer
Requirements to balance productivity, security, reliability: repeatable infra, secure secrets, auditable changes, fast CI, multi-region deploys, policy enforcement.Recommended stack- IaC: Terraform + Terragrunt for environment composition. Terraform modules encapsulate common patterns (VPC, IAM, EKS). Terragrunt enforces DRY and per-region configs.- CI/CD: Git-based pipelines (GitHub Actions or GitLab CI). GitOps for infra and app deploys (Argo CD/Flux) so manifests are the source of truth and rollbacks are simple.- Container/platform: Kubernetes (managed like EKS/GKE) with Helm or Kustomize for templating; namespaces + network policies for tenancy.- Config management: Store runtime config in ConfigMaps/sealed-secrets for non-sensitive items; use Helm charts with values files per environment.- Secrets: HashiCorp Vault (or cloud KMS + Secrets Manager) with dynamic/short-lived credentials, RBAC, audit logging, and AppRole/Kubernetes auth.- Policy & security: OPA/Gatekeeper for admission policies, Terraform Cloud/ Sentinel or Conftest for pre-apply checks, Snyk/Trivy for image scanning.- Observability & compliance: Centralized logging (ELK/Cloud Logging), metrics (Prometheus + Grafana), SLOs, alerting, and immutable audit trails (cloud audit logs).Justification- Terraform + GitOps reduces drift and accelerates safe, auditable changes. Managed K8s reduces ops burden for global teams.- Vault and dynamic secrets reduce blast radius and meet compliance (rotation, audit).- Policy-as-code enforces guardrails early, preventing insecure changes without blocking developer velocity.- Git-based CI with ephemeral environments supports fast feedback for distributed teams.Migration considerations from ad-hoc scripts- Begin with an inventory of scripts and environments; categorize by criticality.- Wrap existing scripts into Terraform null_resource or small modules to preserve behavior, then refactor gradually into modules.- Introduce GitOps for one service/region as a pilot; add automated tests and canary rollouts.- Enforce RBAC and create service accounts for automation, migrate secrets to Vault in phases (read-only -> dynamic creds).- Provide self-serve templates, documentation, and runbooks; run training and pair-programming sessions.- Measure safety: track deployment frequency, mean time to restore, and audit logs to prove compliance improvements.This stack balances developer productivity (templates, GitOps), security (Vault, OPA, scanning), and reliability (managed K8s, Terraform state locking, observability) while allowing incremental migration from ad‑hoc scripts.
HardTechnical
24 practiced
Case study: A public postmortem excerpt says: 'On 2025-10-20, DNS change caused control-plane rate limiting; service registrations were delayed causing health-check failures and cascading retries across services resulting in 40 minutes of degraded service.' Based on this excerpt, propose immediate detection rules, mitigations you would add to runbooks, and three long-term engineering fixes.
Sample Answer
Immediate detection rules (alerts you’d add)- DNS health: alert if >1% DNS lookup latency >500ms or NXDOMAIN/servfail rate >0.5% across clients for 5m. Metric: resolver_success_rate, resolver_latency_p50/p95.- Control-plane rate-limiting: alert when 429 rate exceeds baseline +X sigma (e.g., 429s >0.5% of requests for 5m) and request_retry_rate spikes.- Service registration lag: alert when service registration latency > 10s or registration_count < expected_count for 2 consecutive polls.- Health-check flaps: alert when health-check failure rate increases >100% vs rolling 30m baseline.- Retry amplification: alert when outgoing retry attempts per-minute per-service > threshold (e.g., 5x baseline) and call latencies rise.Runbook mitigations (immediate steps)- Triage: check DNS provider status, resolver metrics, and recent DNS config changes (who/when).- Short-term rollbacks: if recent DNS change, roll back to previous records and verify TTL propagation using dig +trace and multiple public resolvers.- Throttle retries: apply circuit-breaker / backoff policy via feature flag or config to reduce retry fan-out (increase jitter, exponential backoff, max retries=1).- Bypass control-plane: temporarily allow cached registrations/TTL extension and serve using last-known-healthy registry until control-plane recovers.- Mitigate rate-limits: request quota increases with provider or route through alternate resolvers; if internal rate-limiter misconfigured, relax limits carefully.- Post-incident: run consistency check to re-register missed services, annotate incident in changelog.Three long-term engineering fixes1) Resilient service registration: make registration idempotent, support local caching with TTL safeguards and optimistic refresh; allow services to operate on last-known registry for configurable degraded window.2) Control-plane hardening: add adaptive rate-limiters (leaky-bucket with client-weighting), backpressure signals, and prioritized control-plane queues to protect critical registration traffic; add canarying of DNS/metadata changes with automated safety gates.3) Observability + automated rollback: end-to-end tracing from DNS → registration → health-checks; synthetic monitors that exercise registration flow and auto-trigger rollback if regression detected; runbook-driven automation to throttle retries and extend TTLs automatically during DNS/control-plane incidents.Key tradeoffs: favor safety and graceful degradation over aggressive enforcement; apply feature flags to rollout fixes.
HardSystem Design
27 practiced
Design a zero-downtime deployment and schema migration strategy for a service that has thousands of tenants and cross-region database replicas. Include safe schema evolution patterns, feature flags, phased rollout, backward compatibility rules, and rollback mechanisms when schema changes cannot be reversed.
Sample Answer
Requirements & constraints:- Zero downtime for thousands of tenants, cross-region DB replicas, transactional correctness, support safe rollback when schema changes are non-reversible, low-latency reads, and ability to phase rollout by tenant/region.High-level approach:1. Prepare migration pattern: make changes backward-compatible first, deploy code that can handle both old and new schema, then perform non-breaking migration, switch traffic, finally cleanup.2. Use feature flags + phased rollout per-tenant/region.3. Automate and monitor every step with runbooks and safety gates.Architecture & core components:- Migration service (or tool like gh-ost, pt-online-schema-change, custom job) with orchestration.- Feature flag system (per-tenant rollout).- Dual-write layer in application (adapter) to write to old and new schema simultaneously for canary tenants.- Background backfill workers and verification jobs.- Migration control-plane (CI/CD + operator) that sequences schema changes across primary then replicas.- Observability: metrics, synthetic tests, slow query log, audit of row counts, schema checksums.Safe schema evolution patterns:- Expand-only changes first: add nullable columns, new tables, indexes (concurrent index creation).- Use shadow columns/tables: create new column/table and keep write-path dual-writing for canary tenants.- Dual-read/feature-flagged read path: read from new column for flagged tenants, fallback to old path.- Data backfill in background with idempotent workers; validate with checksums.- For renames / destructive changes: perform as two-step: add new, migrate, switch reads, then remove old after wide verification.Phased rollout:- Canary: pick small set of tenants in one region (via feature flag) => enable dual-write + read-from-new for those tenants.- Gradual ramp: increase percent/tenant cohorts by automation (e.g., 5% -> 25% -> 100%) per region, waiting for health gates.- Region-aware ordering: roll primary region first, ensure replica replication lag minimal, then other regions.Backward compatibility rules:- Every release must be able to operate with either schema variant; clients and services must tolerate missing columns.- Avoid breaking changes in a single deploy; remove deprecated fields only after all clients are verified.- All db clients validate schema version; application records schema version metadata.Rollback and non-reversible changes:- Prefer forward-only safe steps. But if irreversible change is needed: - Use blue-green for application: keep old app and old schema in parallel until rollback window passes. - Maintain backups & logical snapshots before final destructive migration. - Implement compensating migrations: e.g., keep deleted data in an archive table with mapping to recover. - Orchestrated rollback: stop writes to new path, switch feature flags back, enable old code path, replay from WAL/backups to restore missing rows. - If full rollback impossible, implement “forward-fix” plan: hotfix code to handle migrated state and reconcile data.Automation & safety gates:- CI runs schema linter and simulation tests.- Pre-migration dry-run on staging with production-sized dataset (or sampled tenants).- Automated checks between phases: replication lag < threshold, error-rate, latency, DB locks, row-count validation, checksum parity.- Human approval required at key gates; automated rollback triggers on metrics breach.Monitoring & SLOs:- Track SLOs (latency, error-rate, replication lag), migration-specific metrics (rows processed/sec, percent complete, checksum pass/fail).- Synthetic transactions per tenant group.- Alerting and runbooks for common failures (long locks, high replication lag, checksum drift).Example sequence:1. Add new nullable column and concurrent index.2. Deploy app that dual-writes to old+new for canary tenants; feature-flag reads to new for those only.3. Backfill historical rows via background workers and validate checksums.4. Gradually enable read-from-new via feature flags across tenants/regions.5. After full rollout and stability window, schedule removal of old column in a separate maintenance window (or keep for long-term).6. If rollback needed at any stage, flip feature flags to old behavior, halt dual-write, and use audit logs to reconcile.Trade-offs:- Dual-writing increases complexity and latency; acceptable for safety.- Long-lived deprecated columns increase schema clutter; mitigate with lifecycle policy.- Full offline migrations faster but unacceptable for zero-downtime.This plan ensures availability by decoupling schema change from cutover, uses feature flags for controlled rollout, enforces backward compatibility, and provides rollback and forward-fix paths with automated checks and human gates—aligning with SRE goals of reliability and safe automated operations.
MediumTechnical
28 practiced
You find demos showing internal dashboards and several open-source dependencies. From public sources, identify security and compliance signals you would investigate immediately as an SRE and propose three mitigations SREs should prioritize to reduce attack surface and compliance risk.
Sample Answer
Signals to investigate immediately- Publicly accessible UI endpoints or demos (open dashboards, unauthenticated routes, exposed ports) — verify DNS, TLS, and IP allowlists.- Leaked secrets in repos or commits (API keys, cloud creds, CI tokens) — search public GitHub, GitLab and CI logs with secret-scanning tools.- Outdated or vulnerable OSS components (high/critical CVEs, unmaintained libs) — check dependency manifests, CVE databases, and vendor advisories.- Exposed cloud metadata endpoints or misconfigured storage (public S3/GCS buckets, permissive IAM policies).- Insecure auth/config (default credentials, no MFA, wide RBAC) and permissive CORS or IP rules.- Evidence of third‑party integrations with broad scopes, or risky licenses/compliance flags.Three prioritized SRE mitigations (what to do first and why)1) Lock down access to dashboards and internal endpoints- Actions: enforce network-level controls (private subnets, VPN, IP allowlists), require single-sign-on + MFA, implement least-privilege RBAC for dashboards and APIs, and enable strong TLS.- Why: immediate reduction of exposure; prevents casual public discovery and unauthorized read/write access.2) Secrets discovery, revocation, and centralized secrets management- Actions: run automated secret scanners across repos and CI logs, rotate/ revoke any exposed keys immediately, onboard secrets into a vault (HashiCorp/Cloud KMS) and inject at runtime, remove credentials from images and configs.- Why: leaked credentials are high-risk and often lead to lateral movement and data exfiltration.3) Harden supply chain and dependency patching- Actions: produce SBOMs, enable SCA (Snyk/Dependabot/OSS-Fuzz), pin and sign dependencies and images, enforce image scanning in CI/CD and fail builds on critical CVEs, establish scheduled patching and canary rollouts.- Why: many compromises stem from vulnerable/compromised OSS; automation reduces patch windows and compliance gaps.Complementary controls: enable audit logging and alerting for anomalous access, implement runtime protections (WAF, EDR), and document compliance evidence (artifact retention, change logs) for auditors.
Unlock Full Question Bank
Get access to hundreds of Understanding the Company's Infrastructure Context interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.