Large Scale Infrastructure Challenges Questions

Awareness of engineering and operational challenges at massive scale including global network optimization, multi region failover and redundancy, integration of cloud and on premise systems, security and compliance at scale, performance and latency for a global user base, cost optimization across large fleets, and maintaining reliability without exponential operational complexity. Candidates should demonstrate thinking about architecture patterns, trade offs, monitoring and incident response at scale, and strategies for evolving platform capabilities as load and feature sets grow.

MediumTechnical

32 practiced

Explain active-active and active-passive multi-region architectures. For each describe typical failure modes, operational complexity (deployment, testing, data coordination), and cost implications. Provide a short decision matrix to help choose between them based on service characteristics.

Sample Answer

Active-active vs active-passive multi-region architectures

Definitions:- Active-active: multiple regions serve live traffic simultaneously, often behind global load balancing. Data is either multi-master or geo-replicated with conflict resolution.- Active-passive: primary region serves traffic; one or more passive regions stay in standby (replicated data) and are promoted on failover.

Typical failure modes:- Active-active: split-brain/consistency conflicts, cross-region network partitions, replication lag causing stale reads, global load-balancer misrouting.- Active-passive: delayed failover, failover automation errors, data divergence if replication lags, traffic spikes on failover region.

Operational complexity:- Deployment: active-active requires coordinated multi-region deployments and blue/green release across regions; active-passive needs deployment to passive regions but can stagger activation.- Testing: active-active needs chaos testing for partition and conflict resolution; active-passive focuses on failover drills and warm-start validation.- Data coordination: active-active demands conflict resolution, strong/causal consistency strategies or compensating logic; active-passive can use async replication with careful recovery procedures.

Cost implications:- Active-active: higher steady-state cost (full capacity in all regions), higher network egress and operational overhead.- Active-passive: lower steady cost (standby can be smaller), but potential cost for rapid scaling, longer RTO, and occasional expensive emergency fixes.

Decision matrix (short):- Low latency global users + strict RTO/RPO small -> Active-active- Strong single-writer consistency required + cost-sensitive -> Active-passive- Traffic predictable, can tolerate brief failover -> Active-passive- High write volume across regions or need continuous availability -> Active-active

As SRE guidance: choose based on RTO/RPO, consistency model, cost tolerance, and operational maturity; start simpler and evolve after validating automation and tests.

EasyTechnical

32 practiced

Explain the CAP theorem and how its trade-offs apply when designing a globally replicated user profile service. Give a concrete architecture choice for favoring consistency and one for favoring availability, and describe the user-visible impacts and failure modes for each choice.

MediumTechnical

41 practiced

A globally replicated relational database shows replication lag spikes during nightly ETL jobs. Outline your investigation plan: which metrics to collect (I/O, replication queue depth, lock waits), experiments to isolate the cause, and mitigations such as throttling the ETL, adjusting replication settings, or changing ETL windowing.

Sample Answer

Investigation plan (goal: determine why nightly ETL causes replication lag spikes, then mitigate without breaking SLAs)

1) Collect baseline metrics (before/during/after ETL)- Replication-specific: replica_io_lag / replica_sql_lag, replication queue depth (binlog_send_queue_size / relay log size), last_applied_lsn, per-replica apply rate (txn/sec).- DB resource: TPS, long-running transactions, lock waits (lock/row lock counts, wait time), InnoDB/pg stat: buffer pool hit ratio, checkpoint activity.- I/O & storage: disk IOPS, avg read/write latency, iowait, throughput (MB/s), queue length (iostat, nvme-cli).- CPU/memory: CPU utilization, context switches, page faults, swap.- Network: bandwidth, retransmits, latency between primary and replicas.- OS/DB events: checkpoint/flush rates, wal/redo generation, autovacuum or compaction activity.- Application-level: ETL batch sizes, transaction sizes, number of affected rows, statement types (bulk INSERT/UPDATE/DELETE).- Tools: Prometheus + Grafana, pt-heartbeat/pg_stat_replication, iostat/vmstat/sar, tcpdump if needed, slow query logs.

2) Hypotheses to test (common causes)- ETL creates huge transactions causing large WAL/binlogs and long apply times on replicas.- ETL causes storage saturation (high write IOPS/latency) slowing replication apply.- ETL locks hot tables/rows blocking replication SQL apply (serialization/lock waits).- Network saturation between primary and replica.- Replica apply single-threaded bottleneck (apply worker limits).

3) Experiments to isolate cause (do on staging if possible)- Re-run ETL in staging with identical dataset while instrumenting metrics to reproduce.- Run ETL but pause replication on a test replica: observe primary resource usage — if primary I/O/CPU spikes but no replica lag, likely apply-side issue.- Run ETL against primary with replicas isolated (network limited) to separate network vs storage/CPU.- Break ETL into smaller transactions / batch sizes and measure delta on replication lag.- Convert a subset of ETL statements to use bulk COPY or batched upserts to compare performance.- On a replica, replay only relay logs locally (simulate apply) to measure SQL-apply throughput and detect lock contention.- Increase replica parallel apply workers (if DB supports parallel replication) temporarily and measure improvement.- Use pt-deadlock-logger / db-specific lock diagnostics to find blocking statements.

4) Mitigations (short → medium → long term)Short term (quick wins)- Throttle ETL: add rate-limiting between batches, increase sleep between transactions, limit concurrent writers.- Reduce transaction size: commit more frequently to reduce relay log chunk sizes and lock durations.- Schedule ETL window where SLOs tolerate lag; add staggered start times across regions.

Medium term- Tune replication settings: enable parallel apply, increase replica worker threads, tune commit flushing (group_commit), adjust wal_sender/receiver buffers.- Offload ETL writes to a writable staging DB or use logical replication to a separate writable node to avoid impacting primary.- Use row-based binlog (if currently statement-based) to reduce replication ambiguity and large statements.

Long term- Scale storage: faster disks (NVMe), more iops, provisioned throughput.- Redesign ETL: incremental CDC rather than full-table writes; use bulk-load-friendly patterns (COPY, partition swap).- Add read-only materialized replicas for reporting; separate OLTP and analytics paths.

5) Validation & rollback- Implement mitigations on staging, run full ETL, measure replication lag and SLA impact.- Gradually roll to prod during low-risk window and monitor closely; have rollback (stop ETL, revert throttling) ready.- Postmortem: document root cause, actions, dashboards/alerts to detect recurrence (alert on sustained replica_sql_lag > threshold).

This approach isolates whether the bottleneck is primary write generation, network, or replica apply, prioritizes low-risk mitigations (throttling, batching), and progresses to config/architecture changes if needed.

MediumTechnical

40 practiced

Design a cross-region caching strategy for user session data where some fields require low-latency consistency (e.g., authentication token) and other fields can be eventually consistent (e.g., preference flags). Explain cache partitioning, TTLs, replication, and fallback patterns to the authoritative store.

Sample Answer

Clarify requirements:- Strong read-after-write/low-latency consistency for auth-critical fields (auth token, user session validity).- Eventual consistency acceptable for user preferences, UI flags.- Cross-region reads with low latency; tolerance for temporary divergence for non-critical fields.- High availability and clear rollbacks during region outages.

Design (high level):- Multi-tier cache: (1) local per-node in-memory LRU (very short-lived), (2) regional Redis cluster (primary cache), (3) authoritative store (global DB like Spanner/Cockroach / primary user store).

Cache partitioning:- Partition by user-id using consistent hashing so a user’s session keys map to the same regional cache shard; ensures even load and easier targeted invalidation.- Separate key namespaces for strong vs eventual fields, e.g., session:{user}:auth and session:{user}:prefs. This allows different policies per namespace.

Replication & consistency:- Auth namespace: synchronous replication across at least majority of regional replicas (or use a distributed consensus-backed store / Redis with CRDT or Redis Raft) so reads in-region see most recent writes. Use short TTL (e.g., 30s–120s) and write-through from app -> cache -> authoritative store (or write to DB then invalidate/refresh cache synchronously) to guarantee freshness.- Prefs namespace: asynchronous replication between regions (eventual). Longer TTLs (minutes to hours). Use update propagation via CDC or pub/sub to reduce load on authoritative DB.

TTLs & eviction:- Auth token: very short TTL plus Signed Tokens (JWT) with expiry to reduce need for lookups; keep a blacklist/allowlist in strongly-consistent store for revocation. Use “cache-aside” or write-through with immediate invalidation on write.- Preferences: longer TTL and support stale-while-revalidate: serve stale cached value while background job refreshes from authoritative store.

Fallback patterns:- Read-through / cache-aside for both namespaces: app checks cache, on miss reads authoritative store and populates cache.- Stale-while-revalidate for prefs: return cached value immediately, trigger async refresh; return stale only up to a max window configured.- For auth: on cache miss, synchronously read authoritative store; if authoritative store unreachable, fail-safe: allow token if locally validated signature + not blacklisted (configurable risk), else deny — document as SRE-runbook with incident thresholds.- Circuit breakers and graceful degradation: if cross-region replication lag exceeds threshold, route reads to local authoritative replica or force revalidation.

Operational considerations:- Instrument replication lag, cache hit ratio, TTL expirations, and token revocation latency. SLOs: auth validation p99 < X ms; correctness SLO for revoked tokens < Y seconds.- Automated reconciliation jobs for prefs (detect drift), and periodic cache priming for high-traffic users.- Security: encrypt cache-in-transit and at-rest, rotate keys, and monitor suspicious access patterns.

Trade-offs:- Strong consistency for auth increases latency and operational complexity; mitigate with short JWTs and local signature checks.- Eventual prefs reduce cross-region write amplification at cost of temporary divergence—acceptable for UI flags.

This design balances low-latency auth correctness with scalable, eventually-consistent session metadata, with clear fallback, monitoring, and runbook steps for SRE operations.

HardSystem Design

39 practiced

Design a testable disaster recovery (DR) system that supports automated full failover drills for critical services including synthetic verification tests and data integrity checks. Describe how to schedule drills, isolate test traffic from production, automate validation, and handle rollback after failed tests while minimizing customer impact and meeting compliance requirements.

Sample Answer

Requirements & constraints:- Full automated failover drill for critical services with minimal customer impact- Synthetic verification + data integrity checks- Scheduled + ad-hoc runs, isolation of test traffic, automated validation, rollback on failure- Auditability for compliance, RBAC

High-level architecture:- Orchestrator (Kubernetes Job/Argo Workflows or Jenkins/Concourse pipeline) drives drill steps- DR Playbook service (YAML/JSON per-service drill definitions)- Environment controller (provisions isolated DR namespace/tenants, test network overlays)- Traffic switcher (feature-flagged DNS / traffic steering via Envoy or global load balancer)- Synthetic test runner (k6/Robot/pytest suites) and data-integrity checker (schema checks, checksums, row counts, test DB clones)- Observability & validation engine (Prometheus + custom validators) reads SLOs, metrics, logs- Audit & reporting (immutable logs to S3 + signed attestations)

Drill flow:1. Pre-checks: capacity, backups, schema compatibility, consent windows (business calendar)2. Snapshot production data to isolated test copies (snapshots, logical backups) with PII masking3. Provision isolated DR environment: separate VPC/subnets, namespaces, service accounts4. Controlled failover: orchestrator triggers traffic switcher to route a small percentage (canary) of synthetic traffic only to DR env; production traffic untouched5. Synthetic verification: health endpoints, end-to-end flows, latency/throughput tests run as suites; data-integrity: replay subset of write workload against test DB and verify checksums/consistency6. Progressive ramp: if green, increase percentage and run full failover simulation; if any validator fails, trigger rollback

Isolation of test traffic:- Use separate DNS names / header-based routing or mTLS client certs to ensure synthetic requests are distinguishable- Network-level isolation via test VPC peering and egress controls- No production write paths enabled from test env unless using read-only snapshots

Automation & validation:- Validators codified as pass/fail with thresholds; observability exposes SLOs and slis- Orchestrator polls validators and enforces timeouts- Failure modes captured: critical, warning with remediation playbooks- All actions logged with timestamps, operator IDs, CI run IDs for compliance

Rollback & minimizing customer impact:- Default to "non-disruptive" canary-first approach; only escalate to full traffic switch after multi-stage green checks- On failure: immediate traffic re-route back to production via traffic switcher (atomic DNS/ratelimit rollback), terminate DR env to prevent drift, and run automated remediation (restore snapshots, re-sync)- Post-failure: run automated incident creation with attachments, and require manual approval for repeat attempts

Compliance & audit:- Immutable logs, signed attestations, encryption-at-rest/in-transit, PII masking, role-based approvals, retention policies- Pre-drill approvals and blackout windows recorded; drill reports with evidence (screenshots, test results, hashes) stored for audits

Trade-offs:- Full production write simulation expensive and risky — prefer sampled replay onto masked clones- Faster drills vs deeper tests: balance via tiering of services (critical vs non-critical)

Implementation notes:- Use Infrastructure-as-Code (Terraform) and GitOps for playbooks- Use feature flags and canary tooling (Flagd/LaunchDarkly + Istio/Envoy)- Continuous improvement: run quarterly full drills, weekly mini-canaries, and automated postmortems.

Unlock Full Question Bank

Get access to hundreds of Large Scale Infrastructure Challenges interview questions and detailed answers.

Join thousands of developers preparing for their dream job.