Provide a clear, evidence based overview of your technical foundation and demonstrated credibility as a technical candidate. Describe programming and scripting languages, frameworks and libraries, databases and data stores, version control systems, operating systems such as Linux and Windows, server and hardware experience, and cloud platforms including Amazon Web Services, Microsoft Azure, and Google Cloud Platform. Explain experience with infrastructure as code tools, containerization and orchestration platforms, monitoring and observability tooling, and deployment and continuous integration and continuous delivery practices. Discuss development workflows, testing strategies, build and release processes, and tooling you use to maintain quality and velocity. For each area, explain the scale and complexity of the systems you worked on, the architectural patterns and design choices you applied, and the performance and reliability trade offs you considered. Give concrete examples of technical challenges you solved with hands on verification details when appropriate such as game engine or platform specifics, and quantify measurable business impact using metrics such as latency reduction, cost savings, increased throughput, improved uptime, or faster time to market. At senior levels emphasize mastery in three to four core technology areas, the complexity and ownership of systems you managed, the scalability and reliability problems you solved, and examples where you led architecture or major technical decisions. Align your examples to the role and product domain to establish relevance, and be honest about gaps and areas you are actively developing.
HardSystem Design
80 practiced
Design a globally-distributed, highly-available key-value store to serve 10M reads/sec and 1M writes/sec storing 50PB of data, with a 99.99% read availability SLA and acceptable eventual consistency for writes. Describe sharding, replication, compaction/compaction strategy, read paths, failure modes, and how you would monitor and scale the system. Include choices for storage engine and why.
Sample Answer
Requirements & constraints (clarifying): 10M reads/s, 1M writes/s global, 50PB logical data, 99.99% read availability SLA, eventual consistency acceptable for writes. Target SRE concerns: operational simplicity, predictable tail latency, rapid failure recovery, low human ops.High-level design- Global active-active by region. Each region has a full set of shards (local replicas) to serve reads with single-region latency and availability. Global traffic fronted by DNS + GSLB and regional ingress LB; clients prefer nearest region.Sharding- Consistent-hash-based logical shards (vnodes). Start with ~10k-50k vnodes to enable even distribution and low-rebalance cost.- Map key -> vnode -> shard leader. Leaders are per-shard for write sequencing; any replica can serve reads.- Re-sharding: move vnodes between physical nodes; use background streaming of SST/WAL segments to avoid long downtime.Replication & consistency- Replication factor: 3 (in-region replication across AZs); for geo-redundancy, one replica per of 3 regions (or RF=5 with 3 sync in-region, 2 async cross-region). Because writes can be eventual, choose leader sync to local AZ replicas and async cross-region replication to keep reads local and availability high.- Write flow: client -> regional LB -> shard leader -> write-ahead log (fsync) -> replicate to local follower replicas (sync quorum) -> ack to client. Async tailing to remote region replicas.- Read flow: client -> local region replica (prefer follower if up-to-date enough). Use leader/follower read selection policies and staleness bounds. Support read-after-write by redirecting reads to leader or reading with causal token when required.Storage engine & why- Use an LSM-based engine like RocksDB (or a tuned fork) on NVMe SSDs: - RocksDB handles high write throughput, compaction strategies, SSTables, bloom filters for fast key checks. - Good blkt-level IO patterns and mature tooling.- Tier cold data into object storage (S3-like) using blob offload + metadata in RocksDB; use erasure coding for cold-tier to reduce footprint.- Hot storage on NVMe for low tail latency, cold on HDD-backed object store.Compaction strategy- Use leveled compaction for predictable read amplification and low tail latency on hot ranges; tune L0/L1 thresholds.- Compaction throttling and IO-scheduling to keep P99 latency under SLA. Prioritize compaction on high-read hot shards.- Incremental/manual compaction windows during off-peak; adaptive compaction rate dependent on CPU/IO load.- For large SSTs, use parallel multi-threaded compaction and avoid large monolithic compactions by splitting key ranges.- Tombstone handling: expedited compaction/GC for heavy-deleted ranges; background anti-entropy compaction across replicas to reconcile deletes.Read path & optimizations- Reads served from in-memory block cache + bloom filters to avoid IO. Cache partitioned per-shard and per-node.- Use short-lived local LRU caches and a distributed cache (optional) for small hot keys.- Read repair: if follower returns stale, schedule async repair or on-demand leader fetch depending on staleness tolerance.- Provide options: strongly consistent read (go to leader), bounded-staleness read, or fastest-local read (best-effort eventual).Failure modes & mitigation- Node crash: vnodes auto-failover to replicas; consistent hashing + vnode reassignment; background rebuild from snapshots + WAL.- Disk failure: immediate mark node read-only, redirect reads; rebuild replica on new node using incremental SST/WAL streaming.- Network partition/region outage: local reads continue from remaining region replicas. For cross-region partitions, accept writes locally (since eventual) but record vector clocks/causal metadata to reconcile.- Compaction/backpressure overload: if compaction backlog exceeds threshold, apply backpressure on writes via rate-limiter or spill to cloud storage; alert SRE.- Split-brain: avoid by using small sync-quorum for commit and consensus (Raft) per shard for leader election; use monotonic timestamps and anti-entropy for converging divergent writes.Scaling- Scale storage horizontally: add nodes and reassign vnodes; background streaming of SSTables.- Scale reads by adding replicas in-region; scale writes by splitting hot vnodes (rehash or range-split).- Use autoscaling policies based on disk utilization, IOPS, CPU, and compaction backlog metrics.- Use tiering to keep active working set on NVMe; cold-offload reduces prime storage need.Monitoring, SLOs, and operational practices- Key SLIs: read success rate, read P50/P95/P99/P999 latency, write commit latency, replication lag, compaction backlog, disk IOPS/latency, CPU, GC, disk full %, recovery time for node/region.- Alerts: read availability drop, P99 latency breaches, replication lag > threshold, compaction backlog > threshold, disk nearing capacity, sustained high WAL replay time.- Synthetic tests: canary reads/writes, cross-region health probes, and chaos tests (node kill, AZ down).- Observability: per-shard metrics, distributed tracing for tail latency, detailed logs for rebalances/compactions, anomaly detection.- Runbooks & automation: automated failover, automated rebuilds, throttled rolling upgrades, capacity expansion automation, and playbooks for region outage.Operational optimizations & tradeoffs- Use RF=3 for most active data for cost; for critical datasets use more replicas or cross-region sync.- Use erasure coding for cold tier to reduce storage cost at expense of repair complexity/disk IO.- RocksDB gives good throughput but requires careful compaction tuning and monitoring; alternative (e.g., a OLTP KV store) evaluated if range-queries dominate.- Prioritize read availability: serve reads locally from replicas, accept eventual writes, and design fast anti-entropy to meet convergence targets.Quantitative sizing (back-of-envelope)- 10M reads/s across 3 regions => ~3.3M r/s per region. With 1000 nodes per region => ~3.3k r/s per node; pick nodes sized with NVMe and CPU for that IO. 50PB logical at RF=3 => ~150PB raw; choose sharding and tiering to keep hot working set <20% on NVMe.This design emphasizes local-read availability, predictable tail latency (via compaction tuning and caching), safe leader-based writes with async geo-replication, automated scaling, and SRE-oriented monitoring/runbooks to meet 99.99% read SLA while keeping operational complexity manageable.
MediumTechnical
96 practiced
Compare blue/green, rolling, and canary deployment strategies. For a critical payment service with strict latency SLAs and a database-backed state, recommend a deployment strategy, explain rollback mechanics, how to minimize customer exposure to regressions, and what telemetry you would collect during rollout to make automated promote/revert decisions.
Sample Answer
Blue/green, rolling, and canary — short comparison:- Blue/Green: Two identical environments (blue=prod, green=new). Switch traffic atomically (load balancer/DNS). Fast rollback (switch back). Requires double capacity and careful DB schema compatibility.- Rolling: Gradually replace instances in-place (batches). No extra capacity needed; simpler for stateless services. Rollback slower (need to redeploy previous binary across batches).- Canary: Subset of traffic/users routed to new version (percentage-based), observe metrics, then increase. Combines gradual exposure with control; requires traffic steering and telemetry.Recommendation for critical payment service with strict latency SLAs and DB-backed state:- Use a hybrid: Blue/green for service binaries + canary for traffic routing while ensuring DB schema compatibility. Rationale: Blue/green gives atomic cutover and quick rollback path; canary reduces blast radius and validates real traffic behavior against latency SLAs. Because DB is stateful, enforce backward/forward compatible migrations (expand-then-contract).Rollback mechanics:- If canary metrics fail, automated: stop increasing canary, shift canary traffic back to baseline. If issue detected post-cutover in blue/green, flip traffic to previous environment instantly. Ensure DB migrations are reversible or performed in safe phases (expand first, deploy code, backfill, then remove).Minimize customer exposure:- Route a small % of traffic (1–5%) for initial canaries; target internal users or low-value accounts first.- Use feature flags to disable risky features server-side without redeploy.- Run canaries during low-traffic windows and limit payment types initially.- Ensure retries, idempotency keys, and circuit breakers are in place.Telemetry to collect (for automated promote/revert):- Safety/functional: error rate (transaction failures), payment decline rate, 5xx/4xx counts, business metrics (successful payments/min).- Performance: P99/P95/P90 latency, queue lengths, DB query latency, tail latencies.- Resource: CPU, memory, connection pool saturation, DB locks, replication lag.- User-impact: drop-offs, retry rates, reconciliation mismatches.- Health signals: circuit breaker tripped, exceptions per endpoint, rate of database transaction conflicts.Decision logic (automated):- Promote when canary (vs baseline) shows no regressions: error delta < threshold (e.g., +0.1%), latency P99 within SLA margin, resource usage stable for N minutes.- Revert when any critical signal breaches thresholds (errors, P99 latency, DB replication lag) or when anomaly detector flags statistical divergence.- Use rolling windows, Bayesian/relative risk comparisons, and require sustained good behavior (e.g., 30–60 minutes) before full promotion.Operational practices:- Automated runbooks and playbooks for manual override.- Smoke tests hitting key payment flows in canary.- Alerting and dashboards exposing canary vs baseline side-by-side.- Practice rollbacks in chaos drills; verify DB migration reversibility before deployment.
HardTechnical
96 practiced
You're choosing storage for an analytics pipeline that will ingest event streams and support ad-hoc SQL analytics and near-real-time dashboards. Compare using: (A) object storage + batch processing (e.g., S3 + Spark + Parquet), (B) a managed data warehouse (BigQuery/Redshift), and (C) a streaming OLAP engine (Druid/ClickHouse). For each option evaluate ingestion latency, query performance, cost model, operational burden, and best-fit use cases.
Sample Answer
Clarifying constraints: event streams, ad-hoc SQL, near‑real‑time dashboards. I’ll compare (A) object storage + batch (S3+Spark+Parquet), (B) managed DW (BigQuery/Redshift), (C) streaming OLAP (Druid/ClickHouse) across ingestion latency, query performance, cost model, operational burden, and best-fit use cases.(A) Object storage + batch (S3 + Spark + Parquet)- Ingestion latency: High (minutes→hours). Typically micro-batch or scheduled jobs write Parquet.- Query performance: Good for large scans and complex ETL; ad‑hoc SQL requires compute cluster (Spark/Presto) and may have higher query latency for interactive dashboards.- Cost model: Low storage cost; compute-heavy jobs mean intermittent high cost when clusters run. Good for cold or large-volume historical analytics.- Operational burden: Medium–high: manage job scheduling, cluster autoscaling, schema evolution, partitioning, vacuum/compaction.- Best-fit: Backfill, epochal batch analytics, heavy ETL, cost-sensitive long-term storage rather than sub-minute dashboards.(B) Managed data warehouse (BigQuery/Redshift)- Ingestion latency: Low–medium. Streaming inserts available (seconds), but clustered ingestion and table availability may be seconds–tens of seconds.- Query performance: Excellent for ad‑hoc SQL and complex analytical queries with mature optimizer; interactive dashboards respond well (seconds).- Cost model: Managed compute + storage pricing. BigQuery: serverless pay-per-query (scan bytes) good for spiky workloads; Redshift: reserved/cluster cost favors steady load.- Operational burden: Low: minimal infra ops, automated scaling (BigQuery), backups, and maintenance handled by provider; still need partitioning, cost governance, and schema design.- Best-fit: Interactive BI, ad‑hoc exploration, teams that want low ops and predictable SLAs.(C) Streaming OLAP (Druid / ClickHouse)- Ingestion latency: Very low (sub-second → seconds) with native streaming ingestion.- Query performance: Excellent for low-latency slice-and-dice, timeseries, top-k, group-by; tuned for high-concurrency dashboards.- Cost model: Storage + continuously running compute. Self-hosted clusters incur steady infra and operator costs; cloud managed options reduce ops but cost more.- Operational burden: High if self-hosted: capacity planning, compaction, segment management, node failover, real-time vs batch indices. Requires expertise to tune.- Best-fit: Near-real-time dashboards, high-concurrency streaming analytics, alerting, SLA-critical observability pipelines.Trade-offs and SRE concerns:- If SLOs require sub-5s freshness and high concurrency, choose streaming OLAP but budget for ops and monitoring (autoscaling, compaction, data retention).- If minimal ops and flexible ad‑hoc SQL with variable load, managed DW (BigQuery) minimizes toil and gives predictable performance; watch query cost and schema design.- If primary need is low-cost long-term storage and heavy batch ETL, object storage + batch is cheapest but unsuitable for low-latency dashboards.Recommendation (SRE lens): For mixed requirements, consider a hybrid: stream events into a streaming OLAP for real‑time dashboards and alerts, and also write to object storage (Parquet) for cheap historical retention and large-scale reprocessing; use managed DW for ad‑hoc BI where teams prefer SQL and low operational burden.
EasyTechnical
94 practiced
Explain Docker image vs container, how layering and copy-on-write work, and the operational implications for SREs (image size, CVE scanning, startup time). Include how you troubleshoot slow container start times and what runtime/kernel features (overlayfs, cgroups, seccomp) are relevant.
Sample Answer
An image is a read-only, versioned artifact (stack of layers + metadata) that describes a filesystem and how to run a process. A container is a runtime instance of an image: it combines the image’s read-only layers with a writable thin layer and isolated namespaces, cgroups, and process state.Layering & copy-on-write (COW)- Images are built as immutable layers (each layer is a diff). When you run a container, the runtime mounts those layers (often via overlayfs) and provides a top writable layer. On writes, COW copies only the changed files into the writable layer, avoiding full copies and saving space.Operational implications for SREs- Image size: larger images increase network transfer (pulls), disk usage, and potentially startup time. Prefer minimal base images, multi-stage builds, and prune old images.- CVE scanning: scan images for vulnerabilities (CI pipeline + registry scanning). Smaller, well-audited images reduce attack surface; pin base images and run regular rebuilds.- Startup time: influenced by image pull time, unpacking, and init costs. Smaller compressed layers and registry proximity (caching) help.Troubleshooting slow container start1. Measure where time is spent: image pull vs unpack vs runtime creation. Use docker/podman events, containerd ctr, kubelet logs, and registry metrics.2. If pull is slow: check network, registry throughput, use image layers caching, local registry mirror, or pre-pulled DaemonSet.3. If unpacking/overlayfs is slow: inspect storage driver, disk I/O, inode pressure, and file counts (many small files slow overlay). Consider squashing layers, combining small files, or using faster disks.4. If runtime init slow: check entrypoint, healthchecks, DNS wait, or app warm-up. Profile startup commands and add readiness probes.5. Check node resource contention: CPU, IOPS, and cgroup throttling.Relevant runtime/kernel features- overlayfs: implements layered mounts/COW—critical for performance; watch for kernel bugs and inode limits.- cgroups: enforce CPU/memory/io limits and can cause throttling; tune limits to avoid OOMs or throttling during startup.- seccomp: syscall filtering for security; overly strict profiles can block needed syscalls and cause failures.- namespaces: provide isolation (pid/net/mount/user).- user namespaces and capabilities: reduce privilege surface.- io_uring/blk-iopoll (kernel features) and storage driver choices (overlay2 vs aufs) can affect performance.Best practices- Keep images small and immutable; build reproducibly.- Scan images early and often; automate rebuilds for patched bases.- Use registry mirrors and pre-pull for critical services.- Monitor node resource utilization and tune cgroups; enable appropriate seccomp profiles that balance security with functionality.
EasyTechnical
106 practiced
Explain how a hash table works (basic structure), collision-handling strategies (chaining vs open addressing), resize policies, average/worst-case time complexities, and memory trade-offs. Then describe how you'd use a hash table to deduplicate streaming logs at SRE scale and what pitfalls you must avoid (cardinality, memory blowup, eviction).
Sample Answer
A hash table maps keys to buckets using a hash function; internally it stores (key, value) pairs in an array indexed by hash(key) mod capacity. Collisions occur when different keys map to the same bucket.Collision-handling:- Chaining: each bucket holds a linked list or dynamic array of entries. Insert/search traverse the chain. Simple, good when load factor >1, easier to resize.- Open addressing: store entries directly in the array and resolve collisions by probing (linear, quadratic, or double hashing). No extra pointers; requires careful handling of deletions (tombstones) and load factor <1.Resize policies:- Grow (and sometimes shrink) when load factor crosses thresholds (e.g., grow when load >0.7). Resize rehashes all entries—expensive but amortized cost is acceptable if done exponentially (double capacity).- Incremental or background rehashing reduces pause impact.Time/space:- Average: O(1) insert/lookup/delete if hash function is good.- Worst-case: O(n) for chaining (all keys in one bucket) or probing (clustered), if adversarial hashes or poor resizing.- Memory trade-offs: chaining uses extra pointers for chains; open addressing uses denser arrays but requires slack for low load factor. Larger capacity reduces collisions but increases memory.SRE-scale streaming log deduplication (practical approaches and pitfalls):- Naive in-memory hash set: fast but vulnerable to cardinality blowup. Use only for bounded windows.- Use time-windowing + eviction: keep keys for a sliding window (TTL), evict by timestamp (queue + hashmap) to bound memory.- Approximate structures: Bloom filters (or counting Bloom filters) drastically reduce memory at cost of false positives (acceptable for dropping duplicates) but no deletions unless counting BF used.- External stores: Redis with TTLs or RocksDB can offload memory; partition keys across instances to scale.- Hybrid: Bloom filter to quickly drop most duplicates, and a backing store for exactness for a short window.Pitfalls to avoid:- Unbounded cardinality → OOM. Always cap retention window or use approximate structures.- Eviction semantics: deleting too early causes false negatives (missed duplicate detection); deleting too late wastes memory.- Hash collisions and adversarial input: use a strong non-cryptographic hash (e.g., xxHash) or keyed hash to avoid DoS.- Hot-key skew: partitioning and rate-limiting needed.- Rehash pauses: perform background/incremental rehashing for production.Monitoring and metrics: track cardinality, memory usage, filter false-positive rate, eviction rate, and OOM alerts. Test with production-like traffic to tune TTLs, filter sizes, and partitioning.
Unlock Full Question Bank
Get access to hundreds of Technical Background and Skills interview questions and detailed answers.
