Data Infrastructure Technology Selection Questions
Deep understanding of specific technologies relevant to complex system design. Master databases (PostgreSQL, Cassandra, DynamoDB, Elasticsearch), message queues (Kafka, RabbitMQ), caching systems (Redis), search engines, and frameworks. Understand their strengths, weaknesses, trade-offs, operational characteristics, scaling patterns, and common pitfalls. Be able to justify technology choices based on specific system requirements.
HardSystem Design
40 practiced
Design an automated online schema migration system that supports rolling schema changes, background backfills, feature-flag gating, automatic safety checks, reversible migrations, and observability. Describe orchestration, per-service compatibility checks, migration state storage, and how to ensure safe rollbacks.
Sample Answer
Requirements:- Safe, automated online schema migrations with zero/low downtime- Rolling schema changes + background backfills- Feature-flag gating and reversible migrations- Automatic safety checks and observability- Per-service compatibility verification and safe rollbackHigh-level architecture:- Migration Orchestrator (central service)- Worker fleet (executors) for DDL, backfills, checks- Per-service Compatibility Service (lightweight agents)- Migration State Store (immutable events + current state)- Feature-flag integration (FF service)- Monitoring & Alerting (metrics, traces, audit logs)Orchestration:- User submits a migration bundle (DDL + backfill job + companion code + safety checks + rollback plan).- Orchestrator validates bundle, assigns unique migration-id, writes an append-only event to Migration State Store.- Phased plan: validate → schema deploy (compatible form) → backfill in background → swap reads/writes behind feature-flag → cleanup.- Orchestrator drives state machine, retries idempotently, and enforces rate limits and concurrency controls.Per-service compatibility checks:- Agents run “compatibility probes” against service versions (static analysis of client code, runtime API checks).- Contract tests: ensure new schema doesn’t break queries (SELECT/INSERT shape), using sampled traffic replay in a sandbox.- Orchestrator requires explicit approval from failing/non-compatible services or auto-schedules rolling deploys with canary gates.Migration state storage:- Use an append-only event store (e.g., distributed log like Kafka or durable DB) + materialized view for current state.- Store: migration-id, author, bundle checksum, phased state, per-target progress, safety-check results, timestamps, audit trail.- Immutable events enable audit and deterministic rollback.Background backfills:- Implement idempotent backfill tasks partitioned by key ranges; workers process in parallel with rate limiting and priority to foreground traffic.- Backfills emit progress and error metrics; orchestrator can pause/resume based on SLOs (latency, CPU, replication lag).Automatic safety checks:- Static checks: forbid unsafe DDL (e.g., full table rewrite without using online techniques), check indexes, FK impacts.- Dynamic checks: pre/post data invariants, row counts change thresholds, replication lag, query latency and error rates.- Chaos/resilience checks: run migration in shadow cluster with production snapshot for high-risk changes.Feature-flag gating & reversible migrations:- Changes designed with “expand-contract” pattern: e.g., add nullable column or new index (expand), deploy code to use it behind feature flag, backfill, then make column non-null/cleanup (contract).- Feature flag toggles controlled by orchestrator; flags annotated in migration bundle to ensure safe progression.- Rollback strategy: for each phase define immediate-safe vs eventual-safe rollback. Orchestrator supports: - Fast rollback: flip feature flag off, switch reads/writes back, stop backfills. - Full schema rollback: reverse DDL if safe; otherwise mark as degraded and retain compatibility layer (views, triggers) until services update.- All rollbacks are run through same state machine and safety checks; require automated canary rollback and manual approval for destructive reversions.Ensuring safe rollbacks:- Keep migrations reversible where possible (store inverse operations with checks).- Maintain compatibility layers (views, virtual columns, shadow writes) so old code continues while rollback occurs.- Use staging/canary: apply to small subset, monitor metrics plus custom invariants, then expand. On anomalies, orchestrator triggers automated rollback steps.- Pre-flight “dry-run” in replay cluster reduces surprises.- Enforce kill-switch thresholds (replication lag > X, error rate > Y) that automatically pause and initiate rollback.Observability & Alerting:- Expose metrics: per-migration progress, backfill throughput, latency, error rates, replication lag, service compatibility status.- Detailed logs, traces, and audit trail per migration-id.- Dashboards and alerts integrated with runbooks; SLO guardrails cause automatic throttling or halt.Scalability & reliability:- Stateless orchestrator with leader-election; workers autoscale.- State store replicated and backed up; events immutable for audit and safe replays.- Rate-limited parallelism to avoid overload; backoff on hotspots.Trade-offs:- Complexity vs safety: expand-contract + feature flags increase steps but reduce risk.- Full automated DDL rollback is not always possible; require human-in-loop for destructive reversions.- Shadow/replay infrastructure cost vs reduced risk — justify for high-impact systems.This design gives SREs automated, observable, and reversible migrations with per-service safety checks and clear rollback paths while protecting availability and SLOs.
HardTechnical
44 practiced
You receive frequent Under-Replicated Partitions (URP) alerts in Kafka production. Create an incident response plan: immediate triage steps, temporary mitigations to stop alert storm, root cause checklist (disk, GC, network, broker restarts), and longer-term fixes to prevent ISR thrashing and frequent URPs.
Sample Answer
Immediate triage (first 0–15 min)- Acknowledge incident, set incident channel and owner.- Snapshot current state: count of URPs, impacted topics/partitions, alert timestamp histogram.- Check cluster-wide health quickly: - Kafka metrics: UnderReplicatedPartitions, OfflinePartitionsCount, LeaderElectionRate, BrokerUp. - Zookeeper/Controller status. - Broker host health: CPU, load, disk free/inodes, network errors, GC pause metrics.- If customer impact unknown, query consumer lag for affected partitions and SLO pages.Temporary mitigations to stop alert storm (15–60 min)- Silence noisy alerts with TTL (not resolve) while you investigate to prevent paging.- If a small set of brokers is flapping ISR, mark them maintenance or ACL them from leadership election: set broker.exclude from controller or use preferred replica election off.- Increase controller.quorum.voting/replica.lag.thresholds? (short-term, cautious) — prefer silencing alerts over changing thresholds broadly.- Throttle replica fetchers or reduce replication traffic temporarily (network congestion case).- If broker JVM is GC-thrashing, restart that broker gracefully (see restart checklist) or trigger controlled rolling restart with rack-awareness.Root-cause checklist (investigate in parallel)- Disk: - Disk full, high utilization, high iowait, read-only mounts. - OS dmesg for I/O errors.- JVM/GC: - GC pause logs, long young/old pauses, OutOfMemory signs. - Heap usage vs configured, metamemory pressure.- Network: - Packet loss, interface errors, NIC counters, MTU mismatch, switch flaps. - Cross-check inter-broker and leader-to-follower replication latency.- Broker process: - Broker logs (replication errors, connection refused, replica not available), thread dumps. - Broker process restarts/flapping history.- Controller/ZK: - Controller leader flaps, ZK latency, session expirations.- Configuration drift: - Recent changes (ops/deploys/rolling restarts, JVM flags, OS kernels, disk drivers).- Resource saturation: - CPU steal (noisy neighbors), container limits, cgroups throttling.- Security: - TLS/SASL failures causing connection drops.Broker restart checklist (if needed)- Choose one broker at a time, avoid simultaneous restarts.- Decommission leadership: preferPreferredReplicaElection=false; use partition reassignment or setbroker to not be leader if possible.- Ensure ISR caught up or set replica.fetch.max.bytes appropriately.- Graceful stop: kafka-server-stop.sh, wait for clean disconnect.- Validate post-restart: metrics, GC, replication resumes.Longer-term fixes to prevent ISR thrashing and frequent URPs- Capacity and resource tuning: - Right-size broker JVM, enable G1GC with tuned pauses; monitor GC and tune heap. - Provision disks with headroom; use NVMe or separate logs disk; monitor iostat. - Network: increase replication bandwidth, use dedicated replication NICs or QoS.- Broker config hardening: - Increase replica.lag.time.max.ms and replica.lag.max.messages conservatively to reduce false positives. - Tune replica.fetch.max.bytes, replica.fetch.wait.max.ms, num.replica.fetchers. - Enable rack awareness and configuraition to spread leaders.- Operational: - Automated health checks and graceful rolling-restart runbooks. - Pre-emptive autoscaling or capacity alerts (disk, NIC, GC). - Canary config rollouts and config drift detection.- Observability: - Rich dashboards: per-topic ISR size, per-broker replication lag, GC, disk IO, network. - Correlate events: controller elections, broker restarts, network flaps. - Retain detailed GC and thread-dump logs for postmortem.- Chaos/testing: - Run controlled chaos tests (terminate brokers, network partitions) to validate recovery.- Process: - Post-incident RCA, identify corrective actions, track as tickets, and update runbooks and alert thresholds based on learned behavior.Why this helps- Immediate steps protect availability and reduce noise. Parallel checklist ensures focused root-cause discovery (disk, GC, network, controller). Short-term mitigations buy time; long-term fixes reduce recurrence by addressing capacity, configuration, and observability.
MediumTechnical
55 practiced
You must size a Kafka cluster for ingesting 1 GB/s of messages (average message 1 KB), retention 7 days, and replication factor 3. Describe how many brokers you'd provision, disk capacity per broker, network bandwidth, partition counts, and other considerations (controller capacity, JVM tuning). State assumptions and how you'd validate them.
Sample Answer
Assumptions- 1 GB/s ingest = 1,000 MB/s ≈ 1,000,000 messages/sec at 1 KB each.- 7 days retention (24*7 = 604,800 sec) -> raw data = 1 GB/s * 604,800 = 604,800 GB ≈ 605 TB.- Replication factor (RF) = 3 -> replicated storage ≈ 605 TB * 3 = 1,815 TB (~1.8 PB).- Multi-AZ deployment, rack-aware replication, steady-state steady ingest (plan for 25–50% headroom for spikes).Storage / Broker count- Goal: keep per-broker usable disk < ~60–80 TB (for manageability, rebuild times).- If we pick 36 brokers: per-broker replicated storage = 1,815 TB / 36 ≈ 50.4 TB.- Add 30% headroom for compaction, segments, and rebuilds -> provision ~66 TB per broker.- Recommendation: 36 brokers with 80 TB raw disks each (use RAID or JBOD per Kafka best practice; allow OS filesystem and logs), leaving ~66 TB for Kafka data after OS and safety margins.- If you prefer larger clusters with faster recovery, split into 48 brokers -> ~38 TB used per broker + headroom.Network- Ingest 1 GB/s from producers -> cluster ingress = 1 Gbps sustained (note: 1 GB/s = 8 Gbit/s). — Correction: 1 GB/s = 8 Gbit/s. (Important for NIC selection.)- Replication traffic: for RF=3, two extra copies mean ~2 GB/s inter-broker write traffic (16 Gbit/s) if producers write only once; actually replication can be pipelined and leader forwards to followers, so network needs to handle leader-to-follower peak flows.- Total cluster network egress+ingest ≈ 3 GB/s ~ 24 Gbit/s peak.- Per-broker NIC: use 25 Gbps NICs minimum (10 Gbps too tight for replication and peaks); 25 Gbps recommended, 40/100 Gbps if aggregating multiple roles or co-locating other services.Partition count and throughput- Partitions provide parallelism. Estimate per-partition sustained throughput: conservative 20–100 MB/s depending on disk and OS page cache. Using conservative 50 MB/s per partition leads to 20 partitions to cover 1 GB/s; but we must distribute load and support consumer parallelism and leader distribution.- Recommended total partitions: 2,000–3,000 partitions (gives parallelism and safe per-partition throughput). With 36 brokers, that's ~55–85 partitions per broker.- Ensure even leader distribution; use preferred leader balancing.Controller and broker sizing / CPU / RAM- Brokers: 16–32 vCPUs, 64–128 GB RAM. Keep heap moderate (see JVM tuning).- Controller nodes: use 3 dedicated controller replicas (or KRaft controllers) if possible to isolate controller load; controllers need CPU and good network but less disk.- Partition count impacts controller load: many partitions -> more metadata -> higher CPU and GC pressure on controller and brokers.JVM & OS tuning- JVM: use G1GC, avoid very large heaps. Heap = 12–32 GB depending on metadata load. Example: 32 vCPU / 64-128 GB host -> set -Xmx=24-32g. Keep off-heap memory for page cache.- G1GC flags: enable pause target (e.g., -XX:MaxGCPauseMillis=200), use -XX:+UseG1GC, tune region sizes if needed.- File descriptors: increase to >100k.- OS: vm.swappiness=1, ensure direct IO where appropriate, disable transparent hugepages, set net.core.somaxconn, tcp_keepalive and net.ipv4.tcp_tw_reuse for high connections if needed.- Disk: use NVMe SSDs for log dirs; ensure high sequential write IOPS and throughput. Use multiple disks or large NVMe to get sustained 1 GB/s cluster-wide writes.Other considerations- Replica placement: rack awareness to tolerate AZ failures.- Recovery: with large disks, re-replication can saturate network; consider throttle settings for replica fetch (replica.fetch.max.bytes, replication.quota.window.num, controller.replication.throttles).- Quotas: configure client and replication quotas to prevent producer from overwhelming cluster.- Monitoring & alerts: track BytesIn/Out, UnderReplicatedPartitions, ISR shrink/grow rate, request latency, IO saturation, GC pause times, disk utilization per broker.- Backups: consider tiered storage or cold storage if 7-day retention expensive at this scale.Validation plan1. Microbenchmarks: use kafka-producer-perf-test and kafka-producer-tools to ramp to 1 GB/s of realistic message sizes and key distribution.2. End-to-end tests: producers -> cluster -> consumers with target consumer groups to validate consumer throughput and latency.3. Failure tests: kill a broker, simulate AZ failure, measure recovery and availability; validate under-replicated partitions and replication throttle behavior.4. Long-run soak test: run sustained load for days to verify retention, disk growth, log rolling, GC, and compaction behavior.5. Observability: ensure dashboards for disk, network, GC, request metrics; define SLOs (latency, availability) and alert thresholds.How to iterate- Start with a pilot cluster (e.g., 6–12 brokers) to validate per-broker throughput assumptions on chosen hardware.- Measure per-partition throughput, disk write amplification, replication traffic, and recovery times; extrapolate to full cluster sizing and adjust broker count, partitions, and disk size.Summary recommendation (practical)- Provision ~36 brokers, each with 25 Gbps NIC, 16–32 vCPU, 64–128 GB RAM, ~80 TB raw NVMe disk (≈66 TB usable for Kafka).- Total partitions: ~2,000–3,000.- Use G1GC with heap 24–32 GB, 3 dedicated controllers (or KRaft), monitor and validate via load tests and failure injection.
MediumSystem Design
54 practiced
Design Elasticsearch index mapping and shard sizing for an application with 100M documents (avg 2 KB/doc) supporting full-text search and aggregations. Decide number of shards, shard size target, mapping choices (types, analyzers), and index lifecycle management (ILM) for retention and hot/warm/cold tiers.
Sample Answer
Requirements & assumptions:- 100M docs × ~2 KB = ~200 GB of raw data. Expect Lucene/segment overhead, doc_values, replicas, and future growth → plan for ~2× overhead => ~400–500 GB total storage.- Needs full‑text search + aggregations (fast searches, efficient aggs).Shard sizing & number:- Target shard size: 30–50 GB (sweet spot for search/merge performance and recoverability).- Primaries required = ceil(estimated primary storage / target_shard_size). - Primary storage ≈ 200 GB → use target 40 GB → 200/40 = 5 → choose 6 primary shards for wiggle room and easier scaling.- Replication: replica factor 1 (one replica) for availability → total shards = 12, total storage ≈ 400 GB.- Justification: 30–50 GB balances recovery time, heap pressure, and query parallelism. Start with 6 primaries; reindex to change shard count later if needed.Mapping choices:- Disable _all; explicitly map fields.- Text fields used for full‑text: use language‑aware analyzer (e.g., "english" or custom with lowercase, stopwords, asciifolding, stemming). Use multi-fields for exact-match/sorting: - "title": { "type":"text", "analyzer":"english", "fields": { "keyword": {"type":"keyword","ignore_above":256} } }- Keyword fields for aggregations and filters (e.g., ids, enums). Prefer doc_values (default) for aggregations.- Numeric/date types: use long/date/keyword appropriately for efficient range queries and aggs.- Nested vs object: use nested only if you need per-array-element queries/agg correctness; otherwise use flattened/object to save space.- Avoid storing large text in both text and keyword; use multifield.- Use doc_values true for fields used in aggs/sorts; disable _source on large rarely‑needed fields or store selectively.- Consider runtime fields for ad‑hoc transformations but avoid for heavy query load.Analyzers & indexing:- Custom analyzer: lowercase + asciifolding + stopwords + porter/english stemmer.- Use edge_ngram on a dedicated "autocomplete" subfield, not on primary analyzed field (to keep index size small).- Use index.time_series or rollover physical indices for scaling if write pattern is time-based.ILM (hot/warm/cold) & retention:- Hot phase (0–7/30 days): fast nodes (NVMe, more CPU/heap), replicas 1, frequent refresh (default), no force_merge; keep recent data for low latency.- Warm phase (7–90 days): move to warm nodes (cheaper storage, less CPU), reduce replicas to 0 or keep 1 depending on RPO, set number_of_replicas=0 to save space if acceptable, allow segment merges; optionally shrink to fewer shards (use _shrink to reduce shard count) to improve query efficiency.- Cold phase (90–365 days): move to cold nodes (spinning disk), set replicas=0, freeze index or set priority low; use force_merge to 1 segment before moving to cold to reduce segment overhead.- Delete phase: retention policy delete_index after N days (e.g., 365 days) or move to long term store (backup snapshots) for compliance.- Use ILM rollover: create write alias, rollover when primary shard size ~40 GB or age threshold hit.Operational considerations:- Monitor shard sizes, query latency, merge times, and JVM heap. Use Prometheus/Grafana, Elasticsearch X‑Pack or OpenSearch metrics.- Reindex plan: to change analyzers/shard count, reindex into new index with zero‑downtime alias swap.- Test performance with realistic dataset; adjust shard count if query concurrency or growth demands it.- Plan capacity: nodes sized so that heap ≤ 32 GB (use G1GC), keep < 50% disk usage per node, reserve headroom for merges.This configuration balances search/aggregation performance, recoverability, and cost while enabling lifecycle automation and future scaling.
EasyTechnical
50 practiced
Explain ACID vs BASE and common distributed consistency models (strong, eventual, causal, read-your-writes). From an SRE perspective, how do these models influence replication setup, failover decisions, client libraries, and runbook actions during network partitions? Map each model to real products (Postgres, Cassandra, DynamoDB) and give operational implications.
Sample Answer
ACID vs BASE (brief)- ACID (Atomicity, Consistency, Isolation, Durability): strong transactional guarantees within a single node or tightly coordinated cluster. Good for correctness (e.g., serializability), but often limits availability/latency across partitions.- BASE (Basically Available, Soft state, Eventual consistency): accepts temporary inconsistency for higher availability and partition tolerance; converges over time.Common distributed consistency models- Strong consistency: reads reflect the latest committed write (linearizable). Example: single-master Postgres (or distributed systems using Paxos/Raft). Operationally you prioritize synchronous replication, failover that preserves a single leader, and minimize split-brain risk.- Eventual consistency: updates propagate asynchronously; reads may be stale but system converges. Example: Dynamo-style systems (Cassandra, DynamoDB default). Operationally use asynchronous replication, prioritize availability; expect reconciliation/compaction; monitor anti-entropy.- Causal consistency: preserves causality (if A caused B, reads reflect that). Middle ground; fewer implementations in mainstream DBs but supported in some client libraries and research systems.- Read-your-writes: a session guarantee ensuring a client sees its own prior writes (common in DynamoDB when using consistent read settings or session tokens).Map to products + SRE implications- Postgres (ACID, strong within a primary-standby via synchronous replication if configured): replication — usually primary with async replicas or synchronous replicas for safety; failover — prefer automated leader election (Patroni/PGAutoFailover) but be conservative to avoid split-brain; client libs — expect transactions, retries must be idempotent; runbook — promote a synced replica, check replication lag, failover only if quorum/consensus achieved.- Cassandra (BASE, eventual consistency, tunable per-request consistency like QUORUM, ONE): replication — multi-datacenter, asynchronous; failover — node replacement with hinted handoff/repair, no single leader; client libs — choose consistency level per operation (QUORUM for stronger guarantees); runbook — run nodetool repair, check hinted handoffs, monitor tombstones and compaction, handle anti-entropy after partition.- DynamoDB (BASE/eventual by default, offers strongly consistent reads optionally, supports conditional writes): replication — fully managed multi-AZ; failover — AWS manages; client libs — use conditional writes and consistent read flags when needed; runbook — less infra ops, but design for retries/backoff and monitor throttling/replication metrics.Operational choices during partitions- Strong: prioritize correctness — reject writes if leader unavailable; runbook: failover only after leader loss confirmed, ensure WAL/fsync integrity, coordinate maintenance windows.- Eventual: prioritize availability — allow writes on different partitions, reconcile later; runbook: monitor divergence, run repairs/anti-entropy, expect higher conflict-resolution overhead.- Causal/read-your-writes: ensure client-session routing to a compatible replica (sticky sessions or tokens); runbook: if session-affine replica fails, re-establish session state or redirect with sync check.Practical SRE recommendations- Define SLOs that map to consistency needs; pick data-store + replication strategy accordingly.- Instrument replication lag, conflict rates, repair backlog; expose metrics for runbook thresholds.- Build client libraries or middleware that allow per-request consistency tuning, idempotency keys, and exponential backoff.- Automate safe failover with health checks and quorum-based decisions; include clear runbook steps for partition detection, promote/rollback, and post-recovery repair tasks (replay, repair, compaction).This framing helps you choose trade-offs and write precise operational runbooks per product and SLA.
Unlock Full Question Bank
Get access to hundreds of Data Infrastructure Technology Selection interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.