Technical Stack and Infrastructure Questions
Addresses the technology stack and platform infrastructure choices that support a product. Topics include cloud architecture, deployment platforms, service and data layer design, observability and monitoring, scalability and performance considerations, tooling for build and release, and assessing technical debt or modernization needs in the stack. Also covers evaluation of BI tools, ETL platforms, and how infrastructure choices affect team velocity and operational cost.
MediumSystem Design
27 practiced
Design a VPC/subnet and load-balancing topology for a highly available service across three AZs. Include public/private subnets, NAT design, internal LBs for service-to-service traffic, and explain route-table and security-group considerations to avoid single points of failure.
Sample Answer
Requirements/constraints (clarify): support HA across 3 AZs, internet-facing endpoints for clients, private app tiers, service-to-service internal load-balancing, no single point of failure, secure least-privilege access, outbound internet for updates.Proposed topology (per VPC):- Use a single VPC (e.g. 10.0.0.0/16) with three AZs (A, B, C).- In each AZ create: - Public subnet (10.0.X.0/24) for ALB, NAT gateways (if colocated), bastion (optional). - Private app subnet (10.0.Y.0/24) for stateless app instances / ECS tasks. - Private data subnet (10.0.Z.0/24) for stateful services (databases, caches) — restrict access.High-availability components:- Internet Gateway attached to VPC for public subnets.- Internet-facing ALB (Application Load Balancer) deployed across all 3 public subnets; health checks to route only to healthy targets.- Internal ALB/NLB (scheme: internal) deployed across all 3 AZs in private app subnets for service-to-service traffic (use ALB for HTTP, NLB for TCP/UDP).- NAT Gateway in each AZ, placed in that AZ's public subnet to provide AZ-local outbound; configure private subnets’ route tables to use their AZ's NAT GW. Having one NAT GW per AZ avoids cross-AZ dependencies and single points of failure.- For S3/other AWS services, create VPC endpoints (gateway or interface) to avoid NAT egress for heavy traffic.Routing:- Public subnets: route 0.0.0.0/0 → IGW.- Private subnets: route 0.0.0.0/0 → NAT-GW in same AZ (route table per AZ or per-subnet).- Internal load balancers and databases remain reachable only via private CIDR; no direct IGW routes.- Use route tables and subnet assignments so if an AZ fails, its subnets and NAT/GW go down but remaining AZs continue serving.Security groups and network ACLs:- ALB (public) SG: allow inbound 80/443 from 0.0.0.0/0; outbound to App SG on app port(s).- App SG: allow inbound only from ALB SG (reference security group) and from Internal-LB SGs for service-to-service; outbound to DB SG on DB ports.- Internal-LB SG: allow inbound from other service SGs (or specific CIDR if needed); outbound to App SG.- DB SG (stateful): allow inbound only from App SG on DB port; no public inbound.- Use SG references (not CIDR) to keep rules explicit and avoid port exposure.- NACLs: keep mostly permissive (stateless) with broad allow for intra-VPC traffic; avoid complex NACLs unless needed—SGs provide primary control.Avoiding single points:- Deploy ALBs and internal LBs across all AZs with cross-zone load balancing enabled (if needed).- NAT gateways per AZ so private subnets don't route cross-AZ for egress; optionally use Elastic IPs per NAT.- Multi-AZ DB (read replicas / cluster) or managed RDS Multi-AZ for DB HA.- Health checks + autoscaling groups per AZ to maintain capacity; distribute ASG minimums across AZs.- Use Route53 with health checks and multi-A record or failover routing for cross-region failover if needed.Operational considerations:- Use SSM Session Manager instead of bastions when possible (removes public host).- Monitor LB target health, NAT costs, AZ-level metrics; alert on AZ loss, unhealthy targets, cross-AZ traffic spikes.- Test failure scenarios: terminate an AZ, simulate NAT loss, remove ALB subnet to validate resilience.This topology gives full redundancy across three AZs, private service-to-service traffic via internal LBs, AZ-local NAT for no single egress point, and security-group based least-privilege access.
HardTechnical
22 practiced
For a global user profile service used by many interactive features, analyze the trade-offs between eventual and strong consistency. Discuss impacts on latency, availability under network partitions (CAP), implementation complexity, user experience, and how SREs should set SLOs in each case.
Sample Answer
Context: a global profile store used in many interactive flows (login, feed personalization, messaging). Choosing eventual vs strong consistency affects UX, ops, and SRE responsibilities.Definitions & CAP trade-off- Strong consistency (linearizability/serializability): reads reflect latest confirmed writes. Under network partitions CAP forces you to sacrifice availability (AP vs CP).- Eventual consistency: writes propagate asynchronously; system remains available during partitions but reads may be stale.Impacts- Latency: Strong consistency often requires cross-datacenter consensus (Paxos/Raft, leader-follower sync) → higher tail latency on reads/writes. Eventual can serve local reads from nearest replica → lower latency.- Availability under partitions: Strong consistency = degrade availability (reject or block ops during partition). Eventual = continue serving reads/writes locally, risking divergence.- Implementation complexity: Strong: distributed consensus, leader re-election, read-repair, linearizable caches. Eventual: conflict resolution, vector clocks/CRDTs, background reconciliation, anti-entropy — testing correctness is harder.- User experience: Strong gives deterministic behaviour (no surprise: profile updates visible immediately) — important for critical flows (billing, auth). Eventual acceptable for non-critical surfaces (recommendations, presence) but can cause confusing UX (wrong name/avatar) if not mitigated.SRE implications & SLOs- For strong consistency services: - SLOs emphasize read/write latency (P50/P95/P99) and availability during partitions (e.g., 99.99% availability excluding planned maintenance). - Error budget should be small for stale-read incidents; alert on leader lag, election rate, quorum loss. - Runbooks: failover procedures, quorum-restoration steps, controlled degraded-mode notifications to clients.- For eventual consistency services: - SLOs should include "staleness" targets (e.g., 99% of reads reflect writes within X seconds) plus availability targets. - Monitor convergence metrics: anti-entropy throughput, conflict rate, percent of reads served from remote vs local. - Error budget spans both availability and data freshness; alerts on divergence beyond threshold.Operational mitigations (both):- Hybrid approach: use strong consistency for critical fields (auth flags, privacy settings) and eventual for noncritical attributes; expose read-repair or read-after-write caches for UX smoothing.- Client strategies: read-your-writes (sticky sessions), optimistic UI with rollback, causal metadata.Recommendation summary- Choose strong when immediate correctness is critical and you can accept higher latency/availability trade-offs. Choose eventual for global low-latency, highly available experiences where temporary staleness is tolerable, with explicit SLOs on propagation time and robust reconciliation.
MediumSystem Design
23 practiced
Design a log ingestion pipeline for 10 TB/day of logs: collectors, buffering layer, short-term indexing, cold storage, and search. Discuss resilience for bursty traffic, backpressure handling, and scaling strategies for both storage and query layers.
Sample Answer
Requirements & constraints:- Ingest 10 TB/day (~115 MB/s sustained; allow bursts 5x), low loss, <1 min indexing for short-term search, cold archive to S3, multi-AZ resilience, support queries (alerts/forensics).High-level architecture:Clients → Edge Collectors (k8s/Autoscaling) → Buffering (Kafka/Managed Kinesis) → Short-term Index (Elasticsearch/Opensearch hot nodes + warm nodes) → Cold Storage (S3 + partitioned Parquet/JSON + Glue/partitioned metadata) → Search API / Query Engine (coordinator + cached results)Components & responsibilities:- Edge Collectors: lightweight agents (Fluentd/Vector) with local disk buffer, TLS, auth, drop/transform rules. Autoscale behind LB. Validate & enrich (metadata, sampling).- Buffering Layer: Kafka cluster with retention hours-days, topic partitioning by tenant/source, replication factor 3. Handles burst absorb and persists for replay.- Short-term Index: Hot nodes (NVMe) for <7 days, warm nodes (HDD) for 7–30 days. Use ILM to roll indices to warm then cold. Index templates optimized (doc-values, minimal analyzers).- Cold Storage: Daily partitioned files to S3 with lifecycle; maintain index snapshots and searchable metadata in a catalog (e.g., Athena).- Query Layer: API layer that fans queries to hot/warm ES; for deeper queries, fallback to S3 via Athena or Presto. Add caching (Redis) for hot dashboards.Resilience for bursty traffic:- Local agent disk buffers prevent immediate loss.- Kafka absorbs bursts; provision partitions and producers withcks backoff.- Autoscaling for collectors and ES hot nodes; pre-warm additional capacity if scheduled events expected.Backpressure handling:- Clients receive HTTP 429 with Retry-After when collectors overwhelmed.- Collectors apply local drop policies (priority, sampling) and circuit-breaker to avoid cascading failures.- Producers to Kafka use async with retries, exponential backoff; monitor lag and trigger scaling/alerts.Scaling strategies:- Storage scaling: Kafka partition rebalancing and tiered storage; ES scale horizontally by adding shards/nodes and using ILM to move older shards to warm nodes; S3 for infinite cold capacity.- Query scaling: Separate query and ingest clusters, add coordinating/query nodes, use query throttling, prioritize SLOs (alerts over ad-hoc). Use read replicas for ES and pre-aggregated indices for dashboards.- Operational: CI/CD for configs, automated rolling upgrades, capacity runbooks, and chaos tests.Observability & SLOs:- Metrics: ingestion rate, Kafka lag, ES indexing rate, query latency, error rates. Alerts on sustained lag, disk pressure, GC.- SLOs: e.g., 99% of logs indexed within 60s, 99.9% write availability. Define error budget and automated mitigation (e.g., reduce retention, enable sampling).Trade-offs:- Near-real-time full-text search costs vs. cheaper S3 cold queries — use hybrid approach.- Managed services (MSK/OpenSearch Service) reduce ops but limit tuning.This design provides durable ingestion, burst resilience, clear backpressure paths, and independent scaling for storage and query layers.
HardTechnical
27 practiced
Design a realistic load and chaos testing strategy for a microservices architecture that must handle 200k RPS. Cover test harness design, traffic generation using realistic user journeys, dependency simulation, throttling/backpressure tests, and safety mechanisms to avoid collateral impact on production.
Sample Answer
Requirements & scope- Target: sustained 200k RPS across microservices with realistic user behavior, plus chaos scenarios (node/network/db failures) and backpressure validation.- Non‑functional targets: p95 latency, error budget, downstream SLOs, cost/BL.High-level strategy1. Test harness- Central orchestration (K8s Job/CICD pipeline) that spawns distributed load clients across multiple regions/AZs to reach 200k RPS. Use Terraform/Helm to provision ephemeral load clusters.- Components: traffic controller (scenario scheduler), distributed generators (k6/Locust/Gatling/Vegeta), dependency simulator (service-virtualization), telemetry exporter (OpenTelemetry -> Prometheus + Tempo + Grafana), result aggregator.2. Realistic traffic generation- Model user journeys using stateful scenario scripts (k6 JS or Locust Python): login, browse, add-to-cart, checkout, background polling, retry/backoff logic, session stickiness, think-times sampled from production traces.- Drive inputs from production telemetry (sampling spans/flows) to replicate request mix, payload sizes, auth tokens, and geographic distribution.- Seed test data with isolated namespaces and idempotent APIs to avoid corrupting real user data.3. Dependency simulation- Use service virtualization (WireMock, Mountebank) and/or synthetic backends for external APIs; run a “failure plane” that can introduce latency/HTTP 5xx, connection resets.- For databases, use isolated clusters with controllable latency (tc/netem) and replica lag simulation; for caches, emulate eviction spikes.4. Throttling & backpressure tests- Gradually ramp to and hold 200k RPS, then spike bursts (+50–100%) and sustained overloads to exercise rate limiters, circuit breakers, and queuing systems.- Validate correct behavior: reject-fast vs. queue growth, client retry budgets, and graceful degradation (feature flags).- Measure queue lengths, threadpool saturation, GC pauses, and downstream throttles.5. Chaos testing layer- Integrate Gremlin/Chaos Mesh to inject pod/node failures, network partitions, DNS flaps, API gateway outages, JVM OOMs, and disk pressure during load.- Run targeted experiments: kill X% of pod replicas, throttle egress to payment service, induce DB primary failover.6. Safety & blast radius controls- Run primarily in pre-prod or cloned environments with production‑like scale; if partial prod runs are required: - Canary + approval gates, time windows, and explicit runbooks. - Rate-limited IPs/VPCs, dedicated test accounts, and resource quotas. - Enforce billing caps and automated shutdown on threshold breach. - Read‑only and synthetic-data modes; route test traffic via dedicated ingress with header tagging; auto-disable if key metrics (user-facing errors, SLO breach) exceed thresholds. - Use RBAC to limit who can start tests.7. Observability & validation- Baseline: collect metrics (RTT, p50/p95/p99, errors, retries), traces, logs, and infra (CPU, mem, queue depth, network).- Automated analysis: anomaly detectors and canary-style health checks; post-test report with bottlenecks and actionable recommendations.Runbook & automation- Preflight checks (capacity, quotas), blue/green namespaces, automated rollbacks, and incident channels.- Schedule progressive test matrix: unit load -> 10% -> 50% -> 100% -> spikes + chaos. After each test, run postmortem and iterate.Why this works- Realistic user journeys + production-derived traces ensure fidelity.- Distributed generators and dependency virtualization let you hit scale without affecting prod.- Controlled chaos and safety gates exercise resilience while minimizing collateral damage.
EasyTechnical
26 practiced
Design a log retention and access policy for application logs that serves three user groups: on-call SREs, developers, and auditors. Consider retention durations, indexing strategies, GDPR/PII handling, and read/ingest costs.
Sample Answer
Situation: We need a practical log retention & access policy that serves on-call SREs (fast incident response), developers (debugging & feature troubleshooting), and auditors (compliance/history), while controlling costs and protecting PII under GDPR.Policy summary:- Retention windows: - Hot tier (searchable, full indexing): 30 days — used by on-call SREs for alerts/incident triage. - Warm tier (partial indexing, compressed): 6 months — used by developers for debugging and trend analysis. - Cold/Archive (object storage, metadata index only): 2–7 years depending on compliance — used by auditors; retrieval is slower/paid.- Indexing strategy: - Full indexing for structured fields needed in queries/alerts (timestamp, service, host, request_id, error_code). - Store raw message in compressed blob; use tokenized or sampled full-text for long-term. - Use shard-by-service and time-based indices to speed retention deletions.- GDPR / PII handling: - Identify PII fields and apply field-level redaction or hashing at ingest for developer/devops views. - Keep a separate secure vault with reversible encryption for audit/legal with strict access, retention triggers, and documented justification for any re-identification. - Provide automated deletion workflows to honor user data erasure requests (delete from hot/warm, mark for purge in cold).- Access control & audit: - RBAC: On-call SREs: read on hot/warm for operational namespaces; Developers: read on warm/limited hot for their services; Auditors: read-only on archive & selected warm data via jump-box with session recording. - All accesses logged and retained for 1 year.- Cost controls: - Use tiering (hot/warm/cold) to minimize expensive indexed storage. - Sampling/ingest filters for high-volume debug logs; increase sampling dynamically on incidents. - Lifecycle policies to auto-roll indices and delete expired data.- Operational notes: - SLA: hot search <1s; warm <10s; cold retrieval within 24h or on-demand faster via paid restore. - Periodic review (quarterly) of retention based on usage metrics and compliance needs.This balances rapid operational needs, developer debugging, compliance auditing, cost efficiency, and GDPR-safe handling.
Unlock Full Question Bank
Get access to hundreds of Technical Stack and Infrastructure interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.