Architecture and Technical Trade Offs Questions

Centers on system and solution design decisions and the trade offs inherent in architecture choices. Candidates should be able to identify alternatives, clarify constraints such as scale cost and team capability, and articulate trade offs like consistency versus availability, latency versus throughput, simplicity versus extensibility, monolith versus microservices, synchronous versus asynchronous patterns, database selection, caching strategies, and operational complexity. This topic covers methods for quantifying or qualitatively evaluating impacts, prototyping and measuring performance, planning incremental migrations, documenting decisions, and proposing mitigation and monitoring plans to manage risk and maintainability.

HardTechnical

29 practiced

Outline a migration plan to convert a sessionful stateful service that relies on sticky sessions into a stateless service suitable for horizontal scaling. Include options for session tokens, centralized session stores, token size/security trade-offs, and a rollout plan minimizing user disruption.

Sample Answer

Requirements & constraints:- Remove sticky sessions; enable horizontal scaling, low latency (<50ms), secure sessions, support large active-user base, minimal user disruption, rollback path.

High-level approach:1) Support two session strategies in parallel: centralized store (short-term) and stateless token (long-term). Migrate progressively.

Options & trade-offs:- JWT (self-contained): pros — no centralized read, fast; cons — larger token, revocation hard, sensitive claims must be signed+encrypted if confidential. Use short-lived access JWTs (e.g., 5–15m) + refresh tokens.- Opaque token + centralized store (Redis/DynamoDB/ElastiCache): pros — immediate revocation, small token size, centralized control; cons — extra network hop, availability/latency dependency.Hybrid: short-lived JWT for request auth + opaque refresh tokens validated against a store to handle logout/revocation.

Security & token size:- Keep JWT minimal: user id, roles, exp, jti. Sign with RS256; optionally encrypt sensitive claims. Limit payload to <1KB to avoid header bloat. Rotate keys via KMS and expose JWKS endpoint.- For opaque tokens, store session data in Redis with TTL; include device fingerprint & IP history for risk checks.

Migration/rollout plan:1) Instrumentation & infra: provision highly available Redis with replication, add monitoring, rate limits.2) Dual-mode API: accept both sticky-session cookies and new tokens. Implement adapter layer that reads cookie session, issues opaque token/JWT on first request.3) Backend changes: make services read session from token or centralized store; remove dependency on server-local memory.4) Canary: enable stateless path for 5% of traffic; validate session continuity, performance, error rates.5) Gradual ramp: 5→25→50→100%, while removing sticky-session routing. Maintain sticky as fallback for a short window.6) Revocation & key rotation: deploy token revocation list in store; support immediate logout.7) User experience: transparently refresh tokens; force re-login only if necessary.8) Rollback: revert LB sticky routing; maintain dual-mode for quick switch.

Testing & validation:- Automated integration tests, chaos testing (network partition, Redis failover), load testing under expected scale.- Metrics: auth latency, error rate, session miss rate, breadth of token sizes, cache hit ratio.

Outcome & learnings:- Expect small performance hit for centralized store; mitigate with local caches/P2P caching and very short JWTs. Hybrid approach balances scalability, revocation, and security.

EasyTechnical

32 practiced

Explain token bucket and leaky bucket rate-limiting algorithms, along with fixed-window and sliding-window counters. For an API gateway protecting against bursty clients and ensuring fair usage, which algorithm do you prefer and why?

Sample Answer

Token bucket and leaky bucket are two common token-based rate-limiting models; fixed-window and sliding-window are counter-based approaches. Short definitions first, then trade-offs and recommendation.

Token bucket:- Tokens accumulate in a bucket at a fixed rate (r tokens/sec) up to capacity B. Each request consumes tokens; if tokens available request proceeds, otherwise it's denied or queued.- Allows bursts up to B while enforcing long-term average rate r.- Good for bursty traffic where you want to smooth but permit occasional spikes.

Leaky bucket (as queue):- Incoming requests enter a queue; they are processed at a constant drain rate. If queue is full, excess requests are dropped.- Enforces a steady output rate; smoothes bursts by queuing, but limits burst acceptance.- Equivalent to a fixed-rate serializer; less flexible for allowed bursts.

Fixed-window counter:- Count requests per fixed interval (e.g., per minute). Simple, low overhead.- Prone to boundary effects: clients can send two bursts at window edges and exceed intended rate (burstiness).

Sliding-window counter:- More accurate: either implemented by rolling window buckets or exact timestamps to compute requests in last window length. Mitigates boundary issues compared to fixed-window.- Slightly more complex and higher storage/compute.

Recommendation for an API gateway protecting against bursty clients and ensuring fair usage:- Use token bucket (possibly combined with sliding-window for global fairness metrics). Token bucket permits controlled bursts (important for legitimate clients) while enforcing average rate; bucket size lets you tune tolerated burst. For fairness across many clients, maintain per-client token buckets and global quotas; combine with sliding-window counters or Redis-based token storage to prevent sync issues across nodes.- If you need strict constant output (e.g., downstream capacity constraints), use leaky-bucket behavior (or set token bucket drain to small rate).- For simple low-cost limits, fixed-window is easiest but vulnerable to window-edge spikes; prefer sliding-window over fixed-window if accuracy matters.

Key practical notes:- Implement per-client state (tokens or counters) with distributed store (Redis) with TTLs.- Choose token refill rate and bucket size based on SLA and expected burstiness.- Add exponential backoff /Retry-After headers and monitoring to tune limits.

HardTechnical

29 practiced

You must prototype and benchmark a new microservice to ensure it meets a p95 latency SLA of 150ms under 5k RPS. Describe a test plan including load-generation approach, environment parity, metrics to collect, how to identify bottlenecks, and what acceptance criteria you would use before production rollout.

Sample Answer

Overview: I’d create a repeatable benchmark that validates p95 ≤ 150ms at 5k RPS while isolating variables. Plan covers load generation, environment parity, observability, bottleneck triage, and clear acceptance gates.

1) Requirements & constraints- Traffic: 5,000 RPS steady-state, realistic request mix (payload sizes, endpoints).- SLA: p95 latency ≤ 150ms, SLO error budget 0.1% for 5-min windows.

2) Environment parity- Use an environment as close to prod as possible: same instance types, container runtime, JVM/GC settings, networking (VPC, subnets), sidecars, autoscaling config, and datastore versions.- If cost prohibits full parity, run a scaled full-stack cluster (e.g., 1:1 infra but lower replica count) and validate scaling linearity.

3) Load generation approach- Use distributed load generators (k6 / Gatling / Vegeta) in multiple zones to avoid generator becoming the bottleneck.- Reproduce traffic pattern: ramp-up (10m), warm-up (5m), steady-state (30m), stress (spike + soak).- Inject realistic think-times, concurrency distribution, and error scenarios (timeouts, retries) and include end-to-end and single-endpoint tests.

4) Metrics to collect- Latency histograms (p50/p90/p95/p99) at client and server.- Throughput (RPS) and success/error rates.- Resource metrics: CPU, memory, GC pauses, thread/connection pools, socket usage, NIC, disk I/O, DB latency/throughput, cache hit rates.- Application metrics: request queue lengths, queue wait time, downstream call latencies, semaphore/lock contention, database pool exhaustion.- System metrics: network latency, packet drops, kernel run-queue.- Traces: distributed tracing (span durations for each hop).

5) Identifying bottlenecks- Correlate latency spikes with resource metrics and traces.- Use flamegraphs (CPU), async-profiler or perf for hotspots.- Inspect GC logs and thread dumps for pauses or locking.- Check queue lengths, connection pool saturation, and backpressure propagation.- Isolate components: run with local in-memory mocks to see if datastore or network causes the issue.

6) Remediation & iterative testing- Fix the highest-impact items (e.g., DB indexes, caching, batch processing, connection pool tuning, reduce allocations).- Re-run tests after each change; keep experiment logs and versioned configs.

7) Acceptance criteria before rollout- Production-like environment test with steady 5k RPS for at least 30 minutes where: - p95 latency ≤ 150ms client-observed - p99 latency within an agreed secondary threshold (e.g., ≤ 300ms) - Error rate ≤ 0.1% - No resource saturation (CPU < 80% sustained, DB pool not exhausted, < 5% packet loss) - Traces show no single hop causing >50% of p95- Run a canary rollout (5-10% traffic) with monitoring for 1–2 hours; if metrics remain within thresholds, progressively increase.

8) Reporting- Deliver a benchmark report: test topology, configs, raw metrics, latency histograms, flamegraphs, root-cause analysis, changes made, and final pass/fail against acceptance criteria.

This plan makes benchmarking reproducible, focuses on end-to-end user impact, and provides clear gates for safe production rollout.

HardTechnical

48 practiced

You must decide whether to use read replicas in-region or cross-region replication to reduce read latency for global users. Given sample numbers (read latency baseline 120ms, in-region replica reduces to 40ms, cross-region replication adds $X/month per replica and increases write propagation latency by 200ms), describe how you would quantify the cost vs latency trade-off and define thresholds for action.

Sample Answer

Approach: turn latency improvements into measurable business/value metrics, then compare against replica cost + operational risk. Steps and formulas I would use:

1) Define metrics- Users impacted (U) per region, read QPS per user (r), fraction of reads served by replica (f).- Baseline read latency Lb = 120ms, in-region Li = 40ms (delta = 80ms saved), cross-region latency Lx = depends (assume similar to Li for remote users) but write propagation cost Wadd = +200ms.- Replica cost Crep = X $/month per replica.- Business value per ms saved per request Vms (estimated from conversion uplift, retention, SLA penalties, or internal cost of latency; if unknown, run sensitivity analysis).

2) Compute monthly latency savings (ms-month)ms_saved_per_read = Lb - Li = 80msmonthly_reads = U * r * 3600 * 24 * 30 * fmonthly_ms_saved = ms_saved_per_read * monthly_reads

3) Convert to monetary benefitmonthly_value = monthly_ms_saved * Vms

4) ROI and decision thresholdROI = (monthly_value - Crep) / CrepAction rules:- Deploy in-region replica if monthly_value >= Crep (ROI >= 0) and SLOs require lower p95/p99 latency.- Prefer replicas in regions with largest (monthly_value / Crep) ratios first.- Consider cross-region only if in-region replica not feasible; include write-cost trade-off: ensure added write propagation (Wadd) * #writes * cost_per_ms_write_loss < benefit_for_reads.

Example numeric threshold:If Vms = $1e-6 per ms (i.e., $0.000001 per ms), U=100k active users, r=0.01 reads/sec, f=0.8:monthly_reads ≈ 100k*0.01*2.592e6*0.8 ≈ 2.07e9 readsmonthly_ms_saved ≈ 80 * 2.07e9 ≈ 1.656e11 msmonthly_value ≈ 1.656e11 * 1e-6 = $165,600 — clearly justifies replicas if Crep = X is much smaller.

5) Sensitivity and constraints- Vary Vms and U to find break-even X.- Account for write-consistency penalties: compute extra user-impact ms due to Wadd for writes and subtract from benefit.- Add operational costs (monitoring, failover), and risk: stale reads, complexity.

6) Non-monetary thresholds- If p95 read latency target < 100ms, deploy in-region replicas regardless of ROI for key regions.- If write consistency SLA would be violated by +200ms, require application-level mitigations (read-after-write routing) before cross-region replicas.

Conclusion: quantify per-region monthly benefit (monthly_ms_saved * Vms), subtract the replica cost and write-propagation penalty; deploy where ROI>0 and/or where latency SLOs demand it. Use sensitivity analysis to set conservative Vms and decide prioritization.

EasyTechnical

57 practiced

Explain the CAP theorem and its implications when designing distributed services. Provide concrete examples of systems that intentionally prioritize Consistency over Availability and vice versa, describe how Partition tolerance constrains choices, and explain how eventual consistency models fit into CAP in real product scenarios.

Sample Answer

CAP theorem: in a distributed system you can’t simultaneously guarantee Consistency (all nodes see the same data at the same time), Availability (every request receives a response, even if stale), and Partition tolerance (system continues to operate despite network partitions). With partitions possible in real networks, you must choose between Consistency and Availability during partitions.

Implications and examples:- Prioritize Consistency over Availability (CP): systems that must be correct even if some requests fail. Example: traditional relational databases in a distributed config—Google Spanner and many strongly-consistent replicated databases (e.g., single-leader Raft clusters) will reject or delay responses during partition to avoid serving stale/ divergent data.- Prioritize Availability over Consistency (AP): systems that prefer to respond even if data may be stale. Example: DNS and many Dynamo-style stores (Amazon DynamoDB original design, Cassandra in certain configs) accept writes on multiple partitions and reconcile later.

How Partition tolerance constrains choices:- Because partitions can and do occur, distributed services cannot ignore P; they must design for how the system behaves under partition. That forces a runtime trade-off: during a partition either serve potentially inconsistent responses (choose A) or block/return errors to preserve global consistency (choose C).

Eventual consistency in products:- Eventual consistency is an AP-friendly model where updates propagate asynchronously and replicas converge over time. Practical use cases: shopping cart services, social feeds, caching layers, or analytics where slight staleness is acceptable. To make this safe, products add mechanisms: vector clocks or versioning to reconcile conflicts, read-repair, hinted handoff, quorum reads/writes (tunable consistency), and business rules (last-write-wins, CRDTs) to ensure correct convergence.

Takeaway: design choices depend on correctness vs. user-perceived availability. State clearly which operations need strong consistency (payments, inventory reserved) and which can tolerate eventual convergence (notifications, metrics) and apply the appropriate replication/consistency strategy.

Unlock Full Question Bank

Get access to hundreds of Architecture and Technical Trade Offs interview questions and detailed answers.

Join thousands of developers preparing for their dream job.