Scalability and Future Extension Questions

Design systems that scale: handle 10 items, 1000 items, 10,000 items efficiently. Design for future feature additions without major refactoring. Use abstraction and interfaces to allow flexibility. Discuss how your solution would adapt if requirements changed. This shows you think beyond the immediate requirement.

EasyTechnical

37 practiced

Describe what a 'stateless' service means in a microservices context and why statelessness aids scalability and future extension. Provide a concrete example of converting a session-backed service that stores in-memory sessions to a stateless design, and explain trade-offs in latency, operational complexity, and failure modes.

Sample Answer

A stateless service in microservices means each request contains all information needed for processing and the service does not rely on server-held (local) session state between requests. The service’s behavior depends only on the request and external persistent stores, not on in-memory session data tied to a particular instance.

Why this helps scalability and extension:- Horizontal scaling becomes simple: any instance can handle any request—load balancers don’t need sticky sessions.- Faster deployment and rolling upgrades: instances can be added/removed without session migration.- Easier fault recovery and geo-scaling: traffic can be routed to nearest healthy instance without state reconciliation.- Enables reuse: other services can call it without coordination about session affinity.

Concrete example: converting in-memory session store to stateless using JWTs- Current: Web tier creates server-side session object (user ID, roles) in RAM; LB uses sticky sessions.- Convert: On login, auth service issues a signed JWT containing user id, roles, expiry. Client sends JWT in Authorization header on each request. Each microservice validates signature and expiry (using local public key or cached JWK), then authorizes request—no local session store required.- Alternative: use a centralized session cache (Redis) accessed by all instances if tokens are inappropriate (e.g., immediate revocation needed).

Trade-offs- Latency: JWT validation is fast (local crypto + cached keys) and typically lower latency than a remote session lookup. However, if you use a centralized cache (Redis), you add a network hop per request increasing latency.- Operational complexity: JWTs simplify runtime (no session storage) but require secure key management, rotation, and precautions about token size and claims. Centralized caches add operational overhead (HA, failover, scaling).- Failure modes: Stateless with JWTs avoids session store as single point of failure—if key management fails or tokens are compromised you risk large blast radius. With a centralized cache, cache failure or network partition can make sessions unavailable or inconsistent; sticky sessions hide load balancing issues but are brittle for scaling and recovery.- Security and revocation: JWTs are harder to revoke immediately; you must use short expiry or a revocation list (which reintroduces state). Central store supports immediate invalidation.

Summary: Stateless design improves scalability, resilience, and agility for solutions architects, but requires attention to auth token lifecycle, key management, revocation strategy, and possible added latency/ops when choosing between pure stateless tokens and shared session stores.

MediumSystem Design

33 practiced

Design an API Gateway layer that centralizes cross-cutting concerns: routing, authentication/authorization, rate-limiting, A/B testing, and observability. Describe a plugin/policy mechanism to add new per-request behaviors (e.g., request transforms, per-tenant headers) without redeploying downstream services and how you'd ensure the gateway itself scales and remains resilient.

Sample Answer

Requirements:- Functional: central routing, authN/authZ, rate limiting, A/B testing, request transforms, per-tenant headers, metrics/tracing, health checks.- Non-functional: low latency (<50ms added), high availability, multi-tenant, hot-config updates without redeploy, observability.

High-level architecture:- Edge layer: lightweight L4/L7 load balancer (e.g., ALB/Envoy) -> API Gateway cluster (stateless pods) -> Service mesh / downstream services.- Shared backing services: distributed config store (Consul/etcd), policy catalog (git-backed + API), auth provider (OAuth/OIDC/JWKS endpoint), rate-limit store (Redis/BigTable), metrics/tracing backend (Prometheus + OpenTelemetry collector), CI/CD for policies.

Plugin / Policy mechanism:- Policy model: small DSL or WASM-based policies packaged as immutable artifacts. Each policy exposes lifecycle hooks: onRequest, onResponse, onError.- Runtime: Gateway embeds a WASM runtime (wasmtime/WAMR) or a Lua/VM isolation layer. Policies are loaded from config store and cached; each request executes ordered policy chain (route -> global -> tenant -> route-specific).- Management: policies versioned in Git; CI validates and publishes to config store. Gateway polls config or uses push notifications (gRPC streaming) to hot-swap policies without restarting.- Examples: request-transform policy modifies body/headers; tenant-header policy injects X-Tenant from JWT claims; AB-test policy reads header/cookie and sets routing weight.

Authorization & Rate-limiting:- AuthN: JWT/OIDC validation at gateway (JWKS caching). AuthZ: policy-based checks (via ABAC/RBAC) with optional external PDP for complex rules.- Rate-limit: token-bucket implemented via Redis/clustered store with local leaky-bucket caches for bursts and fallback to consistent hashing.

Routing & A/B:- Routing: route table with predicates (method, path, headers, tenant); can use weighted upstreams for A/B. A/B policy decides bucket and sets upstream tag.- Canary/blue-green via header/tag-based routing and traffic weight adjustments in control plane.

Observability & Security:- Tracing: inject context and export spans via OpenTelemetry. Logs structured JSON with trace IDs. Metrics: request latency, policy execution times, rate-limit rejects.- Security: mTLS between gateway and services, input validation, size limits, circuit breakers.

Scalability & Resilience:- Stateless gateway pods behind LB; scale horizontally. Use readiness probes and graceful shutdown.- Sticky state minimized; use external stores for any stateful needs. Local caches with TTL reduce backend calls.- Failure modes: config-store unavailability -> use last-known config; rate-limit store fail -> degrade to conservative limits; policy crash isolation via WASM sandbox and per-policy CPU/memory caps.- Autoscaling: HPA based on CPU, request latency, and queue depth; partition traffic per region/zone to reduce blast radius.- Operational: canary policy rollout, real-time metrics dashboards, SLOs and alerting, playbooks for rollbacks.

Trade-offs:- WASM gives safety and performance but increases complexity; Lua is faster to integrate.- Centralized gateway simplifies concerns but is a critical control plane—mitigate with redundancy, observability, and limited per-request work.

MediumSystem Design

30 practiced

Design an event-driven architecture for propagating item lifecycle events (create/update/delete) to downstream consumers: search indexer, recommender, notification service. Discuss message format choices, ordering guarantees, partitioning keys, idempotency strategies, and how producers can evolve without breaking future consumers. How would you enforce and test event schema compatibility?

Sample Answer

Requirements (clarify): reliable delivery of item create/update/delete to three consumers, per-item ordering, high throughput, low latency, safe schema evolution, and ability for consumers to be idempotent and tolerant of duplicates.

High-level architecture:- Producers publish events to a durable pub/sub (Kafka, Pulsar, or cloud Pub/Sub). One topic "item-events" (or logical topics per domain).- Consumers (search indexer, recommender, notification) subscribe independently; each has its own consumer group.

Message format and schema:- Use a binary schema (Avro or Protobuf) stored in a Schema Registry to enforce contracts and efficient serialization. Envelope fields: - key: item_id (string) — used as partition key - headers/metadata: event_type (create/update/delete), event_id (UUID), timestamp, version, source, causal_id - payload: typed record (the item state for create/update, or tombstone/partial for delete)- Example: Avro record for envelope + nested item record in Schema Registry.

Ordering guarantees & partitioning:- Partition by item_id (hash of item_id) so all events for the same item land in the same partition → per-item ordering guaranteed (within a partition). Document that cross-item ordering is not guaranteed.- If stronger global order required, use single-partition topic or sequence service (but limits throughput).

Delivery semantics & idempotency:- Use at-least-once delivery (Kafka-default). Enforce idempotency at consumers: - Each event has event_id and sequence_number (monotonic per item). Consumers persist last-processed sequence_number per item; ignore older or duplicate event_ids. - For operations that are not fully idempotent (e.g., counters), use upserts based on sequence or last-write-wins with timestamp/sequence. - For deletes use tombstone events (nullable payload or explicit delete_type) and consumer-side reconciliation. - Maintain compacted topic for current item state for late-joining consumers to bootstrap.

Producer evolution & schema compatibility:- Use Schema Registry with compatibility rules: - Backward compatibility: new schema readable by old consumers (add optional fields with defaults). - Forward compatibility: old producers readable by new consumers if needed. - Semantics: prefer backward-compatible changes (add optional fields, deprecate fields rather than remove).- Version fields in envelope; avoid breaking changes to headers.- If a breaking change is necessary, create a new topic version (item-events-v2) and perform migration with dual-write or converter bridges.

Enforcing & testing schema compatibility:- CI pipeline that registers schemas to Schema Registry in a sandbox and runs automated compatibility checks (schema-registry APIs).- Consumer-driven contract tests: each consumer publishes the schemas it expects and CI runs schema compatibility and sample payload validation.- Integration tests: - Property tests that validate serialization/deserialization between producer and consumer codegen. - End-to-end staging with schema evolution scenarios (old producer → new consumer; new producer → old consumer). - Chaos tests: duplicate, out-of-order, late, and missing events to validate idempotency and recovery.- Monitoring: track consumer lags, dead-letter queues for deserialization errors, and schema mismatch alerts.

Operational considerations and trade-offs:- Topic partition count should be high enough for throughput but consistent hashing ensures per-item affinity. Repartitioning is expensive — use sticky hashing and consider key namespace shard strategy if item hot-sharding expected.- Compaction helps retention of latest state while retaining event store in a separate append-only topic if full audit log is required.- Provide libraries/SDKs for producers/consumers that wrap envelope creation, schema registration, event_id/sequence generation to reduce accidental breaking changes.

This design gives per-item ordering, scalable parallelism, safe schema evolution, and clear idempotency strategies enforced by event metadata and CI-backed schema governance.

HardSystem Design

29 practiced

A customer needs ACID-like guarantees across multiple microservices for critical flows, but you cannot change underlying databases. How would you design application-level patterns (sagas, compensating transactions, two-phase commit emulation) to provide stronger consistency guarantees? Discuss trade-offs in complexity, latency, and how your design can be extended as new services and flows are added.

Sample Answer

Requirements (clarify):- Strong "ACID-like" guarantees for critical flows spanning services (atomicity + isolation + durability desirable; strict serializability likely impossible without DB changes).- Cannot change underlying DBs; must work at application layer.- Low-to-moderate additional latency acceptable for critical flows; extensible to new services.

High-level approach:- Use a hybrid of orchestrated Sagas with optional two-phase-commit (2PC) emulation for highest-criticality flows. Provide a transaction coordinator service, per-service transaction adapters, durable event/log store, and a compensator library.

Core components:1. Transaction Coordinator (TC): orchestrates steps, tracks state in durable store (append-only, immutable log) and enforces timeouts/retries.2. Service Adapters: each microservice implements idempotent APIs, a prepare/commit/compensate interface, and writes an event to local outbox.3. Outbox + Reliable Delivery: local transactional outbox pattern per service to ensure TC sees state changes.4. Compensator Library: standardizes compensating actions and retry/backoff.5. Visibility/Monitoring: dashboards and manual-reconcile tools.

Patterns:- Orchestrated Saga (default): TC invokes service actions sequentially/parallel, records success; on failure, triggers compensating actions in reverse order. Pros: simpler, lower latency than global locks. Cons: eventual consistency; need compensations.- 2PC Emulation (for strict flows): implement application-level prepare/commit where prepare causes service to persist a “prepared” state and lock resources (optimistic or pessimistic), commit flips to committed. Use coordinator to gather prepare ACKs, then commit or rollback. Pros: stronger atomicity; Cons: higher latency, risk of long-held locks and blocking, complex timeout/recovery, depends on services supporting prepare semantics and durable prepared state.- Idempotency + Outbox: ensure retries safe; transaction log durable to recover coordinator after crash.

Data flow example:1. Client requests multi-service transaction to TC.2. TC issues prepare (if using 2PC emulation) or action call (saga) to each Service Adapter.3. Each service writes local change + outbox event in same DB txn and returns prepared/ok.4. TC commits: sends commit or triggers compensations.

Trade-offs:- Complexity: 2PC emulation significantly increases per-service complexity (locks, prepared state) and operational burden. Orchestrated sagas are easier to adopt incrementally.- Latency: 2PC adds roundtrips and locking windows -> higher latency. Sagas have lower average latency but visible eventual consistency window.- Availability: 2PC reduces availability under partitions; as CAP dictates, cannot have perfect consistency + availability. Sagas favor availability.

Extensibility:- Design TC and Service Adapter interfaces as versioned contracts (prepare/commit/compensate/status). New services implement the adapter and register capabilities.- Use feature flags to route flows to Saga vs. 2PC emulation per flow.- Use a schema for logged steps to enable automated compensator generation and human-in-the-loop reconciliation.- Provide client libraries and templates (idempotency keys, outbox patterns) to reduce onboarding cost.

Operational considerations:- Strong monitoring, alerting for stuck prepares, compensations failures, and reconciliation UI.- Define SLAs for max prepare time and compensation retry policies.- Maintain audit trail for compliance.

Recommendation:- Default to orchestrated Sagas + outbox + idempotency for most flows; reserve 2PC emulation for a small set of ultra-critical flows after cost/benefit analysis. This balances complexity, latency, and extensibility while meeting near-ACID guarantees where truly needed.

MediumTechnical

25 practiced

Design a rate-limiting and throttling strategy to protect an API that must support bursts and scale to customers with up to 10k items per account. Compare token-bucket vs leaky-bucket and centralized counters vs distributed local limiters with reconciliation. Explain how you would support per-tenant, per-user, and per-endpoint limits and extend to new SLA tiers later.

Sample Answer

Situation / goals briefly: We need an API rate-limiting/throttling architecture that (1) supports short bursts, (2) scales to tenants with up to 10k items (high fan-out), (3) supports per-tenant, per-user and per-endpoint policies, (4) is extensible to new SLA tiers.

Recommendation summary:- Use token‑bucket for burst support and leaky‑bucket semantics for smoothing where steady rate is required. Token bucket allows tokens to accumulate (burst), leaky-bucket enforces constant drain (smooth).- Implement distributed local limiters at the API edge (per-instance token buckets) backed by a centralized policy and occasional reconciliation via a counters/aggregator service. This hybrid gives low-latency enforcement and global safety.

Design details:1. Enforcement layers - Edge (primary): each gateway instance runs local token-bucket limiters keyed by tenant:user:endpoint. Tokens refilled per configured rate. This minimizes latency and handles bursts up to local capacity. - Global coordinator: a centralized counter/aggregator (Redis/Scylla with atomic increments or CRDTs) maintains global usage windows and policy state for hard caps and cross-instance coordination. Edge nodes periodically (e.g., 100ms–1s) reconcile with global counters to avoid global overcommit. - Fallback: when coordinator unreachable, edges degrade to conservative local limits (fail‑closed or per-tenant throttling) to protect backend.

2. Counters vs local limiters trade-offs - Centralized counters: accurate global view, good for absolute hard limits, but higher latency and single-point scaling needs. - Local limiters: low latency, scalable, handle bursts. Risk of over-issuing across nodes; reconciliation and conservative quotas (divide global allowance among nodes) mitigate this.

3. Multi-dimension policy - Policy model: hierarchical keys: tenant -> user -> endpoint. Evaluate limits in order: endpoint-specific > user-specific > tenant SLA. Use most restrictive result. - For large tenants (10k items), use sharded keys and rate-limits per logical shard (e.g., per-item-group) to avoid hot keys; apply aggregate tenant cap too.

4. SLA tiers & extensibility - Define tier parameters: rate, burst size, priority, weighting. Store policies in central policy store; edges fetch via push config (gRPC/stream) and cache with TTL. - For premium tiers, provide higher token refill rates, larger burst bucket sizes, and priority queuing (weighted fair queuing) under contention.

5. Resilience & metrics - Use circuit-breakers, backpressure headers (Retry-After), and expose metrics (requests/sec, throttled, rejected) per dimension for billing/alerts. - Autoscale coordinator (partition by tenant hash). Use optimistic local issuance and compensating actions (reject/penalize) if reconciliation shows overuse.

Why this approach:- Token-bucket at edge gives required burst handling with low latency.- Centralized counters handle hard global guarantees and SLA enforcement.- Hybrid design balances correctness, performance and scalability and is extensible by updating policy store for new SLAs.

Unlock Full Question Bank

Get access to hundreds of Scalability and Future Extension interview questions and detailed answers.

Join thousands of developers preparing for their dream job.