Scalable System Architecture and Design Principles Questions

Architectural patterns principles and decision making for building systems that are maintainable resilient and able to scale. Coverage includes service decomposition and trade offs between microservice architectures and monoliths; layered and n tier architecture patterns; event driven design and command query responsibility segregation pattern; choosing synchronous versus asynchronous communication and its impact on correctness and latency; design principles such as loose coupling high cohesion separation of concerns and single responsibility; state management and session handling and when to favor stateless designs; application programming interface design versioning and contract management; front end and user experience considerations such as resource loading and progressive rendering; migration strategies for evolving systems and incremental refactoring; and how to weigh latency throughput reliability cost and development velocity when selecting architectures. Candidates should illustrate pattern selection with concrete examples and justify operational and developer experience implications.

HardSystem Design

55 practiced

Design a scalable event-sourcing + CQRS architecture for a high-throughput domain. Discuss event storage considerations (append-only model), snapshotting strategies, projection rebuilds, read-model partitioning, retention policies, and operational processes for replaying events and evolving event schemas while preserving backwards compatibility.

Sample Answer

**Clarify requirements & constraints**- Throughput (events/sec), latency for reads, retention/regulatory, multi-region, ordering guarantees, max aggregate size, failure/DR needs.

**High-level architecture**- Write model: API → Command handlers → Validate → Persist events (append-only) → Publish to event bus (Kafka).- Read model: Consumers build projections into partitioned read stores (Cassandra/Elastic/Redis) per query pattern.- Snapshot service: async snapshotter stored in object store + metadata in DB.

**Event storage (append-only)**- Use immutable log (Kafka for streaming + durable event store like EventStoreDB or S3+index) guaranteeing ordered per-aggregate partition key.- Ensure idempotent writes with optimistic concurrency (expected version).

**Snapshotting**- Snapshot frequency by aggregate traffic: either size-based (every N events) or time-based; store snapshots externally and keep pointer in event stream metadata.- Use incremental snapshots for large aggregates.

**Projection rebuilds & read-model partitioning**- Store projection offsets; support parallel replay by partitioning by aggregate id hash ranges.- Rebuild by replaying from snapshot sequence numbers or from t0; use Kafka consumer groups with stateful processors (Kafka Streams/Flink).- For large domains, partition read models by entity-type and query workload.

**Retention & compaction**- Keep full history for regulatory aggregates; for others implement TTL + summarized tombstones or archived cold storage.- Use compaction on Kafka topics keyed by aggregate id to retain last state and necessary history.

**Operational processes: replaying & schema evolution**- Replaying: provide safe playbooks — deploy consumer in replay mode, use feature flags, rate-limit, monitor catch-up lag, run on staging first.- Schema evolution: version events with explicit schema version field; use tolerant readers (add fields optional, ignore unknown fields), adapters/transformers for old versions during replay, maintain transformation library; when breaking change required, run projection migration using dual-write or blue-green projection builds.- CI: include contract tests, consumer-driven contracts, and automated replay tests against snapshots.

**Trade-offs & reliability**- Kafka gives throughput and retention but needs compaction and careful retention policies.- Snapshots speed rebuilds but add complexity for consistency — always persist snapshot after event durable write and ensure snapshot metadata atomicity.- Monitor with metrics (LAG, event rates, projection error rates) and run automated rollback drills.

HardTechnical

92 practiced

Case study: Your company wants to migrate a critical monolithic order-management system to an event-driven microservices architecture without significant downtime. Produce a migration plan that covers dual-writing, event schema evolution, consumer onboarding, data reconciliation, feature flags, canary rollout, rollback strategy, and how to preserve integrations with third-party systems during the transition.

Sample Answer

**Overview — goals & constraints**Migrate a critical monolith to event-driven microservices with zero/near-zero downtime, preserving third‑party integrations, enabling safe rollback, and ensuring data correctness.

**1) Clarify & prepare**- Inventory domains, owners, data flows, SLAs, and third‑party contracts.- Choose event backbone (Kafka/Rabbit/Kinesis) + schema registry (Avro/JSON Schema + compatibility rules).- Define strangler pattern: incrementally peel off order subdomains.

**2) Dual‑writing strategy**- Implement a write adapter in the monolith that publishes events for new/changed state and continues local DB writes.- Where feasible, introduce a transactional outbox (DB table + reliable publisher) to guarantee at‑least‑once semantics and avoid distributed transactions.- Start with producer SDKs that encapsulate idempotency keys.

**3) Event schema evolution**- Use schema registry with backward/forward compatibility rules.- Version events via schema metadata, avoid breaking changes; use additive fields and canonical enums.- Provide consumer-side adapters to map older/newer schemas during transition.

**4) Consumer onboarding**- Onboard services one domain at a time. Each consumer reads events in “consumer mode” from the topic while monolith is source‑of‑truth.- Implement idempotent processing and store consumer offsets; test locally with replayed topics.- Maintain a consumer readiness checklist (monitoring, DLQ, alerting).

**5) Data reconciliation**- Implement periodic reconciler service that compares monolith canonical state vs aggregated event store / microservice state using checksums or hashes.- Run reconciler in read-only mode with reports, then automated repair jobs gated behind feature flags.- Keep metrics: divergence counts, reconciliation time, repair success rate.

**6) Feature flags & canary rollout**- Use feature flags to route traffic: start with internal users, then percentage-based routing to new services.- Canary pattern: deploy service pair for a subset (e.g., 1–5%) of orders. Compare responses, latencies, and error rates.- Telemetry: end‑to‑end tracing, business correctness checks, SLA monitors.

**7) Rollback strategy**- Rollback via feature flags and Kafka consumer group pausing/rewinding: - If consumers misbehave, pause consumer groups and switch traffic back to monolith. - For DB schema changes, use additive migrations and migration scripts reversible where possible. - Use automated runbooks: steps to pause, revert flags, and replay/fall forward.

**8) Preserve third‑party integrations**- Maintain an integration facade: either keep monolith forwarding to third parties until services fully own integrations, or implement a shared adapter service that forwards calls/events to third parties, honoring rate limits and auth.- For webhooks, mirror outbound events from both systems during cutover and reconcile delivery logs.

**9) Observability & SLOs**- Distributed tracing, business metrics, error budgets, and alerting per domain.- DLQs, retry policies, and consumer health dashboards.

**10) Timeline & governance**- Phased sprints per domain, checkpoints for reconciliation pass/fail, rollback drills, and stakeholder signoff for each cutover.

Why this works: transactional outbox + schema registry ensures durability and compatibility; feature flags + canary enable safe progressive rollout; reconciler + DLQs ensure correctness; facade approach preserves third‑party integrations until cutover is proven.

MediumTechnical

65 practiced

Propose a service decomposition for an e-commerce platform including catalog, pricing, search, recommendations, cart, checkout, and notifications. For each service justify the boundary based on coupling, data ownership, scaling characteristics, failure isolation, and team ownership. Mention potential cross-service integration patterns.

Sample Answer

**High-level decomposition (services & responsibilities)**

- Catalog Service — owns product master data (SKUs, attributes, categories, media). Boundary: authoritative data owner; low write rate, high read. Justification: isolates product lifecycle changes, optimized for reads via CDN/edge caches. Team: Product Content.

- Pricing Service — owns price rules, promotions, taxes. Boundary: pricing rules separate from catalog to avoid coupling; frequent policy changes and CPU for rule evaluation. Scale: compute-heavy during promotions; cache prices per SKU. Team: Revenue/Finance.

- Search Service — owns inverted index, facets, relevance tuning. Boundary: derived view of catalog optimized for queries. Scaling: read-heavy, horizontally scalable (Elasticsearch/OPensearch). Team: Search/UX.

- Recommendations Service — ML models, user-item signals, feature store. Boundary: Datasets and models decoupled from catalog/pricing; eventual consistency acceptable. Scale: CPU/GPU for training, low-latency inference. Team: Data Science.

- Cart Service — session/cart state, reservations semantics. Boundary: user-scoped mutable state requiring strong consistency; frequently updated. Scale: sharded by user; failure isolation important. Team: Checkout/Core.

- Checkout Service — orchestrates payment, inventory hold, order creation. Boundary: transactional workflow spanning services; must ensure correctness. Use Saga pattern for distributed transactions. Team: Checkout/Payments.

- Notifications Service — owns delivery of email/SMS/push, templates and throttling. Boundary: cross-cutting but isolated for reliability and retry logic. Scale: bursty; use queuing. Team: Platform/Infra.

**Cross-service integration patterns**

- API Gateway + REST for synchronous interactions (catalog->search metadata, front-end).- Async event bus (Kafka) for data propagation: Catalog -> Search/Recommendations; Pricing -> Cart/Checkout for price updates.- Saga choreography for checkout (Cart emits OrderRequested, Inventory and Payment services react); or orchestrator for complex flows.- Circuit breakers, timeouts, retries for remote calls; bulkheads per service on infra layer.- Avoid shared DB; use well-defined events and read-models (CQRS) to reduce coupling.

**Operational considerations**

- Monitor service-level metrics (latency, error rates, queues) per service.- Autoscaling policies per workload type (CPU for ML, RPS for search).- Team ownership aligned to bounded contexts to enable independent deploys and failure isolation.

EasyTechnical

51 practiced

Given a medium-sized internal HR application, propose a top-level service decomposition using domain-driven concepts. Describe which capabilities you would separate into services (e.g., user-profile, payroll, onboarding) and justify boundaries in terms of team ownership, performance, security, and integration complexity.

EasyTechnical

60 practiced

Explain the circuit breaker and retry patterns. Describe how you would design exponential backoff with jitter, classify transient vs permanent failures, and combine circuit breakers with bulkheads to protect a downstream dependency in production.

Sample Answer

**Brief definition**- Circuit breaker: stop calling a failing downstream service after thresholds to fail fast, allow recovery probes.- Retry: reattempt transient failures with controlled delay.

**Exponential backoff + jitter**- Formula (base backoff):

text

delay = base * 2 ^ attempt

- Add jitter (recommended: full jitter):

text

delay = random(0, min(maxDelay, base * 2 ^ attempt))

- Why: exponential spaces retries to avoid thundering herd; jitter spreads retries to reduce collision.

**Classify failures**- Transient: network timeouts, 5xx intermittent errors, rate-limit 429 — retry with backoff.- Permanent: 4xx client errors (401, 403, 404), schema errors, auth failures — do not retry; require human/action.

**Combine circuit breakers and bulkheads**- Bulkhead: limit concurrent calls/threads/queues per dependency to isolate failures.- Pattern: put a bulkhead (connection pool/worker semaphore/queue) before dependency; use circuit breaker to open on error rate/time window.- Flow: requests hit bulkhead → if capacity used, fail fast (protecting other services); requests to dependency go through circuit breaker which opens on high error rate and schedules probe after cooldown.- Production knobs: thresholds (error rate %, min requests), cooldown, probe rate, bulkhead size, retry budget per request, observability (latency, errors, open count), and alerting.

**Example settings**- base=100ms, max=10s, maxAttempts=5, circuit: errorRate>50% over 30s with min 20 requests, cooldown=60s, bulkhead=50 concurrent.

Unlock Full Question Bank

Get access to hundreds of Scalable System Architecture and Design Principles interview questions and detailed answers.

Join thousands of developers preparing for their dream job.