System Architecture Communication and Documentation Questions

Assess the candidate ability to describe, document, and communicate system architecture both visually and verbally. Candidates should present what a system does and who uses it, identify major components and how they interact, show data flow and integration points, and explain critical architectural decisions and trade offs. Interviewers expect clear diagrams using standard conventions that show high level views, component interactions, and deployment topology, accompanied by concise narrative documentation. Strong answers include multiple views tailored to the audience, labeled diagrams, and justification of design choices while avoiding unnecessary implementation detail. Candidates should be able to discuss scaling strategies, reliability and operational considerations including failure modes, migration paths, observability, and deployment considerations. The scope includes common architectural building blocks such as microservices, application programming interfaces, databases, caching layers, and message buses, as well as consistency and availability implications and service to service communication patterns, and the connection between technical choices and business context.

HardTechnical

41 practiced

You are reviewing a payments platform architecture that mixes synchronous authorization and asynchronous settlement. The diagram lacks reliability annotations. Identify five critical reliability concerns (for example: duplicate charges, out-of-order processing, partial failures) and for each propose an architecture-level mitigation and how you would record it in the project's risk register and runbooks.

Sample Answer

Framework: I list five critical reliability concerns, each with an architecture-level mitigation and the exact way to record it in the risk register and runbooks so ops and stakeholders can act.

1) Duplicate charges- Mitigation: Enforce global idempotency keys (client-provided or generated at auth time) stored in a single source-of-truth (strongly consistent DB or idempotency service). Use at-most-once processing in async pipeline; dedupe on write.- Risk register: Title: "Duplicate charge risk"; Likelihood: Medium; Impact: High (financial/regulatory); Owner: Payments Platform Lead; Mitigation: Idempotency + audit trail; Detection: monitoring of duplicate-key hits; Residual risk: Low; Target RTO: 1h.- Runbook: Steps to query idempotency store, reconcile duplicates, reverse/compensate transactions, contact ops/legal, run SQL to create compensating reversal, and verify downstream clearing.

2) Out-of-order processing (auth vs settle)- Mitigation: Use explicit event versioning and causal ordering (stream partitioning by payment ID, include sequence numbers) and an ordering guarantee (per-partition single writer or per-payment concurrency control).- Risk register: Title: "Out-of-order settlement"; Likelihood: Medium; Impact: Medium; Owner: Integration Lead; Mitigation: Partitioning + sequence checks + tombstones; Detection: metrics for sequence gaps/reorders.- Runbook: Detect gaps (dashboard), pause consumer, replay from offset, run reconciliation to ensure settlement matches last successful auth.

3) Partial failures across sync/async boundary- Mitigation: Implement transactional outbox pattern: sync auth writes to DB and outbox in same transaction; reliable publisher reads outbox and marks published. Combine with compensation workflows for partial success.- Risk register: Title: "Partial write/publish failures"; Likelihood: Medium; Impact: High; Owner: Platform Eng; Mitigation: Transactional outbox + retry policy; Detection: stale outbox rows older than threshold.- Runbook: Inspect outbox table, manually publish rows, run compensating actions for any orphaned auths, escalate if >X rows.

4) Message loss / DLQ exhaustion- Mitigation: Durable message broker with at-least-once semantics, exponential backoff, DLQ with alerts and automated replay tooling; watermark persisted per consumer.- Risk register: Title: "Lost/poisoned messages"; Likelihood: Low-Medium; Impact: High; Owner: Messaging Owner; Mitigation: Broker durability + DLQ + replay tooling; Detection: DLQ growth alert.- Runbook: Tail DLQ, inspect payload, fix consumer or transform message, replay via tooling, or route to manual processing queue.

5) Reconciliation / state drift between ledger and clearing- Mitigation: Maintain an authoritative ledger (append-only) and nightly/real-time reconciliation jobs that compare ledger vs settlement system; implement anomaly detection and automated holds.- Risk register: Title: "Ledger/clearing drift"; Likelihood: Medium; Impact: Critical; Owner: Finance + Platform; Mitigation: Real-time reconciliation, alerts, auto-hold on anomalies; Detection: Reconciliation mismatch > threshold.- Runbook: Run reconciliation job, steps to isolate offending transactions, create compensating entries, contact clearing partner, restore balances, and post-mortem.

For each risk entry include: detection metric (alert threshold), playbook link, communication plan (stakeholders, SLA), pre-approved compensations, and test frequency (quarterly drills).

MediumSystem Design

52 practiced

Explain how you would include chaos engineering and failure-injection plans in architecture documentation. Provide a concrete short test for cross-region failover including hypothesis, steps to inject failure, expected outcomes, rollback criteria, and how results feed back into the architecture.

MediumSystem Design

45 practiced

You're preparing architecture documentation for a new microservices-based order processing system. Produce a C4-style context and container view: list actors, containers with responsibilities (e.g., API gateway, order service, payment adapter), and external systems (payment provider, shipping). Explain how you'd present differences when showing a lead engineer versus a sales stakeholder.

Sample Answer

Requirements clarification:- System: Microservices-based Order Processing for e‑commerce: accept orders, validate inventory, process payment, arrange shipping, provide customer notifications and dashboards. Non-functional: ~10k TPS peak, <200ms user-facing latency, PCI-DSS for payment, high availability, eventual consistency for inventory.

Context (C4 Level 1 — Actors & System):- Customer (end user) — places orders via web/mobile.- Merchant/Admin — monitors orders, cancels/refunds.- External Payment Provider — card/ACH processor (PCI scope).- Shipping Provider(s) — carrier APIs for labels/tracking.- Inventory System (could be third‑party WMS) — authoritative stock.- Notification System (email/SMS provider).- Order Processing System (this system) — handles order lifecycle.

Container View (C4 Level 2 — Containers & responsibilities):- API Gateway (public) — TLS termination, auth (OAuth/JWT), rate limiting, request routing, basic request validation, API versioning.- Web/Mobile Frontend (SPA / Mobile app) — UI, client-side validation, session handling.- Order Service (microservice) — core domain: create/update/cancel orders, state machine, stores order events in Order DB (event-sourced or relational), exposes REST/gRPC.- Payment Adapter (microservice) — orchestrates tokenization, authorization, captures, refunds via Payment Provider; isolates PCI scope; implements retry/backoff and idempotency keys.- Inventory Service (microservice) — reserve/commit/release stock, integrates with Inventory System via async events and periodic reconciliation.- Fulfillment Service (microservice) — generates shipping requests, prints labels, tracks status; integrates with Shipping Provider adapters.- Notification Service — async notifications (email/SMS/push) via Notification Provider; templating and retry.- Orchestration/Workflow Engine (orchestrator/CDC) — coordinates long-running flows (e.g., Saga coordinator) for distributed transactions and compensations.- Event Bus / Message Broker (Kafka/Rabbit) — reliable async messaging for domain events, eventual consistency and decoupling.- API/Integration Gateway for external systems — secure outbound connections, circuit breakers, credential vaulting.- Observability Stack — metrics (Prometheus), tracing (Jaeger), logs (ELK), alerting.- CI/CD & Infra (k8s, Helm, secrets manager, infra as code).

Data stores & responsibilities:- Order DB (immutable events or relational for queries).- Read-model(s) / CQRS caches for fast queries.- Token Vault / Payment Vault (if storing tokens) — PCI-compliant.

Security & ops notes:- Isolate Payment Adapter and Token Vault into PCI-scoped network segments.- Use mTLS, RBAC, WAF, rate limiting.- Implement circuit breakers, bulkheads, and backpressure.

Presentation differences: Lead Engineer vs Sales Stakeholder- Lead Engineer: present container diagram with tech choices, data flows, sequence for happy/failure paths, storage schemas, scaling plans, SLA targets, monitoring dashboards, trade-offs (event-sourcing vs relational, sync vs async), deployment topology, security segmentation, and runbook highlights. Show sequence diagrams for payment flow, saga compensations, and failure recovery.- Sales Stakeholder: present high-level context and value-centric benefits: how system reduces checkout friction, supports multiple payment providers, ensures compliance, scales for peak events, and reduces failed shipments. Use simplified diagram: Customer → API Gateway → Order Processing → Payment Provider / Shipping / Inventory. Emphasize time-to-market, integration effort, cost estimates, and risk mitigation (PCI handling delegated to Payment Adapter). Avoid low-level tech; include a short implementation timeline and key milestones.

Deliverables for both:- Annotated C4 diagrams (context + container), sequence example for order checkout, component responsibilities table, security & compliance appendix, non-functional requirements mapping, estimated T-shirt sizing and integration checklist.

EasyTechnical

43 practiced

Explain how you would tailor an architecture document for a non-technical executive versus a platform engineering team. Provide at least three content and presentation differences and include a one-line example executive summary for a new payment gateway integration.

HardSystem Design

47 practiced

Design and document a globally-distributed messaging platform for 100 million users where 95th percentile end-to-end latency must stay under 200ms. Describe high-level components, partitioning/sharding strategy, replication model, cross-region routing, capacity assumptions, common failure modes, and list three architectural trade-offs you made and why.

Sample Answer

Requirements (clarify): 100M global users, 95th-percentile end-to-end latency <200ms, durable messaging, high availability (>=99.95%), geo-aware delivery, cost-conscious.

High-level architecture:- Clients → Edge POPs (CDN-like) → Global API Gateway → Regional Ingress Brokers → Partitioned Message Stores + Replication → Delivery Workers → Edge Egress POPs → Clients- Control plane: Metadata service (catalog of partition ownership), Presence service, Routing fabric, Monitoring & autoscaling.

Partitioning / sharding:- Topic-based then key-hash partitioning: messages partitioned by (topic, user_id % P). Choose P large (e.g., 100k) to allow fine-grained scaling.- Hot keys handled via dynamic splitting and sticky routing to dedicated partitions.

Replication model:- Multi-master per region for local writes; async cross-region replication with per-message causal metadata (vector clocks or CRDT-like sequence) and conflict resolution rules. For critical ordered channels, use leader-per-partition with leader in region of majority of subscribers.

Cross-region routing:- Clients connect to nearest POP. If a subscriber in Region B subscribes to publisher in Region A, the regional broker consults metadata; if data not local, proxy fetches from remote replica (fast-path: recent writes pushed via async tailing to other regions), else read-from-leader via geo-replication. Use WAN-optimized gRPC streams and compression, edge caching for fan-out.

Capacity assumptions (order-of-magnitude):- 100M users, 10% active/day → 10M DAU; peak concurrent 1M. Avg messages/user/day 50 → 500M msgs/day ~5.8k msgs/sec avg, peak 10x → 58k msgs/sec. Average msg size 1KB → peak throughput ~58 MB/s.- Storage retention: 7 days → ~3.5 TB/day → ~24.5 TB total. Add replication factor 3 → ~74 TB.

Failure modes & mitigations:- Region outage: failover via metadata to other regions, client reconnection to nearest healthy POP. Data loss window minimized by tail-based async replication and write-ahead logs.- Network partitions: degrade to read-local/write-local; reconcile using causal metadata; rate-limit to avoid overload.- Hot partitions: detect and split, migrate replicas; throttle or backpressure publishers.- Broker overload: autoscale brokers, circuit breakers, client backpressure.

Three trade-offs:1) Consistency vs latency: chosen eventual/causal consistency across regions with leader-for-ordered-partitions. Rationale: strict global consensus would break 200ms SLA; causal + leader for critical flows gives practical balance.2) Replication style: async cross-region replication reduces write latency but risks small window of divergence. Acceptable given conflict resolution and short window; synchronous replication would increase latency and cost.3) Partition granularity: many small partitions (high P) for elasticity and fine-grained scaling at cost of more metadata and routing complexity. Chosen because expected skew and need to isolate hot keys outweigh metadata overhead.

Observability & ops:- End-to-end tracing, per-partition metrics, SLA alerting, chaos testing. Automated partition rebalancer, blue/green deploys, and DR runbooks.

Unlock Full Question Bank

Get access to hundreds of System Architecture Communication and Documentation interview questions and detailed answers.

Join thousands of developers preparing for their dream job.