System Architecture Communication and Documentation Questions

Assess the candidate ability to describe, document, and communicate system architecture both visually and verbally. Candidates should present what a system does and who uses it, identify major components and how they interact, show data flow and integration points, and explain critical architectural decisions and trade offs. Interviewers expect clear diagrams using standard conventions that show high level views, component interactions, and deployment topology, accompanied by concise narrative documentation. Strong answers include multiple views tailored to the audience, labeled diagrams, and justification of design choices while avoiding unnecessary implementation detail. Candidates should be able to discuss scaling strategies, reliability and operational considerations including failure modes, migration paths, observability, and deployment considerations. The scope includes common architectural building blocks such as microservices, application programming interfaces, databases, caching layers, and message buses, as well as consistency and availability implications and service to service communication patterns, and the connection between technical choices and business context.

HardTechnical

54 practiced

Explain how you would represent and keep up-to-date a cross-team API dependency graph to identify single points of failure. Describe the workflow to collect data, the diagram artifacts, how to mark criticality, and an actionable mitigation plan for critical dependencies.

Sample Answer

Requirements & goals:- Maintain an accurate, queryable cross-team service/API dependency graph to surface single points of failure (SPOFs), drive SLO-aligned risk prioritization, and produce actionable mitigations.

Data collection workflow:1. Instrumentation: require each service to publish a lightweight dependency manifest (YAML/JSON) in CI (service name, endpoints used, call type sync/async, cardinality, SLA/SLO, owner, auth). Validate manifest with schema in PR checks.2. Passive discovery: ingest traces (distributed tracing like OpenTelemetry), metrics (client side request counts/error rates), and config (service mesh, DNS, kube services) to infer dependencies and call volumes.3. Reconciliation pipeline: daily ETL that merges declared manifests with observed telemetry, flags mismatches, notifies owners for confirmation via automated Slack/email triage.4. Governance: dependency manifest is required for deployment; quarterly audit.

Diagram artifacts:- Graph DB backend (Neo4j or timeseries-enabled DAG store) with nodes: service, data-store, external API; edges: call type, qps, error_rate, success_rate, latency, SLA.- Two views: global high-level topology and per-SRE/team interactive diagram (filter by environment, criticality).- Exportable artifacts: SVG diagrams for runbooks and a machine-readable graph used by automation.

Marking criticality:- Compute risk score per node/edge: Risk = f(consumer_count, aggregate_qps, error_budget_usage, SLO_margin, recovery_time_objective, multi-region presence). Weight SPOF if many high-SLO consumers depend on a single provider.- Tag nodes: Critical/High/Medium/Low; highlight edges exceeding thresholds (e.g., >50% of total calls to service X).- Surface “blast radius” by traversing graph from a node to measure impacted consumer SLOs and traffic percent.

Actionable mitigation plan for critical dependencies:1. Immediate: add circuit breakers, rate-limits, retry budget, and fallback behavior in consumers; runbook entries with rollback and traffic-cut instructions.2. Short-term (weeks): increase redundancy (multi-region, replicas), caching layers, and degrade gracefully (cached responses, reduced features).3. Medium-term (1–3 months): refactor to reduce coupling (introduce async queues, bulkheads, service facades), replicate stateful components, and SLA contract upgrades.4. Long-term: eliminate SPOF by introducing alternative providers or splitting responsibilities; add automated failover and chaos testing.5. Validation: create SLO-based tests in CI, scheduled canary/chaos runs, and yearly dependency review.

Operationalization:- Integrate graph with pager/incident tooling to auto-attach impacted-services in alerts.- Run weekly reports of top 10 SPOFs and track mitigation tickets in backlog with SLA-driven prioritization.- KPIs: reduction in number of critical SPOFs, time-to-mitigate, and incident frequency attributable to dependencies.

This approach blends declared data, telemetry, automated reconciliation, SLO-driven risk scoring, and an operational mitigation playbook so the organization can find and fix SPOFs predictably.

HardTechnical

41 practiced

You must convince executives to fund observability improvements. Produce a one-page architecture diagram and a concise narrative that links current observability gaps to business KPIs, quantifies outage risk and cost, and recommends prioritized, measurable improvements. Explain how you tailored the diagram and narrative to an exec audience.

Sample Answer

Architecture (one-page ASCII diagram — executive view):

User Traffic → Load Balancer → Services Cluster (Auth, Payments, API) → Datastores │ ┌──────────────┴──────────────┐ │ │ Metrics Pipeline → Observability Platform (Dashboards / Alerts / SLOs) │ │ Tracing & Logs ← APM / Distributed Tracing ← Error Events │ Incident Automation (Runbooks, Pager, Playbooks, ChatOps) │ Postmortem & Analytics (RCA, Cost & Trend Reports)

Concise narrative linking gaps → business KPIs → recommended roadmap:

Current gaps:- Low signal-to-noise alerts and no end-to-end tracing; MTTR = 90 min; average P1s = 6/month.- No SLOs for Payments; unknown error budget burn.- Fragmented telemetry causing blind spots for customer-impacting failures.

Business impact & quantification:- Average revenue at risk during P1 (Payments outage): $120k/hour.- Annualized outage cost = 6 incidents/month * 1.5 hours * $120k = $10.8M.- Hidden productivity & brand costs ≈ $1.2M/year.- Total estimated risk ≈ $12M/year.

Recommended prioritized, measurable improvements:1) Instrumentation + Distributed Tracing (60 days) — goal: reduce MTTR by 50% (to 45 min). KPI: MTTR, percent of requests with trace = 95%.2) Define SLOs for top-3 services (30 days) — goal: detect error budget burn early. KPI: SLO compliance, incidents prevented.3) Alert Rationalization & Runbook Automation (45 days) — goal: reduce alert noise by 70%; automate first-response playbooks. KPI: actionable alerts per day, time-to-ack.4) Centralized Observability Platform + Cost/ROI dashboards (90 days) — goal: single pane for execs with monthly trend and cost of downtime. KPI: dashboard adoption, reduction in high-severity incidents.

Estimated investment vs. benefit:- One-time + annual tooling & engineering: $800k first year.- Projected avoided outage + productivity savings year 1: $4–6M -> 5–7x ROI.

How I tailored this to execs:- Diagram is high-level, non-technical, showing flow from user impact to business analytics.- Narrative ties concrete metrics (MTTR, incident count, $/hour loss) to bottom-line risk and ROI.- Recommendations are outcome-focused, time-boxed, and measurable so executives can approve investment with clear expected payback and KPIs for follow-up.

MediumTechnical

59 practiced

Given an architecture that includes producers, a message queue, and multiple consumer services, list the runbook actions and documentation items you would produce for the 'producer backlog' incident scenario where messages are piling up. Include triage checks, mitigation steps, and longer term changes you would document.

Sample Answer

Situation: Producer backlog—messages piling in queue causing consumer lag, potential OOM/disk fill, SLO risk.

Triage checks (fast facts to collect)- Queue metrics: depth, enqueue rate, dequeue rate, oldest message age, retention usage.- Consumer health: pod count, CPU/memory, restart rate, thread pool metrics, consumer lag per partition.- Producer health: recent error rates, throttling, retries, message size distribution.- Broker health: disk %, network I/O, controller state, ISR, under-replicated partitions.- Alerts and timelines: when did backlog start, correlated deploys/traffic spikes.- Dead-letter queues (DLQ): are errors accumulating?- Authentication/authorization/ACL errors in logs.

Immediate mitigation (order-of-operations)1. Declare incident, page owners (broker, consumer, producer teams).2. Apply short-term consumer scale-up: increase consumer replica count or consumer threads if safe.3. If scaling blocked, throttle producers (rate-limit) or enable backpressure/config to reduce input rate.4. If broker resource constrained: add disk, free space, increase retention temporarily, or add broker capacity.5. Prioritize critical messages: create temporary consumer that processes high-priority topics or set up expedited path.6. Pause non-essential producers (batch jobs, analytics) via feature flags or traffic shaping.7. If message size spike: reject or truncate oversized messages at producer side.8. Monitor progress: confirm dequeue rate > enqueue rate and oldest message age decreasing.9. If data corruption or poison messages: move offending messages to DLQ and resume consumers.

Communication & runbook checklists- Incident status updates cadence and owners.- Rollback playbook for recent deploys.- Criteria for declaring incident resolved.

Post-incident and longer-term documentation/actions- Root cause write-up: timeline, contributing factors.- Capacity planning: broker storage, consumer CPU/memory headroom, partition counts.- Automated scaling: HPA/consumer-autoscaler based on consumer lag and queue depth.- Producer-side hard limits: message size quotas, client-side rate limiting, circuit breakers.- Backpressure patterns: implement retries with exponential backoff, client backoff signals.- Alert tuning: add SLO-based alerts (queue depth thresholds by priority, oldest message age).- Runbook updates: play-by-play mitigations, safe scaling commands, verified rollback steps, contact matrix.- Automation: scripts to quiesce producers, spawn emergency consumers, and move messages to DLQ.- Tests & exercises: chaos tests for consumer failures, load tests to validate autoscaling.- Preventive monitoring: dashboards for end-to-end latency, per-topic throughput, and consumer lag.

This runbook balances immediate safety (throttle/scale) with durable fixes (capacity, backpressure, automation).

MediumTechnical

57 practiced

Produce an observability mapping for a search service that has user-facing frontends, indexers, query API, and a Redis cache. For key user journeys (search query, indexing), map SLIs to concrete metrics, traces, and logs, and propose alert thresholds and retention policies to include in the architecture docs.

Sample Answer

Overview: map two user journeys—Search Query (user → frontend → query API → Redis cache → index/query nodes) and Indexing (ingest → indexer → commit → replication)—to SLIs, concrete metrics, traces, logs, alert thresholds and retention recommendations.

SLIs (examples to include in architecture docs)- Search Query SLOs: - Availability/success rate: % of queries returning 200 with non-empty result (target 99.9 monthly). - Latency: p95 end-to-end user-visible query latency (target <200ms). - Freshness (for recent changes visible): % of queries reflecting index updates within X seconds (target 99% within 30s).- Indexing SLOs: - Index throughput: docs/sec vs expected ingest rate. - Index lag: time between ingest and document becoming searchable (target 99% <60s). - Indexer success rate: batch job success % (target 99.5).

Concrete metrics (instrumentation)- Frontend: - http_request_total{status,endpoint}, http_request_duration_seconds (histogram) - request_errors_total, request_rate - traces: span "frontend.request" with request_id, user_id, route- Query API: - query_requests_total, query_latency_seconds_bucket (histogram), query_errors_total - cache_hit_ratio = redis_hits / (redis_hits + redis_misses) - redis_latency_ms, redis_commands_total, redis_evictions - backend_query_latency, backend_errors - traces: span "query-api" calling redis and index node; propagate request_id- Indexer: - ingest_rate_docs_per_sec, indexing_latency_seconds (per-doc), commit_latency, index_queue_depth - index_replication_lag, indexing_errors_total - traces: span "indexer.batch" with batch_id, size, duration- Infra: - CPU, memory, disk I/O on index nodes, redis memory_used, GC pauses, connection counts - K8s pod restarts, OOM kills, node disk pressure

Traces (distributed tracing)- Trace flow: frontend.request → query-api.span → redis.span (if cache miss triggers) → index-node.span- Required span tags: request_id, user_id (hashed), endpoint, query_hash, cache_hit(bool), shard/id, error code- Sampling: 100% for errors, 1–5% for successful requests; on-demand tail-based tracing for traces exceeding latency thresholds.

Logs (structured)- Use JSON logs with fields: timestamp, request_id, service, level, message, latency_ms, status_code, endpoint, user_id_hash, query_hash, error_stack- Search query logs: log on error, cache miss events, top-N slow queries (latency > 500ms)- Indexer logs: batch start/end with counts, failures with document IDs (or hashes), commit confirmations- Correlate logs with traces via request_id and trace_id.

Alerts and thresholds (examples, tune to traffic)- P0 (pager): - Query success rate < 99.7% over 5m OR p95 latency > 1s over 5m - Indexer failure rate > 1% over 5m OR index lag > 300s (major) - Redis down or connection failures > 5% requests- P1 (page/notify): - Cache hit ratio drops > 20% vs baseline for 10m - Index queue depth > threshold (e.g., > 2× normal) for 10m - Pod crashloop or > 3 restarts in 15m- P2 (ticket): - Gradual drift: p95 latency increase 30% week-over-week, non-urgent errors

Alert rules: use rolling windows, require sustained violation (e.g., 5m for P0) and include runbook links and playbook steps (restart, failover to replica, roll back deploy, investigate trace samples).

Retention policy (practical tiers)- High-resolution metrics: 30 days at 1s–15s granularity- Medium-resolution metrics: 365 days aggregated (1m–5m)- Traces: store full traces 7–30 days (errors & sampled high-latency traces 90 days if cost allows). Keep trace index for quick lookup ~90 days.- Logs: hot storage (30 days searchable), cold storage (archive) 365–730 days depending on compliance. Keep error/exception logs at longer retention (90–365d).- Index of slow queries and top offenders: 365 days aggregate.

Additional notes- Define cardinal SLOs and an error budget policy (e.g., 0.1% monthly error budget); tie to release gating.- Instrument synthetic canaries: scheduled synthetic queries to measure availability, latency, and freshness; treat their failures as high-severity alerts.- Privacy: hash/anonymize PII in logs/traces per policy.- Include dashboards: SLO dashboard (latency, availability, error budget), Cache dashboard (hit ratio, latency), Indexing dashboard (lag, throughput, queue depth), Infra health.

This mapping should be added to architecture docs with concrete metric names, alerting thresholds, runbook links and owners for each alert.

HardSystem Design

52 practiced

Design and document a progressive delivery plan using canaries and feature flags across multiple regions. Include traffic routing diagrams, metrics to watch for each canary stage, rollback criteria, and how you will communicate rollout status and automated gates to oncall and product teams.

Sample Answer

Requirements & constraints:- Deploy new feature across 4 regions with zero-downtime, safety gates, and fast rollback.- Use feature flags (per-region & user-segment), canary routing, automated metric gates, and human approvals for risky stages.- Target SLO: 99.95% availability; preserve error budget.

High-level architecture (ASCII):User -> Global LB (DNS/Anycast) -> Traffic Router (Traefik/Envoy + Service Mesh) | ------------------------------- | | | | RegionA RegionB RegionC RegionD | | | | Canary Pod Stable Pod ... Canary Pod (flag=on) (flag=off)

Progressive rollout plan (stages):1) Local smoke (RegionA single pod): 0.1% region traffic, internal users only (dev/test). - Duration: 15–30m, automated gates pass -> next.2) Region canary (RegionA 5% region traffic): external users subset (10% by cookie/ID). - Duration: 1–2 hours, automated gates + watch.3) Multi-region stagger (RegionB 1% then 5%): add regions one-by-one over 4 hours.4) Global ramp: 25% -> 50% -> 100% across all regions over 8–24 hours, with cooldowns.

Traffic routing diagram (percentages):Global LB: route 1% -> RegionA Canary, 99% -> StableRegionA: Canary 5% of region traffic -> new versionWhen ramping: increase Global -> RegionX Canary percentages per stage.

Metrics & automated gates (per canary stage):- Availability: HTTP 5xx rate <= baseline + 0.5% absolute- Latency: p95 <= baseline * 1.2- Error budgets: Error rate consumes < 20% of allowed burn in 1h- Business metrics: key transactions (checkout, login) conversion delta <= -2%- Resource metrics: CPU, memory, GC pauses within +30% of baseline- Traffic integrity: request success rate, response schema validationAutomated gate logic: compare canary to control using statistical tests (e.g., sequential probability ratio test / Bayesian A/B). Fail on significance at 95% or threshold breach.

Rollback criteria (automated):- Any single metric breaches hard threshold (e.g., 5xx > baseline +1%) => immediate rollback to previous flag state and scale down canary.- Sustained degradation: metric breach for >5 minutes (configurable).- Business KPI drop >5% in 15m.Rollback enforcement: orchestration script toggles feature flag off + traffic router moves 100% to stable; automated incident created.

Manual guardrails:- Require oncall acknowledgement for stage transitions beyond region canary.- Product must sign off before global >50%.- Emergency kill switch in dashboard (flag toggles + global circuit breaker).

Communications & status:- Single source-of-truth rollout page (Confluence/GitOps release notes) + Slack #rollout channel with automated bot updates: - Stage start/complete, metric summary, pass/fail, link to dashboards.- Oncall notifications: - Automated alerts to PagerDuty when automated gate fails or rollback occurs. - Quiet channel for noncritical stage completions.- Product notifications: - Email and Slack summary at each major milestone (region added, 50%, 100%).- Dashboard widgets (Grafana) per region showing canary vs control, with green/yellow/red gates and one-click rollback.

Automation & implementation details:- Feature flags via LaunchDarkly/FF4J with region & user-segmentation toggles.- Traffic routing via Envoy with weighted clusters controlled by CI/CD (Argo Rollouts/Flagger).- Canary analysis using Prometheus metrics + Kayenta or Flagger for automated statistical analysis.- GitOps: rollout manifests and flag states stored in repo; changes applied via pipeline; audit logs retained.

Post-rollout:- 24–72h monitoring window for delayed regressions.- Blameless postmortem if rollback occurred: timeline, root cause, mitigations.- Gradual cleanup: remove temporary flags after stability for traceability.

Operational runbook summary (condensed):- Start stage -> automated gate monitors metrics -> gate passed -> notify oncall/product + proceed -> gate fail or threshold breach -> auto-rollback + page oncall + postmortem workflow.

This plan balances automated safety, regional control, and clear human escalation paths to ensure reliable multi-region progressive delivery.

Unlock Full Question Bank

Get access to hundreds of System Architecture Communication and Documentation interview questions and detailed answers.

Join thousands of developers preparing for their dream job.