Architecture Decision Documentation and Communication Questions

Covers the practices for capturing, organizing, and communicating architectural decisions and the rationale behind them. Candidates should be able to describe how to create architecture decision records and design documents, record alternatives considered, list pros and cons, and show impacts on scalability, cost, maintainability, security, and operations. This topic also covers techniques for communicating decisions to engineers, product managers, and non technical stakeholders, obtaining buy in, handling feedback and dissent, and evolving documentation as requirements change. Interviewers may probe how candidates link decisions to requirements, trace implications across components, and ensure decisions are discoverable and revisited when assumptions change.

MediumTechnical

28 practiced

As a mid-level engineer, you receive feedback opposing your ADR choice from another team. Outline a step-by-step process for handling dissent that preserves relationships and leads to a resolution: include how to gather data, update the ADR, and escalate if necessary.

MediumSystem Design

37 practiced

Design an ADR that records the decision to adopt a circuit-breaker and bulkhead strategy across a set of internal services. The ADR should name libraries or frameworks considered, expected metrics (error rates, open-circuit count), and fallback behavior. Include two alternatives and their operational trade-offs.

Sample Answer

Title: Adopt circuit-breaker + bulkhead pattern for internal service-to-service calls

Status: AcceptedDate: 2025-11-22Context:- Several internal services call shared downstreams (auth, billing, search). Incidents show cascading failures: slow downstream causes timeouts and thread exhaustion, impacting unrelated request types.- Goal: improve resilience and isolate failures while preserving throughput for healthy flows.

Decision:- Adopt circuit-breaker per remote dependency and bulkhead isolation per caller logical group.- Circuit-breaker implementation: Resilience4j (Java) as primary; Polly (C#) and aiobreaker/tenacity for Python services. For Node.js consider opossum.- Bulkheads: thread-pool / semaphore-based bulkheads in application (Resilience4j supports both). Kubernetes-level resource quotas and separate Deployments for heavy/critical flows where appropriate.

Expected metrics (to be collected and alerted):- Error rate per (caller→callee) 5m and 1h windows (%)- Request latency P50/P95/P99- Open-circuit count and duration per circuit- Consecutive failures and failure rate thresholds- Rejected-by-bulkhead / queue-length / thread-pool-saturation- Successful fallbacks count- Mean time to close circuit after open

Configuration defaults:- Sliding window: 60s, minimum 50 calls, failure threshold: 50% → OPEN- Open state wait: 30s then HALF-OPEN probe- Bulkhead size: start with 20 concurrent; queue 50 requests (tunable per service)

Fallback behavior:- Synchronous calls: graceful degradation—return cached response (if available), a reduced payload, or HTTP 503 with Retry-After and user-facing message.- Asynchronous flows: enqueue work for later processing and emit a telemetry event.- All fallbacks must log reason and metric tag indicating "fallback_used".

Alternatives considered:1) Client-side only rate-limiting without circuit-breakers - Pros: simpler; prevents overload - Cons: doesn't detect downstream failures pattern; may still tie up caller threads; no recovery probing; weaker observability.2) Service mesh-level resilience (e.g., Istio/Envoy circuit-breakers + mTLS) - Pros: centralized policy, consistent across languages, no client library changes. - Cons: operational complexity, limited app-level context for fallbacks/caching, slower rollout, extra latency and control-plane dependency.

Operational trade-offs:- Library approach (chosen): greater control and ability to implement application-specific fallbacks and caching; requires per-language integration and config management.- Mesh approach: easier to enforce policies at scale but loses fine-grained application semantics and increases infra surface area.

Rollout plan:- Stage 1: Implement libraries in two non-critical services; expose metrics to Prometheus and dashboards; alert on open-circuit and rejected-by-bulkhead.- Stage 2: Tune thresholds in staging with chaos tests (latency/failure injection).- Stage 3: Gradual rollout (canary) to remaining services; run with feature flag; document patterns and library wrappers to standardize usage.

Consequences:- Reduced blast radius and fewer cascading failures; added operational configs to maintain; requires monitoring, runbook updates, and periodic threshold review.

MediumSystem Design

37 practiced

Write an ADR for a caching decision in a read-heavy service that handles 50,000 reqs/s with 20% cache hit target. Your ADR should specify TTL strategy, invalidation approach, consistency model, cache tier (in-process, distributed), and operational rollback plan. Include at least two alternatives and pros/cons for each.

Sample Answer

Title: ADR — Caching strategy for read-heavy service (50k req/s, 20% cache hit target)Decision: Use a two-tier cache: distributed Redis cluster (primary) + bounded in-process LRU for hot-object amplification.

Context- Service receives ~50k read requests/sec. Target cache hit rate: 20% to reduce backend load and latency.- Data is mostly read-mostly with occasional writes/updates (low-to-medium write rate).- Strong consistency not required; stale reads acceptable up to a short window.

Decision1. Cache tier - Distributed cache: Redis Cluster (clustered, TLS, replication, 3 shards min). - In-process cache: bounded LRU (e.g., Caffeine for Java) with max size per instance to avoid OOM.

2. TTL strategy - Redis: conservative short TTLs (60–300s) depending on object volatility. Use per-key TTL based on data class: - Class A (highly volatile): 60s - Class B (moderately volatile): 300s - Class C (rarely changes): 3600s - In-process: TTL = min(Redis TTL, 120s) + LRU eviction.

3. Invalidation approach - Writes update authoritative datastore, then: - Publish invalidation messages on Redis Pub/Sub (or Redis Streams) with object IDs. - Subscribers: all app instances listen and evict from local caches; also remove key from Redis (del) or set short TTL. - For fail-safe, use versioned keys (objectID:version) to avoid race conditions.

4. Consistency model - Eventual consistency with per-object monotonic versioning. Reads may be stale within TTL window. Strong consistency supported for critical endpoints by bypassing cache or performing read-after-write (optional).

5. Operational & rollback plan - Gradual rollout: shadow caching (collect metrics) → 10% of traffic → 50% → 100%. - Monitoring: cache hit rate, Redis latency, error rates, backend QPS, p95/p99 latencies, memory usage, Pub/Sub lag. - Rollback steps: 1. Switch traffic to bypass cache via feature flag / config (immediate). 2. Disable invalidation publisher to stop churn. 3. Scale down/flush Redis if required. 4. Revert TTLs/in-process config via deployment. - Safety: circuit-breaker to auto-bypass cache if Redis latency > threshold or error rate spikes.

Alternatives (with pros/cons)A) Single-tier distributed cache (Redis only, no in-process) - Pros: Simpler, single source of truth, easier invalidation. - Cons: Higher network overhead and latency per read; needs more Redis capacity to meet 50k req/s with 20% hit; single network bottleneck.

B) CDN / edge caching (for public-read endpoints) + origin cache - Pros: Offloads significant read traffic, lower latency for global users. - Cons: Not applicable for personalized/private data, longer invalidation times, cost and complexity.

C) Write-through / write-back cache - Pros: Write-through keeps cache warm; write-back reduces datastore writes. - Cons: Write-through adds write latency; write-back risks data loss and complex eviction/consistency logic.

Rationale- Two-tier (Redis + local LRU) balances latency and backend load: local cache serves hottest keys with minimal latency; Redis provides shared cache to reach 20% hit without excessive local duplication.- Pub/Sub invalidation + versioned keys minimize stale-read windows while keeping implementation reasonably simple.

Metrics to validate- Achieve >=20% global hit rate, backend QPS reduction proportional to hits, p95 latency improvement, acceptable cache memory utilization, and Pub/Sub propagation <1s.

This ADR can be revisited if write rates increase, hit targets change, or we require stronger consistency.

MediumTechnical

33 practiced

Write an ADR skeleton for choosing single-region vs multi-region deployment for a service with 100ms global latency target and 99.95% availability SLO. The ADR must evaluate latency, failover time, replication lag, operational complexity, and monthly cost. What data or experiments would you collect to support the decision?

Sample Answer

Title: ADR-2025-XX — Single-region vs Multi-region deployment for Service X

Status: ProposedDate: 2025-11-22Authors: [Your Name], SRE, Product

Context- SLOs: 100ms end-to-end global latency, 99.95% availability.- Traffic: global users across NA/EU/APAC; peak RPS ~X.- Data consistency: mostly read-heavy, occasional writes with per-user affinity.

Decision Summary- Evaluate two options: Single-region (primary region + CDN + read replicas) vs Multi-region active-active (or active-passive) with geo-routing.- Preliminary recommendation: Multi-region active-passive if measured network RTTs and failover tests show single-region can't meet 100ms for target geos or cannot meet 99.95% during region outage without unacceptable RTO.

Evaluation Criteria (must be quantified)1. Latency - Measure median/95/99 latency from each geo to candidate regions (including CDN + app + DB). - Target: 95th percentile ≤100ms.2. Failover time - RTO target to maintain 99.95% availability: max allowed outage per month ≈ 21.6 minutes. Translate to allowable failover time and error budget.3. Replication lag - For active-passive/replicated DBs, maximum acceptable lag (e.g., <100ms for user experience; or <1s if tolerable).4. Operational complexity - Number of runbooks, team-hours, on-call pages, changes to CI/CD, schema-migration complexity.5. Monthly cost - Cost model: infra hours, cross-region egress, replication overhead, extra monitoring, and staffing.

Analysis (how to compare)- Latency: Synthetic client probes (real browsers + headless) from representative cities (LA, NY, London, Singapore, Sydney). Record full tail percentiles and percent of requests >100ms.- Failover: Chaos drills — simulate region outage, measure service error rate and failover duration, and impact on SLO.- Replication lag: Load tests with write patterns; measure DB replication lag under normal and peak writes.- Cost: Build cost projection for 3-month steady-state and 1-month peak: compute, storage, cross-region egress, CDN, DB replicas, monitoring, and estimated additional on-call staffing.- Operational complexity: Estimate MTTR from past incidents, number of runbooks, and additional PR review overhead; survey SRE dev-hours to operate multi-region.

Proposed Experiments / Data Collection- Synthetic latency matrix: 1M requests over 48 hours per candidate region; compute p50/p95/p99.- Load + write-fanout test: Ramp writes to peak, measure DB replication lag and data loss windows for active-passive.- Failover drill: Controlled region shutdown during low-traffic window; measure failover time and error budget consumption.- Cost experiment: Deploy minimal multi-region footprint, measure real egress & operation for 1 month.- Canary traffic split: Route 5–10% traffic to alternative region and monitor errors, latency, and consistency anomalies.

Consequences / Trade-offs- Single-region: Lower cost, simpler ops, but likely fails 100ms for distant geos or consumes error budget during regional incidents.- Multi-region: Higher cost and operational burden; better latency and availability; complexity in consistency, schema changes, and testing.

Rollout Plan- Phase 0: Collect metrics (latency matrix, replication tests).- Phase 1: If single-region fails criteria, implement active-passive with automated failover and read replicas near heavy geos.- Phase 2: Iterate to active-active if needed and if budget/ops permit.

Monitoring & Alerts- Geo latency dashboards, replication lag alerts, failover health, error budget burn rate.

Decision criteria- Choose single-region if measured p95 ≤100ms for ≥95% global traffic and failover drills keep monthly downtime <21 minutes with low error-budget burn and projected cost <X% greater than multi-region.- Otherwise choose multi-region.

Appendix: data collection templates, runbook checklist, cost spreadsheet links, experiment scripts.

MediumTechnical

55 practiced

You're asked to lead an ADR review meeting for a change that modifies the auth flow used by multiple services. Draft an agenda that ensures technical concerns, security impact, product requirements, rollout plan, and rollback plan are covered. Explain who should present each section and the decision rules for accepting the ADR.

Unlock Full Question Bank

Get access to hundreds of Architecture Decision Documentation and Communication interview questions and detailed answers.

Join thousands of developers preparing for their dream job.