Architectural Decision Making Questions

Assess how a candidate thinks through major system and technical decisions, including selecting architectures, technologies, and technical strategies. Expect discussion of evaluation criteria such as performance, reliability, scalability, complexity, cost, development velocity, team capability, maintenance burden, and long term evolution. Candidates should explain specific past decisions with clear articulation of the options considered, trade offs accepted, risk mitigation, observed consequences over time, what they would change with current knowledge, and evidence of nuanced judgment when balancing competing priorities. For senior and staff levels, this includes demonstrating influence across teams when making architecture calls, recognizing organization level costs of choices, and surfacing hidden operational or people costs.

HardTechnical

32 practiced

Design a monitoring and alerting strategy that minimizes alert fatigue while ensuring SREs are notified of actionable incidents within two minutes. Discuss signal-to-noise reduction, alert classification, runbook integration, automated remediations, and organizational on-call considerations.

Sample Answer

Goal: notify SREs of actionable incidents within 2 minutes while keeping noise low so on-call stays effective.

Signal-to-noise reduction- Tiered signals: separate telemetry into health-critical (request latency, error budget burn, instance OOM) vs informational (config drift, low disk). Only health-critical pages SRE within 2 min.- Use aggregated, correlated alerts: group by service/host, dedupe repeated failures, and require multi-signal correlation (e.g., 5xx rate + queue depth spike) before paging.- Adaptive thresholds & anomaly detection: baseline per-service using rolling windows and safeties to avoid pages on predictable spikes (use statistical/ML detectors for unusual patterns).- Backoffs/coalescing: implement cooldown windows and exponential suppression for noisy flaps.

Alert classification & routing- Define severity levels: P0 (page → immediate), P1 (page with longer SLA), P2 (ticket), Info. Map to Escalation policies.- Include ownership metadata (service owner, playbook link) in alert payloads for automatic routing (PagerDuty/Alertmanager + service catalog).

Runbook integration- Attach concise runbook steps and RCA checklist in every page: what to check first, key dashboards, quick commands, rollback steps.- Health-check links & runbook snippets via ChatOps buttons (Slack/MS Teams) so on-call can execute steps fast.- Automate playbook validation: syntactic checks and test drills.

Automated remediation- Safe automated actions: automated restarts or auto-scale with guardrails (rate limits, circuit breakers, canary rollout).- Human-in-the-loop for high-risk fixes: require approve action via ChatOps within alert window.- Observe-and-rollback: after remediation, run verification probes and auto-rollback if metrics worsen.

Organizational & process considerations- Define SLOs and error budgets to drive alert sensitivity. Only page when SLO impact likely.- On-call design: small durable rotations, primary/secondary, clear escalation, async handoff docs.- Metrics and continuous tuning: track alert volume per on-call-hour, noise ratio (non-actionable/page), MTTX (detect/ack/resolve). Regularly prune/retune alerts in blameless reviews.- Training & gamified runbook drills to keep response time under 2 minutes.

Trade-offs: stricter suppression reduces false positives but may delay rare true positives — mitigate via multi-signal detection and low-noise “canary” pages. Use tooling: Prometheus/Alertmanager, Datadog/SignalFx, PagerDuty, ChatOps + CI for runbook tests.

MediumSystem Design

30 practiced

Design a disaster recovery (DR) strategy for a multi-region SaaS platform required to survive an entire region outage with RPO = 15 minutes and RTO = 1 hour. Document replication approach, DNS/failover automation, data consistency choices, and runbook/test strategy.

Sample Answer

Requirements & constraints:- Survive full-region outage; RPO ≤ 15 minutes, RTO ≤ 1 hour.- Multi-region active/passive (primary in Region A, cold-warm secondary in Region B).- Strong operational automation, repeatable runbooks, and regular testing.

Replication approach:- Primary → Secondary continuous replication using change-data-capture (CDC) streaming (e.g., Debezium→Kafka or cloud-native CDC like AWS DMS with CDC) for transactional databases. Ensure replication commits shipped within seconds; monitor lag.- For relational DBs: maintain a hot read-replica in Region B with async WAL/replica streaming. To meet 15-min RPO, configure ACKs and monitoring so replication lag < 1 minute typical; fall-back snapshots every 5–15 minutes for recovery of last-known state.- For object storage: use cross-region replication (S3 CRR or equivalent) with versioning and lifecycle; prioritize synchronizing new objects immediately and retain tombstones to avoid divergence.- For caches/ephemeral state: persist critical events to durable store before acknowledging; rebuild caches from primary-backed snapshot or change log on failover.- For queues: use cross-region replication (mirror topics) or durable storage-backed queues; ensure at-least-once semantics and idempotent consumers.

Data consistency choices:- Accept eventual consistency across regions for reads on secondary; write operations must go to the promoted primary.- Use transactional guarantees at primary; implement an outbox pattern so side effects are replayable on failover.- For critical user flows requiring strict consistency, route reads/writes to primary (via active routing or feature flags) or use synchronous replication where latency permits (trade-off).

DNS / failover automation:- Use global load balancing (GSLB) + health checks (e.g., AWS Route 53 failover + ALB/Global Accelerator or multi-cloud GSLB). Health checks: app-level (end-to-end synthetic transactions), infra-level (control-plane), and replication-lag check.- Automate failover with an orchestrator (Lambda/Cloud Function / Step Functions / Automation runbook) that: 1. Detects region failure via multiple alerts (monitoring + synthetic tests). 2. Validates replication lag < RPO threshold; if not, abort and surface manual decision. 3. Promotes read-replica to primary (DB promotion script, update DNS / global LB weights to point to Region B). 4. Rotate keys/certs if needed; update service discovery (Consul/ECS/EKS configs) and CI/CD targets. 5. Run smoke tests and health probes; only then switch traffic.- Use DNS TTLs low (60–120s) combined with GSLB to meet RTO. Keep health checks and route adjustments idempotent and logged.

Runbook & automation:- Maintain an automated runbook repository (executable Runbooks) stored in version control. Steps: - Detection and confirmation checklist. - Pre-cutover checks: replication lag, storage quotas, IAM roles, DNS TTLs, certs. - Promotion steps (DB promote, schema migrations if required, queue rebind, secret rotation). - Post-cutover validation (synthetic transactions, metrics). - Rollback / failback procedure.- Implement runbook automation with safe-guards: dry-run, canary traffic (10% → 100%), human approval gates for high-risk steps.

Testing strategy:- Quarterly tabletop exercises with cross-functional teams to validate decisions and roles.- Monthly game days: short (1–2 hour) failover rehearsals for specific subsystems (DB failover, object-store switchover).- Annual full failover drill simulating region outage; measure actual RTO and data loss.- KPIs: mean time to detect (MTTD), mean time to failover (MTTFo), measured RPO, and RTO; track and report after each drill.- Post-mortem and continuous improvement: record run durations, failure points, and update runbooks/automation accordingly.

Operational controls & monitoring:- Replication-lag dashboards, synthetic transactions, error budgets, and alerting with escalation paths.- Access controls: restrict promotion abilities to a small, auditable set of automation roles or humans with MFA.- Backups: regular immutable backups with retention policy independent of region replication.

Trade-offs and rationale:- Active-passive reduces cross-region write-contention and complexity while meeting RTO/RPO with robust automation.- Synchronous cross-region replication would lower RPO but adds latency and cost; chosen only for extremely critical data if acceptable.- Emphasize idempotency and outbox patterns to simplify recovery and avoid data duplication during replay.

This design provides deterministic, automated failover within the 1-hour RTO while keeping data loss under the 15-minute RPO, plus a disciplined testing and runbook practice to keep the team ready.

HardTechnical

30 practiced

A critical third-party library used across services has a severe security vulnerability. Describe your immediate triage steps to determine impact, coordinate cross-team fixes, establish short-term mitigations (e.g., canary patches, configuration toggles), and long-term prevention (dependency policies, SBOM).

Sample Answer

Situation: A widely used third‑party library in our stacks has a disclosed critical CVE that affects multiple client-facing services.

Immediate triage (first 1–4 hours)- Identify scope: Query SBOMs, dependency trees (snyk/OSSIndex/Dependabot reports), build manifests, and package-locks to enumerate impacted services and versions.- Risk classification: Map CVE severity, exploitability, public PoC, and exposure (internet‑facing vs internal) to assign priority.- Short blast: Notify incident channel, engineering leads, security, product and account teams with a concise impact summary and next steps.

Coordinate cross-team fixes (first 4–24 hours)- Create a triage board (JIRA/issue tracker) with tasks per service: upgrade, patch, test, rollback plan.- Form a small central response squad (security + platform + owners) to coordinate fixes, allocate engineers, and review PRs.- Set SLAs by criticality (e.g., public API: 6–12 hours to mitigation; internal batch job: 48 hours).

Short-term mitigations- Canary / staged rollout: prepare patched builds and deploy to canary / staging first with traffic shaping and increased observability.- Configuration toggles: use feature flags or disable vulnerable functionality where possible.- Network controls: apply WAF rules, firewall filters, or egress restrictions to block exploit vectors.- Monitoring & detection: add IDS signatures, increase logging, and set alerts for anomalous calls related to the library.

Testing & release- Require unit/integration tests exercising the vulnerability surface, automated security scans in CI, and a rollback plan.- Use blue/green or incremental canary with health checks; only promote when metrics are stable.

Long-term prevention- Enforce SBOM and automated dependency inventory for all projects, integrated into CI.- Adopt SCA tools (snyk, Whitesource) with PR gating and automated patch proposals.- Policy & governance: establish dependency lifecycle policy (approved registries, LTS versions, periodic upgrades), threat modeling for third‑party libs, and mandatory security review for transitive dependencies.- Contract/vendor process: subscribe to vendor advisories, require SLA for security patches in vendor contracts, and maintain hotfix playbooks.- Regular audits and tabletop exercises; measure MTTR and reduce via drills.

Outcome & metrics- Track time-to-detect, time-to-mitigate, percent services patched, and regression incidence. Communicate status to stakeholders and clients until closure.

Why this works: It balances immediate risk reduction, controlled rollout to avoid regressions, clear cross-team ownership, and systemic changes to prevent recurrence—aligned to architecture-level controls and delivery processes I’d design as a Solutions Architect.

HardSystem Design

26 practiced

Design a cross-team API versioning policy for a platform with hundreds of internal and external clients. Propose a versioning strategy (URL, header, semantic), deprecation windows, compatibility guarantees, testing and enforcement mechanisms, and a migration pathway for clients.

Sample Answer

Requirements & constraints:- Hundreds of internal/external clients, low-friction migrations, clear break guarantees, auditability, automated enforcement, minimal runtime branching for servers.

Versioning strategy (recommended hybrid):- Major-in-path + semantic in header: /v{MAJOR}/resource for clear routing and discoverability of breaking changes; X-API-Version: {MAJOR}.{MINOR}.{PATCH} (or Accept header) for non-breaking feature flags and gradual rollouts. Rationale: path isolates breaking changes without forcing per-request header for major routing; header allows fine-grained capabilities and server-driven negotiation.

Compatibility guarantees:- PATCH: bug fixes, no behavior change — guaranteed backwards compatible.- MINOR: additive, new optional fields and endpoints — clients on older MINOR must continue to work; server may respond with default behavior for clients not opting-in.- MAJOR: breaking changes — may change payload shapes/semantics; supported concurrently for deprecation window.

Deprecation windows:- Major versions: supported for 12 months + 3 months extended support for enterprise customers (configurable by SLA).- Minor versions: supported at least 90 days; notify earlier if scheduled removal.- Deprecation lifecycle: Deprecation announcement → SDK/docs bump + migration guide (T-90) → soft-deprecation warning headers/responses (T-30) → hard sunset (T=0).

Testing & enforcement:- Contract-first design (OpenAPI with codegen). Each release includes: - Backwards-compatibility checks via automated schema diff (openapi-diff) in CI. - Consumer-driven contract testing: Pact or equivalent; require failing consumer tests to block release. - Integration test matrix: run server against canonical client mocks for last N major versions. - Runtime enforcement: API gateway validates X-API-Version/path and injects warning headers; rate-limits old versions if needed. - Telemetry: log client version, error rates, and migration progress dashboards.

Governance & process:- API Working Group reviews major bumps; change request template including migration cost, rollout plan, and risk assessment.- Version registry catalog (internal portal) listing active versions, owners, deprecation dates.- Automated alerts to owners when older versions cross thresholds.

Migration pathway for clients:1. Discover: developer portal lists breaking changes, migration steps, sample diffs, and SDK updates.2. Feature flags: provide opt-in header values for MINOR capabilities to test without switching MAJOR path.3. SDK/tooling: publish client libraries with dual-version compatibility shims and code-mod scripts (codemods).4. Sandbox & compatibility tests: provide test harness and mock servers for clients to validate.5. Transition support: dedicated migration window with support tickets and metrics; enterprise customers get extended timelines and compatibility adapters if needed.

Trade-offs & rationale:- Path-based major versions give strong isolation at routing and caching layers; header allows flexible feature negotiation without URL churn. Enforcing contract tests and telemetry lowers risk and provides measurable migration velocity.

MediumTechnical

22 practiced

Write a concise ADR (200-400 words) arguing for using eventual consistency with compensating transactions (sagas) for order fulfillment instead of two-phase commit. Include alternatives considered, risks, observability needs, and rollback/compensation examples.

Sample Answer

Title: Use Eventual Consistency with Compensating Transactions (Sagas) for Order Fulfillment

Context:We must coordinate inventory reservation, payment capture, and shipping across independent services (Inventory, Payments, Shipping, Orders). Strong transactional guarantees via distributed two-phase commit (2PC) are expensive, brittle across heterogeneous platforms, and impede availability.

Decision:Adopt eventual consistency implemented with orchestrated sagas (orchestrator service) and compensating transactions for failures. Each step publishes events and advances saga state; failures trigger compensating actions to restore business invariants.

Rationale:- Availability & latency: Saga avoids global locks and blocking coordinator; services remain autonomous and scalable.- Heterogeneous systems: Many external partners (payment gateways, 3rd-party shippers) don’t support XA/2PC; sagas integrate via idempotent APIs and events.- Operational simplicity: Reduced dependency on distributed transaction middleware; easier to evolve services independently.

Alternatives considered:- 2PC/Distributed Transactions: Provides atomicity but requires XA support, increases contention, is not cloud-native, and risks long-running locks and outages.- Eventual consistency with retry-only (no compensation): Simpler but cannot undo irreversible external actions (captured payments, shipped items).- Distributed Saga choreography: Pushes logic into services — reduces central coordinator but increases coupling and makes complex flows harder to reason about.

Risks & Mitigations:- Inconsistency windows: Mitigate with clear order states (PENDING, RESERVED, CHARGED, SHIPPED, CANCELLED) and user-visible status.- Compensating transaction failures: Implement retries, dead-letter queues, and manual remediation playbooks.- Partial failures and timeouts: Use time-based TTL for reservations and automatic compensations.

Observability requirements:- End-to-end tracing (distributed trace IDs), structured event logs, saga state machine dashboard, metrics (saga success/failure rates, compensation frequency), and alerts for stuck sagas or repeated compensations.

Example compensation:- Flow: Reserve inventory → Capture payment → Create shipment.- Failure at shipment creation: compensate by refunding payment (idempotent refund API) and releasing inventory reservation. Log compensation steps and surface to support if compensation fails after retries.

Decision owner: Solutions ArchitectureDate: 2025-12-06

Unlock Full Question Bank

Get access to hundreds of Architectural Decision Making interview questions and detailed answers.

Join thousands of developers preparing for their dream job.