Architectural Decision Making Questions

Assess how a candidate thinks through major system and technical decisions, including selecting architectures, technologies, and technical strategies. Expect discussion of evaluation criteria such as performance, reliability, scalability, complexity, cost, development velocity, team capability, maintenance burden, and long term evolution. Candidates should explain specific past decisions with clear articulation of the options considered, trade offs accepted, risk mitigation, observed consequences over time, what they would change with current knowledge, and evidence of nuanced judgment when balancing competing priorities. For senior and staff levels, this includes demonstrating influence across teams when making architecture calls, recognizing organization level costs of choices, and surfacing hidden operational or people costs.

HardTechnical

30 practiced

Discuss the hidden operational and long-term costs of adopting polyglot persistence across services (using many different database engines). Cover developer onboarding, backup and restore complexity, schema migration challenges, incident response, and propose mitigations and guardrails.

Sample Answer

Polyglot persistence can give the right tool for each job, but it hides significant operational and long-term costs. Below I outline those costs across the requested areas and concrete mitigations/guardrails.

Developer onboarding- Cost: Steeper ramp—engineers must learn different APIs, query languages, data modeling philosophies (relational vs document vs wide-column), client libs, and operational quirks. This increases hiring bar and slows feature delivery.- Mitigation: Maintain a concise “data platform playbook” with idiomatic patterns, example projects, and common anti-patterns. Standardize SDK wrappers and shared libraries to provide uniform CRUD/transaction semantics where possible.

Backup & restore complexity- Cost: Different engines have different snapshot semantics, consistency guarantees, retention tooling, and restore workflows; coordinating multi-db point-in-time recovery is hard and error-prone.- Mitigation: Define RPO/RTO per service, group services into recovery domains, and automate backup/restore via infrastructure-as-code. Use transactional change-capture to a central log (e.g., Kafka) to rebuild state when full restores are infeasible.

Schema migration challenges- Cost: Heterogeneous migration tooling and online-schema-change capabilities vary; some stores lack safe in-place migrations, causing downtime risk and complex dual-write or version negotiation logic.- Mitigation: Enforce backward/forward-compatible schema evolution rules, require feature-flagged rollouts, and mandate migration plans in PRs (including rollback). Prefer tolerant schemas (e.g., nullable fields, versioned documents) and provide shared migration tooling (migrations as code).

Incident response- Cost: Troubleshooting spans multiple consoles, metrics, and failure modes (query hotspots, eventual-consistency surprises, compaction/GC pauses). Runbooks multiply and on-call cognitive load grows.- Mitigation: Create unified observability (traces, structured logs, SLOs) with standardized health/emergency endpoints per datastore. Produce consolidated runbooks and capacity-playbooks; run regular chaos drills across the mixed stack.

Governance & long-term technical debt- Cost: Proliferation of engines increases licensing, patching, security surface, and variant-specific technical debt. Cross-service joins and analytics become costly.- Mitigation/Guardrails: - Approve new engines via a data-platform council with cost/operational risk assessment. - Limit official engines to a curated set; require business justification and exit plan for exceptions. - Centralize cross-cutting concerns: authentication, backup orchestration, monitoring, and schema/migration registries. - Track total cost of ownership (ops hours, license cost, incident MTTR) per engine and review quarterly.

Practical example- If a team chooses a time-series DB for metrics and a document DB for user profiles, require them to: - Document why a relational DB couldn’t meet the need - Provide a migration plan and backup/restore test - Implement SDKs that expose uniform retry/timeout semantics - Add SLOs and alert thresholds to shared dashboards

Bottom line: polyglot persistence is powerful but operationally expensive. Reduce risk with clear approval gates, shared tooling, uniform abstractions, automated backup/migration tests, and ongoing cost/incident telemetry so choices are reversible and maintainable.

MediumTechnical

31 practiced

Design retry, exponential backoff, and circuit-breaker policies for interacting with a third-party payment gateway to strike a balance between user experience and system stability. Include idempotency strategies for payment requests, how you would surface failures to users, and metrics to tune thresholds.

Sample Answer

Requirements & goals:- Reliable, low-latency payment flow for users; avoid double-charges; protect system and gateway from overload; surface clear failure states to users; gather metrics to tune behavior.

Retry + Exponential backoff:- Only retry on transient errors (network timeouts, 5xx from gateway, connection refused). Do NOT retry on 4xx (card declined, invalid request).- Strategy: up to 3 retries with jittered exponential backoff: delays = base(200ms) * 2^i + random(-50..+50) → 200ms, ~450ms, ~950ms. Cap backoff at 2s.- Use async, non-blocking retries so UI can show progress and not block threads.

Circuit breaker:- Per-gateway circuit breaker with states: CLOSED → (on N failures within T) → OPEN for cool-down → HALF-OPEN (probe).- Example thresholds: if ≥5 failures within 60s → OPEN for 60s. In HALF-OPEN send 2 probe requests; if both succeed, CLOSE; else OPEN again with exponential OPEN window (double each retry up to 15m).- Fail fast while OPEN: return deterministic user-facing error and queue telemetry.

Idempotency:- Require client/server-generated idempotency keys for payment intents (UUID v4). Store request results keyed by idempotency key and request signature (amount, currency, merchant). If duplicate request seen, return stored result.- Persist idempotency keys with status (PENDING, SUCCESS, FAILED) and expiry (48–72h). Use strong consistency for lookups.

User experience & surfacing failures:- For in-progress retries, show “Processing payment… (attempt 2 of 4)” with cancel option. For card declines show immediate actionable message. For transient/system errors show “We’re having trouble processing payment — retrying” and after final failure show clear next steps (try again, contact support) with request ID and timestamp.- Provide fallback flows: save payment method and offer retry-at-will or alternative payment methods.

Metrics to tune thresholds:- Success rate, error rate by type (4xx vs 5xx), latency P50/P95/P99 for payment calls, retry counts per transaction, circuit-breaker open rate and duration, idempotency key collision/duplicate counts, user-visible failure rate (payments failed after retries). Alert on sudden increase in 5xx or P99 latency.- Use these to adjust retry counts, backoff caps, and circuit thresholds. Example: if P95 latency > 1s, increase cap on backoff or reduce retries; if gateway 5xx spike > 3% sustained for 5m, lower retry attempts and increase circuit sensitivity.

Trade-offs:- Fewer retries = faster user feedback but more failed payments; more retries = better success but higher latency and risk of gateway overload. Circuit breaker protects both sides; idempotency prevents double-charges.

HardSystem Design

23 practiced

Design tenant isolation strategies for a multi-tenant SaaS product. Compare logical isolation (shared DB + tenant_id) versus physical isolation (separate DB/cluster) across security, scalability, cost, compliance, and operational complexity. Recommend a strategy for a startup anticipating enterprise growth and explain transition considerations.

Sample Answer

Requirements & constraints (assumed): multiple tenants, low initial cost, rapid feature delivery, expected growth to enterprise customers with stricter security/compliance, need for reasonable performance and ability to isolate noisy tenants.

High-level comparison

1) Logical isolation (shared DB, tenant_id)- Security: Isolation by application layer; enforce tenant_id checks, row-level security (RLS) where supported (e.g., Postgres RLS). Risks: buggy code or SQL injection can cause cross-tenant access.- Scalability: Easy to scale reads with replicas and caching. Write scaling limited by single DB instance; can shard by tenant key later.- Cost: Lowest cost — single DB instance/cluster, simpler backups.- Compliance: Can meet some standards with encryption, strong access controls, and audit logging, but stricter standards (PCI scoped per-tenant, some HIPAA requirements) may prefer stronger separation.- Operational complexity: Simpler day‑to‑day ops initially; complexity arises in multi-tenant query optimization, noisy-neighbor mitigation, and schema migrations affecting all tenants.

2) Physical isolation (separate DB per tenant or per cohort/cluster)- Security: Stronger natural isolation — a compromised app credential cannot access other tenants' data at the DB level. Easier for tenant-specific access controls and audits.- Scalability: Better per-tenant performance isolation; you can scale or move heavy tenants independently. More complex to operate at large scale.- Cost: Higher cost (many DB instances, wasted capacity for small tenants). Costs grow with tenant count.- Compliance: Easiest to satisfy stringent compliance/regulatory/customer contractual demands.- Operational complexity: Higher — provisioning, backups, monitoring, schema migrations per-tenant or orchestrated across many DBs; requires automation.

Recommendation for a startup anticipating enterprise growth- Start with a robust logically-isolated architecture designed for safe multi-tenancy: - Use tenant_id + Postgres RLS or equivalent to enforce DB-level access checks. - Enforce parameterized queries/ORM-level tenant scoping. - Encrypt data at rest and in transit; use per-tenant encryption keys if possible. - Implement quotas, throttling, and monitoring/alerting to detect noisy tenants. - Deploy observability: per-tenant metrics, audit logs, and anomaly detection. - Abstract tenant-to-datasource mapping via a Data Access Layer (DAL) or tenancy service and feature flags.- Plan for gradual migration to physical isolation for enterprise customers or when cost-per-tenant justifies it: - Design schema and app so queries are tenant-agnostic (no embedded global joins) and migrations are backward compatible. - Implement a routing layer: tenancy service that maps tenant -> DB/cluster. - Automate provisioning (IaC) to spin up DB instances or clusters per enterprise tenant. - Use logical separation for small/mid tenants; offer “isolated” plans that migrate a tenant to its own DB. - For migration: use logical replication or chunked export/import, apply schema migrations in the target DB, and switch routing atomically (feature-flagged). Run dual-write or read-redirection during cutover to validate. - Test migration paths early with a non-critical tenant to iron out performance and operational steps.

Transition considerations & trade-offs- Migration difficulty: moving large tenants is operationally heavy — build automation and test it early.- Data model choices: avoid cross-tenant foreign keys and global sequences; use UUIDs or tenant-prefixed keys to ease splits.- Backups & restores: per-tenant restore is simpler with physical isolation; emulate with logical backups filtered by tenant for shared DB.- Security posture: for enterprise SLAs and compliance, be ready to offer dedicated databases/clusters and contractually enforce encryption/keys and auditability.- Cost optimization: mix models — shared DB for majority (cost-efficient), isolated DB/cluster for high-value enterprise tenants.

Bottom line: Start with secure, well-instrumented logical isolation to move fast and control cost, but design the system, telemetry, and automation from day one to enable selective migration to physical isolation for enterprise customers or high-resource tenants.

HardSystem Design

30 practiced

Propose a zero-downtime global schema change process for an application deployed across multiple regions and runtime languages. Explain the expand–migrate–contract pattern, deployment sequencing, feature flags, data migration strategies (online migrations, backfills), and verification steps needed to avoid downtime or data corruption.

Sample Answer

Requirements:- Zero user-visible downtime globally- No data loss or corruption- Support multi-region replication and clients in multiple runtime languages- Ability to roll back safely

High-level approach: use the Expand → Migrate → Contract pattern with careful deployment sequencing, feature flags, online-safe migrations, controlled backfills, and verification.

Expand (make additive, backwards-compatible changes)- Add new columns/tables/indexes (nullable or with defaults handled in app), new API fields.- Deploy DB schema change using online DDL (pt-online-schema-change, native non-blocking ALTER, or rolling-create table + copy).- Deploy library/SDK changes and server code that write both old and new fields behind a feature flag (write-through): clients start populating new columns while still reading old ones.

Migrate (move traffic/reads to new schema)- Flip a feature flag to enable readers to prefer new fields only after writers have populated them and backfills complete.- Run incremental, idempotent backfills in small batches per region to populate historical rows; use job queues with rate limiting and resume tokens to avoid load spikes and ensure progress.- For multi-region, coordinate by region: backfill local replica where data resides first, then rely on replication to propagate changes, or run per-region backfills if divergent.

Contract (remove old schema)- After verification that no reads depend on old fields and replication lag is zero, schedule removal of old columns/indexes.- Perform removal via online DDL and deploy code that no longer references old schema.

Deployment sequencing and rollouts- Roll out schema changes first to DB (additive).- Roll out multi-language server/client libraries that can handle both schemas; use canary by region and by client version.- Enable writer flag in canary regions; monitor error rate, latency, replication lag.- Gradually widen rollout; enable reader flag only when safe.

Feature flags- Use tri-state flags: disabled, can-write (write both), prefer-new (read new with fallback), enabled (read-only new).- Tie flags to telemetry and automated rollback on error thresholds.

Data migration strategies- Online migrations: use non-blocking DDL and copy-in-place strategies; ensure migrations are idempotent.- Backfills: resumable incremental jobs, shard by primary key range, run per region to reduce cross-region traffic; use write-consistent reads (read-after-write) to validate.- For transformations requiring recompute, prefer stream-based reprocessing (Kafka + consumers) with dead-letter handling.

Verification and safety- Strong automated checks: row-count parity, checksum comparisons, small-sample dual-read comparisons (read both fields and compare), consistency checks across replicas.- Monitor replication lag, error rates, and business metrics.- Maintain feature-flagged rollback paths for quick revert of reader/writer behavior without schema rollback.- Perform staged cutover windows and final verification before contract step; keep backups and migration logs for forensic rollback.

Trade-offs- Longer migration time for safety; additional storage for duplicate columns.- Complexity across languages demands common client libs or well-documented compatibility layers.

This process ensures each change is additive, verified, can be rolled back, and avoids global downtime or data corruption.

MediumSystem Design

32 practiced

Design a feature-flagging strategy to safely roll out an architecture-changing feature (for example, splitting a read/write path). Include flag types (boolean, gradual rollout), kill switches, rollback plan, monitoring, and how to avoid flag proliferation.

Sample Answer

Requirements:- Safely route traffic to new read/write split with ability to target users, ramp gradually, instant kill, and observability.- Low latency routing, minimal code churn, clear rollback, and avoid long-lived flags.

High-level approach:- Use a centralized feature-flag service + client SDKs in app servers and API gateway.- Place routing decision at the gateway or service layer so behavior change is localized.

Flag types and usage:- Boolean flag: “split_enabled” for quick ON/OFF per environment (staging/prod) — short-lived, emergency use only.- Percentage/gradual rollout: “split_ramp_pct” to send X% of traffic to new path using consistent hashing or bucketing by user-id/request-id.- Targeting flags: user, account, region, or header-based allowlist for canary customers and internal testing.- Mode flag: “split_mode” enum {shadow, active, active_with_fallback} to control whether new path runs in shadow (no effect on responses) before full activation.

Kill switches & rollback plan:- Global boolean kill switch on gateway that immediately routes all traffic to old path.- Per-service fallback: new path wraps calls with timed circuit-breaker → if error/latency > threshold it falls back automatically.- Rollback steps: 1. Flip kill switch (instant). 2. If partial issues, reduce split_ramp_pct to 0. 3. Roll back deployment if code-level bug persists. 4. Postmortem and data migration rollback if needed.

Monitoring & alerting:- Metrics: success/error rate, p50/p95/p99 latency, throughput, DB write consistency, queue lengths, data divergence checks.- Canary differential dashboards comparing new vs old path (shadow traffic) for key metrics and business KPIs.- Automated alerts if delta exceeds thresholds (e.g., error rate +3 sigma vs baseline).- Tracing: distributed traces tagged with flag-version to filter traces for routed requests.- Health endpoints that report flag state and circuit-breaker status.

Avoiding flag proliferation:- Lifecycle policy: every flag must have owner, creation date, purpose, expiry date.- Enforce TTL: default 30 days for temporary flags; require extension request to keep.- Code patterns: wrap flags behind strategy interfaces (FeatureFlag.getTreatment("split")) so tech debt is visible.- CI checks: PR must include flag cleanup plan; code reviewers verify no long-lived flags.- Central registry/dashboard showing active flags, owners, and last-used timestamp.

Operational practices:- Start in shadow mode with a subset of production traffic to validate behavior and metrics.- Gradual ramp: 0% → 1% → 10% → 50% → 100% with stable monitoring windows between steps.- Run end-to-end and consistency checks during shadow to detect divergence before writes go live.

Trade-offs:- Gateway routing centralizes control but adds latency and a single decision point; mitigate with lightweight SDK and caching.- Shadowing doubles execution cost temporarily; acceptable for safety.

This strategy gives controlled, observable rollout with immediate kill/automatic fallback and governance to prevent flag sprawl.

Unlock Full Question Bank

Get access to hundreds of Architectural Decision Making interview questions and detailed answers.

Join thousands of developers preparing for their dream job.