Technical Ownership and Architectural Decisions Questions

This topic assesses a candidates ability to take technical ownership of systems and architecture and to drive high impact technical decisions from proposal through adoption and production. Candidates should be prepared to describe situations where they proposed and defended architectural changes or new frameworks, evaluated tradeoffs between competing approaches, prevented or remediated technical debt, and influenced technical strategy across teams or organizations. Include examples of leading projects end to end — designing solutions, guiding implementation, managing risks and tradeoffs (including between security and functionality), building consensus for controversial choices, and measuring the technical and business impact of those decisions. The description covers domain specific technical ownership such as security or cryptographic projects as well as broader system and platform architecture ownership.

MediumSystem Design

82 practiced

Design an error-budget policy that governs launches and reliability work. Define burn-rate thresholds, automated and manual actions (e.g., freeze launches, prioritize reliability engineering), communication flows with product owners, and how you would measure adherence and outcomes over time.

Sample Answer

Requirements and constraints:- Protect user-facing SLOs while allowing product velocity.- Fast, automated response for rapid degradation; manual governance for sustained risk.- Clear thresholds, actions, and communication path to product owners (POs).- Measurable outcomes and audit trail.

High-level policy (burn-rate tiers calculated as: actual_error_rate / allowed_error_rate over a rolling 28-day window; or computed over last 7/24 hours for short-term bursts):- Normal (burn-rate < 1): No restrictions. Continue launches and planned reliability work.- Elevated (1 ≤ burn-rate < 3): Automated notices to SRE, PO, and release owners. Require lightweight pre-launch checklist for any new launches (safety flags, canary, rollback plan). Prioritize on-call for mitigation.- High (3 ≤ burn-rate < 5): Automatic temporary freeze on non-critical launches; block large feature releases without joint SRE+PO approval. Create high-priority reliability tickets (tracked in backlog with SLA).- Critical (burn-rate ≥ 5 OR SLO violation sustained > 1 hour): Hard freeze on all launches; convene incident war room within 15 minutes; SRE-led remediation takes precedence over feature work. Executive escalation if > 3 hours.

Automated actions:- Continuous calculation of burn-rate and short-term burst detectors.- Integrations: PagerDuty/ops alerts, CI/CD gate that checks current burn-rate before merge/deploy; feature flags toggled automatically for canary rollbacks.- Auto-create reliability tickets with telemetry snapshot and reproducible failure context.

Manual actions / governance:- SRE & PO joint decision required to lift High/Critical freezes. Use a documented checklist for risk acceptance: root cause understood, rollback plan tested, mitigation in place.- Weekly SRE-Product reliability review for Elevated and High incidents; monthly reliability review board for trends and policy updates.- Post-incident: Blameless postmortem with remediation action items prioritized into the roadmap; product must allocate capacity to complete critical fixes within defined timebox (e.g., next sprint or 30 days).

Communication flows:- Automated notifications to SRE team, release engineers, POs, and relevant stakeholders on threshold crossings.- In High/Critical, SRE takes incident lead and initiates live war room; PO provides product context and prioritization trade-offs.- Daily status updates during Critical incidents; summary and action item list shared within 24 hours; follow-up review within 7 days.

Measuring adherence and outcomes:- Dashboards showing: real-time burn-rate, SLO attainment, number of launch freezes, time in freeze, number and age of reliability tickets, MTTR, deployment frequency.- KPIs: - % of incidents with postmortem completed within 7 days - % of High/Critical freezes lifted following documented SRE+PO approval - Reduction in burn-rate and SLO violation hours month-over-month - Time-to-complete prioritized reliability work (target: X% completed within SLA)- Quarterly OKRs tying product release cadence and reliability metrics—e.g., maintain SLO >= 99.95 while allowing N launches/month; if misses occur, require a reliability investment plan.

Scalability and trade-offs:- Use sampling and summarization for large-scale metrics to keep calculations efficient.- Trade-off: stricter freezes reduce risk but slow product velocity; thresholds must be tuned with historical data and business tolerance.- Start conservative, tune burn-rate thresholds after 2–3 quarters of measurement.

Audit and continuous improvement:- Log all automated actions and approvals for audits.- Review thresholds and processes quarterly; adapt to changes in architecture, traffic patterns, and business risk appetite.

EasyTechnical

101 practiced

Design an on-call rotation and escalation policy to reduce burnout while ensuring 24/7 coverage. Specify rotation length, handover procedures, follow-up/backfill roles, compensation or recognition mechanisms, and tooling or practices that reduce noisy pages.

Sample Answer

Situation: At my last SRE org we needed reliable 24/7 coverage without burning out a small team after several people reported fatigue from frequent nights and noisy alerts.

Task: Design an on-call rotation and escalation policy that balances coverage, recovery time, and long-term retention.

Action:- Rotation length: 7-day primary on-call shifts, with 2-week long-running secondary/peer backup rotations. Primary handles pages; secondary supports escalations and complex triage. Weekly rotation gives predictable cadence and limits consecutive nights.- Handover procedure: Mandatory written handover doc in the incident tracker at shift start/end summarizing open alerts, ongoing mitigations, action items, and health dashboard links. 15-minute synchronous overlap for verbal context on critical services at shift change.- Follow-up / backfill: Post-incident owner (PIO) assigned within 24 hours to drive RCA and remediation; backfill rota of volunteers for major incidents so primary can take uninterrupted recovery time the following day. If an on-call misses work due to incidents, they get a swap or an extra recovery day.- Compensation & recognition: Overtime pay for after-hours work OR extra PTO accrual per on-call week (e.g., 4 hours PTO/week), and quarterly “on-call excellence” bonus or shout-outs in all-hands. Track on-call load in performance reviews to reward sustained contributors.- Tooling & practices to reduce noisy pages: Implement alert hygiene (SLO-driven alerting, paging only for actionable alerts), alert deduplication and grouping, rate-limited notifications, runbooks linked from alerts, automated remediation playbooks for common failures, and use of escalation policies in pager (30s->5min->10min rules). Regular quarterly alert reviews with engineers to retire flappers.

Result: This structure reduces consecutive nights, ensures clear ownership and recovery time, and cuts noisy pages by prioritizing SLO-based alerts and automation—improving team morale and decreasing mean time to acknowledge/respond.

HardSystem Design

85 practiced

How would you design SLOs for a service that depends on three upstream services, each with its own SLOs? Show how you would allocate error budget across layers, prevent alert storms, and handle correlated failures when multiple dependencies degrade simultaneously.

Sample Answer

Requirements & constraints:- User-visible SLO for my service: e.g., 99.9% successful requests (p99.9 availability) per month with 300ms p90 latency.- Three upstreams A, B, C each have their own SLOs (e.g., A:99.95, B:99.9, C:99.8).- Want clear error-budget allocation, avoid alert storms, and handle correlated degradations.

High-level approach:1. Measure and decompose- Define end-to-end SLI (success rate + latency) for the business transaction.- Instrument request path to tag failures by root cause: local, A, B, C, network, timeouts, etc.- Compute per-dependency contribution to end-to-end error rate.

2. Error-budget allocation (example math)- Target end-to-end availability = 99.9% -> monthly error budget ≈ 0.1% (≈43.2 minutes).- Use additive reliability approximation: allowable failure contribution from dependencies = allocate proportionally to impact and controllability.- Example split: reserve 50% local (we control), 20% A, 20% B, 10% C. So A gets 0.02% budget (~8.6 min), etc.- Implement both strict (hard cap per dependency) and flexible pooled bucket for transient spikes.

3. Alerting & preventing alert storms- Multi-tier alerts: - Paginate alerts: hard SLO burn (immediate paging) when service-wide error budget burn rate > X%/hr. - Warning alerts: per-dependency burn rate exceeding threshold -> notify on-call but don’t page unless service-level budget is threatened. - Use aggregated alerts: suppress per-instance alerts into a single correlated alert when the same root cause seen across many nodes.- Use burn-rate windows (5m, 1h, 24h) and require sustained burn to escalate.- Circuit-breakers & client-side retries with backoff to reduce overload loops that cause storms.

4. Correlated failures & cascading degradation- Detect correlation: compute cross-correlation of failures across A,B,C; if multiple dependencies fail simultaneously, treat as systemic incident.- Prioritize mitigation by impact: if one dependency failing causes majority of errors, fail fast to degrade gracefully (feature flag or degraded mode).- Implement fallback strategies per dependency: cached responses, degraded feature set, local default values.- Throttle/pacing: dynamically reduce incoming traffic or shed non-critical requests to preserve core functionality.- Use chaos / game days to validate fallbacks and measure actual impact to refine allocations.

5. Operational policies & governance- Error-budget policy: product teams must stop launches when local budget exhausted; dependency owners coordinate remediation when their share is burned.- Runbooks: clear triage steps for “single dependency burn” vs “multi-dependency correlated burn”.- Regular reviews: monthly SLO reviews to reallocate budgets based on observed impact, latency-sensitive paths, and change in upstream SLOs.

Why this design:- Keeps end-user experience central (end-to-end SLO).- Allocates budget according to control and impact, balancing incentives.- Mitigates alert fatigue via tiered burn-rate alerts and suppression.- Handles correlated failures with detection, graceful degradation, and pre-planned fallbacks so reliability goals remain achievable even when multiple dependencies degrade.

HardTechnical

79 practiced

You maintain a distributed cache in front of authoritative databases. Product demands sub-50ms read latency but some features need strong consistency. Propose an architecture and migration plan that provides selective strong consistency while keeping typical reads fast. Discuss cache strategies, invalidation, and trade-offs.

Sample Answer

Requirements & constraints:- Read latency <50ms for typical reads.- Some reads require strong consistency (linearizable or read-after-write).- Operate across distributed caches and authoritative DBs with high availability.- Minimal impact to existing fast-path reads.

High-level architecture:- Front door: API gateway / edge service routes requests.- Dual-path cache layer: - Fast eventual cache (distributed in-memory, e.g., Redis Cluster or Memcached) for low-latency reads. - Strong-consistency path using a read-through coordinator that consults authoritative DB or a cache tier with synchronous invalidation (see below).- Metadata service: per-key policy (consistency level: STRONG / EVENTUAL), TTLs, and versioning (vector clock or monotonic version).- Change-log & invalidation bus: reliable message stream (Kafka) carrying writes and invalidations.- Optional: Read-repair workers and background reconciliation.

Selective strong consistency design:- Tag resources requiring strong consistency (policy service). Clients include a header or call directs reads accordingly.- For STRONG reads: use one of: 1) Read-through to primary DB with short-circuit (bypass cache) — always correct, higher latency. 2) Synchronous cache invalidation + compare-and-set: on updates, write to DB, publish invalidation, and perform synchronous CAS on cache; readers do a conditional fetch with version check and, on mismatch, fetch DB and update cache before returning. 3) Use consensus-backed cache (e.g., strongly consistent Redis via Raft or etcd) for hot keys requiring low-latency strong reads.

Caching & invalidation strategies:- Write path: DB write -> publish event -> invalidate cache key(s) -> confirm invalidation (for STRONG) or let eventual invalidation (for EVENTUAL). Use tombstones for deletes.- For EVENTUAL: asynchronous invalidation; optimistic reads from cache; background revalidation.- For STRONG: synchronous invalidation or use read-through to DB; ensure write ordering with per-key sequence numbers. Optionally implement read-after-write token: client write returns a write-token which strong reads present to guarantee they see >= that version.

Migration plan (incremental, safe):1. Add metadata/policy service and per-key tagging capability.2. Instrument metrics/observability for latency, cache hit/miss, invalidation lag, and error rates.3. Start with bypass-only strong path: for tagged keys, route reads to DB (no cache). Monitor latency; optimize DB read path.4. Introduce synchronous invalidation for a small % of traffic (canary) using reliable bus; verify correctness and observe latency.5. Implement CAS-based cache update for strong keys to reduce read latencies; rollout gradually by service & key.6. For hot low-latency strong keys, deploy consensus-backed cache nodes and migrate those keys.7. Full rollout with feature flags and automatic fallback: if consistency mechanism causes >SLO errors, fallback to DB reads.

Operational concerns & trade-offs:- Latency vs consistency: DB reads guarantee correctness but increase tail latency and load. Synchronous invalidation reduces staleness but adds write latency and complexity.- Throughput: CAS and strong-coherent caches limit write concurrency; careful sharding and backpressure needed.- Complexity & availability: consensus caches increase operational burden; they improve latency but must be highly available.- Observability: track invalidation lag, version mismatches, and error budgets. Automate rollbacks per-service.- Cost: more infrastructure and write amplification for strong keys.

Fallbacks and safety:- Circuit-breaker: if cache/invalidation system degrades, route strong reads to DB.- Rate-limit writes to avoid invalidation storms; batch invalidations for high churn keys.- Periodic reconciliation jobs to detect and repair stale entries.

This design gives typical reads sub-50ms via the fast eventual cache while providing selectable strong-consistency modes per resource, rolled out gradually with visibility and safe fallbacks.

MediumTechnical

92 practiced

Your company must decide between adopting a hosted observability platform or building an in-house solution. Create a decision framework that covers total cost of ownership, feature parity (metrics/logs/traces), vendor lock-in, security and compliance, scalability, team capability, and migration risk. What would you evaluate first and why?

Sample Answer

Decision framework (objective, weighted rubric + evaluation steps)

1) Clarify requirements (first step)- Why first: defines must-haves (retention, ingestion, regulatory controls, SLO targets). Without clear requirements you can’t compare meaningfully.

2) Evaluation criteria (suggested weights)- Total Cost of Ownership (20%): up‑front dev, ongoing ops, infra, support, training. Use 3‑5 year NPV.- Feature parity (metrics/logs/traces + alerting, dashboards, sampling) (20%): coverage, fidelity, query language, correlations (linked traces<>logs<>metrics).- Scalability & performance (15%): ingest/sec, retention, query SLAs, burst handling.- Security & compliance (15%): encryption, RBAC, audit logs, SOC2/GDPR/PII controls, data residency.- Vendor lock‑in & portability (10%): export formats, open standards (OTel), ability to migrate.- Team capability & hiring risk (10%): current SRE/infra skills, onboarding curve to build/maintain.- Migration & operational risk (10%): downtime risk, data migration cost, runbook changes.

3) How to evaluate- Create scoring matrix 0–5 per subcriterion, multiply by weight, sum.- TCO model: estimate engineering FTE months to build + recurring infra vs vendor fees + integration work.- Proof-of-value: pilot ingest of representative traffic, run 30-day queries, synthetic load tests.- Security review: threat model + vendor questionnaire + pen test SLA.- Portability check: ingest/export using OpenTelemetry, sample data export time/cost.

4) Migration risk mitigation- Staged rollout: dual-writing for 4–8 weeks, read-from-source fallback.- Data retention strategy: keep raw events in cheap long-term store; migrate indexes gradually.- Automation: scripts to convert and import historical data; runbooks and runbook drills.

What I’d evaluate first and why- Start with requirements → team capability → TCO. Requirements set acceptance criteria; team capability determines realistic build cost and speed; TCO ties them to business constraints. If requirements are light and team has capacity, building may be viable; if requirements demand compliance, scale, or fast time-to-value, hosted is usually preferable.

Unlock Full Question Bank

Get access to hundreds of Technical Ownership and Architectural Decisions interview questions and detailed answers.

Join thousands of developers preparing for their dream job.