Strategic Technical Decision Making Questions

Focuses on high-level, organization-wide technical decisions that have multi-year consequences, not the operating mechanics of any single technology. Candidates should reason through concrete decision types such as: choosing or consolidating a technology platform, deciding whether to build in-house versus buy or adopt a vendor/open-source solution, committing to (or reversing) a major architecture or infrastructure direction, sequencing technical debt paydown against feature velocity, and standardizing tooling or platforms across multiple teams. Strong answers name the alternatives considered, the criteria used to evaluate them (cost, reversibility, team capability, time horizon, risk), how uncertainty was managed with incomplete information, and how the rationale was communicated to non-technical stakeholders and leadership to build buy-in. This topic is about the reasoning, trade-off framing, and communication behind a strategic technical bet, not a definitional quiz on any specific technology (e.g. this is not the place for 'define eventual consistency' or 'explain canary deployments' style questions; those belong to the underlying technology topics).

MediumTechnical

34 practiced

Your team relies on a third-party payment gateway that has had several day-long outages in the past year. Describe a structured evaluation you would run to decide whether to continue with this vendor, add fallback providers, or build in-house functionality. Include metrics you would analyze, SLA and contractual considerations, technical integration complexity, and migration effort estimation.

Sample Answer

Framework: run a risk/benefit analysis in five phases — data collection, reliability assessment, cost/contract review, technical feasibility, and decision + rollout plan.

1) Data collection (2 weeks)- Collect incidents last 12 months: outage count, duration, MTTR, MTBF, impact windows.- Gather business impact: revenue lost, failed transactions, customer complaints, conversion drop, support costs.- Instrumentation: trace payments end-to-end, add synthetic transactions and latency histograms.

2) Reliability & metrics to analyze- Availability (uptime %), SLA breaches/month, mean time to recovery (MTTR), mean time between failures (MTBF).- Error rates: 4xx/5xx from gateway, payment success rate, retry effectiveness.- Latency P50/P95/P99 for authorization and capture.- Correlate outages to business KPIs (revenue/hour, conversion delta).

3) SLA & contractual review- Compare vendor SLA (uptime, credits, support RTO/RPO) vs required SLOs. Check credit caps, limits, indemnity, data ownership, termination clauses, and change-notice periods.- Verify on-call support escalation, dedicated TAM, and disaster recovery commitments.

4) Technical integration complexity- Assess current integration points (API protocol, idempotency, retries, webhooks), authentication, PCI scope, and data residency.- Evaluate how easy it is to add a second provider (adapter layer, switch logic, reconciliation, duplicate prevention).

5) Migration & fallback effort estimates- Option A: Continue + stronger contracts — low effort (2–4 wks legal + monitoring/Runbook updates).- Option B: Add fallback provider — medium effort (8–12 wks): build adapter, routing rules, reconciliation jobs, test harness, runbook, and failover drills.- Option C: Build in-house — high effort (6–12+ months): payments expertise, PCI compliance, fraud, integrations, ops staffing; high ongoing maintenance cost.

6) Decision criteria- If vendor availability < SLO and business impact high, favor multi-vendor unless cost/Pci prohibits.- Use ROI: compare outage cost + risk vs build/dual-run costs.

7) Rollout & validation- Pilot fallback in read-only or shadow mode, run canary failovers, define SLOs and error budgets, schedule regular failover drills, and add contractual exit triggers.

This structured evaluation provides measurable inputs to choose continue, add fallback, or build, balancing reliability, cost, and operational complexity.

MediumTechnical

46 practiced

You have three proposed reliability projects with approximate costs: A) distributed tracing across services ($200k), B) migrate to single-tenant DB clusters ($500k), C) add regional read replicas ($150k). Describe a cost-benefit framework using customer impact, incident probability reduction, and operational cost savings to evaluate and prioritize these investments, and list stakeholders you would involve.

Sample Answer

Framework (step-by-step):1. Clarify goals & horizon: minimize customer-facing incidents and ops toil over 12–24 months; target ROI/payback ≤ 2 years or reduce SLO breach risk by X%.2. Define metrics and units: - Customer Impact (CI): annual lost-user-hours or revenue risk avoided ($/year) from fewer outages; estimate via historical incident data (mean user-seconds lost × $/unit). - Incident Probability Reduction (IPR): expected absolute reduction in incidents/year (from baseline) × avg severity weight → translate to CI when combined with severity. - Operational Cost Savings (OCS): reduced on-call hours, manual fixes, and maintenance FTEs ($/year).3. Score each project: - Compute Annual Expected Benefit = CI (from fewer incidents) + OCS. - Compute Payback Period = Cost / Annual Expected Benefit. - Compute Risk/Confidence factor (0–1) from implementation risk and time-to-value; adjust benefits by this factor. - Compute Priority Score = (Annual Expected Benefit × Confidence) / Cost (higher = better).4. Example (illustrative numbers): - A: Cost $200k; estimated benefit $160k/yr (faster MTTR, fewer SLO breaches) × 0.9 confidence → Score = (144k)/200k = 0.72 - B: Cost $500k; benefit $120k/yr (isolation reduces blast radius) × 0.7 → (84k)/500k = 0.168 - C: Cost $150k; benefit $90k/yr (read latency + failover, fewer P1s) × 0.8 → (72k)/150k = 0.48 - Rank: A (0.72), C (0.48), B (0.168)5. Sensitivity & trade-offs: - Run sensitivity analysis on key assumptions (incident frequency, revenue impact). - Consider strategic value not captured by dollars (compliance, scalability). - For high-uncertainty but strategic projects (B), consider phased approaches or pilots.6. Decision rules: - Fund projects with Payback < 24 months and Priority Score above threshold. - If budget constrained, split funding: implement tracing (A) first, then regional replicas (C), pilot single-tenant DB for highest-risk customers.

Stakeholders to involve:- SRE/Platform (lead implementation, runbooks, metrics)- Database/DBA team (impact of migration, feasibility)- Backend engineering teams (schema/compatibility)- Product/PM (customer impact, prioritization)- Finance (budgeting, ROI validation)- Customer Success/Support (customer pain, SLA implications)- Security/Compliance (multi-tenancy vs isolation)- CTO/Engineering leadership (final prioritization, strategic alignment)

This approach yields transparent, quantitative prioritization while allowing qualitative/strategic adjustments.

EasyTechnical

65 practiced

List criteria an SRE should use to decide between synchronous (HTTP/gRPC) and asynchronous (message queues, events) API patterns for a new service. Include latency requirements, failure modes, backpressure handling, operational complexity, and end-to-end observability implications.

MediumSystem Design

37 practiced

Design an API versioning strategy for REST/gRPC services used by both internal and external clients. Cover compatibility guarantees, deprecation timelines, discovery/version negotiation mechanisms, backward-compatibility testing in CI, and mechanisms to coordinate cross-team client migrations with minimal disruption.

Sample Answer

Requirements & constraints:- Support REST + gRPC, internal and external clients- Minimize downtime and risk during migrations- Strong backward-compat guarantees; measurable deprecations- Automated verification in CI, observability for coordination

High-level strategy:1. Versioning model- Use semantic versioning for service contracts (vMAJOR.MINOR). MAJOR = breaking change, MINOR = additive non-breaking features/fields.- REST: prefer path-versioning (/v1/) for clear routing + header-based negotiation (Accept-Version) for clients that want content negotiation.- gRPC: version via Protobuf package/service names + generated package versions (e.g., mysvc.v1). Keep old RPCs; add new methods or request/response fields as optional.

2. Compatibility guarantees- Non-breaking changes (adding optional fields, new endpoints/methods, new enum values with UNRECOGNIZED handling) are supported across MINOR bumps.- Breaking changes require MAJOR bump and side-by-side support for previous MAJOR for a policy period.

3. Discovery & negotiation- Expose a /versions or /service-info endpoint and server reflection (gRPC) for available versions.- Support Accept-Version and X-Client-Version headers; Envoy/Ingress can route by header or path.- Default to the latest non-breaking compatible version; refuse requests that require unknown MAJOR with 400 + clear error body pointing to migration docs.

4. Deprecation timelines & communication- Standard policy: Internal clients — 3 months notice; External clients — 6–12 months notice depending on impact (documented in SLA).- Announce via: API changelog, mailing list, developer portal, deprecation headers (Deprecation, Sunset), and dashboard showing client usage by version.- Automatic Sunset: after expiration, route deprecated-version requests to a “migration help” endpoint for a period, then 410 Gone.

5. Backward-compatibility testing in CI- Contract tests: use schema-aware tests (Protobuf descriptors, JSON schema) to enforce no forbidden breaking changes on non-MAJOR commits.- Consumer-driven contract testing (Pact or equivalent) where important clients publish contracts; CI validates provider changes against consumer contracts.- Repository checks: proto linter rules (no removal of fields, use reserved), CI gate that runs schema-diff tools and rejects changes that break compatibility unless MAJOR bump PR is approved.- Integration tests: spin up canary deployment and run automated client suites (internal) to validate runtime behavior.

6. Client migration & cross-team coordination- Telemetry: per-version request counts, errors, latency; create dashboards and alerts when deprecated versions remain >X% traffic past midpoint.- Phased rollout: deploy new MAJOR in parallel; use feature flags and traffic-splitting (Envoy) to route a percentage to new version; test rollback path.- Migration playbook: maintain checklist with required SDK updates, config flags, test steps, rollback instructions, stakeholders, and timeline.- Governance: create a cross-team API working group and a public migration calendar; require teams to publish migration plans and signoffs for MAJOR changes.- SRE responsibilities: enforce CI gates, operate canary infra, maintain dashboards/alerts, run emergency rollback, and own SLA adjustments.

Trade-offs:- Path-versioning simpler for routing; header negotiation more flexible but requires client opt-in.- Side-by-side versions increase operational cost but reduce disruption.

Outcome:- Clear, automated, observable version lifecycle that enforces compatibility via CI, gives clients time and guidance to migrate, and enables SREs to limit blast radius with canaries, dashboards, and rollback mechanisms.

MediumTechnical

37 practiced

For a stateful, low-latency caching service used heavily in reads, compare vertical scaling (larger instances) versus horizontal scaling (sharding across nodes). Discuss operational complexity, failure modes, failover behavior, maximum limits, cost trade-offs, and monitoring signals you would use to trigger scaling actions.

Sample Answer

Clarify assumptions: stateful cache (in-memory), heavy read traffic, low-latency SLAs, must persist state per-shard (or warm-up cost), and predictable workload patterns.

High-level tradeoff- Vertical scaling (bigger instances): simpler — single instance holds more data, no sharding logic. Good for fast time-to-scale and lower operational complexity for small fleets.- Horizontal scaling (sharding across nodes): distributes load and capacity, better headroom and failure isolation at scale but requires routing/sharding logic, rebalancing, and more complex operational workflows.

Operational complexity- Vertical: low—standard autoscaling, fewer components. Downsides: OS/kernel tuning, memory fragmentation, and instance-type dependencies.- Horizontal: higher—need consistent hashing/metadata service, client routing, membership management, automated rebalancing, and state migration tools.

Failure modes & failover behavior- Vertical: single-instance failure => larger blast radius and potential full-cache unavailability until failover/restore. Failover often slower due to state warm-up. Simpler recovery if snapshot/replication exists.- Horizontal: node failures affect only subset of keys; clients can reroute. If replication per-shard exists, can failover quickly. Complexity: split-brain during rebalancing, transient hot-shard failures.

Maximum limits- Vertical: bounded by largest available instance (memory, CPU, NIC). Diminishing returns and vendor limits.- Horizontal: scales linearly with nodes; practical limits come from coordination overhead, metadata bottlenecks, and increased tail latency if many shards.

Cost trade-offs- Vertical: may be more cost-effective at small scale (less cross-node overhead), but large instances often have worse $/GB and limited burst NIC bandwidth.- Horizontal: better $/GB at scale, can use heterogeneous instances, but adds operational and networking costs (replication traffic, control plane).

Monitoring signals to trigger scaling- Resource KPIs: 95/99/99.9th latency, p50/p95/p99 CPU, memory utilization, page faults, GC pause times.- Throughput/pressure: ops/sec, read QPS per node, NIC saturation, socket counts.- Cache-specific: eviction rate, miss rate, cold-start rate, time-to-warm, replication lag, rebalancing progress.- Health/availability: error rate, failover frequency, node join/leave churn.- Business signals: sudden increase in client traffic, SLO burn rate or error budget consumption.

Scaling policies (examples)- Vertical scale-up when node memory >75% and eviction rate rising, and when cluster size small.- Horizontal scale-out when p95 latency exceeds threshold while per-node CPU/NIC near capacity or when miss rate increases despite more memory pressure.- Prefer proactive horizontal scaling for predictable growth; use vertical as short-term mitigation or for simple deployments.

Operational best practices- Automate rebalancing and rolling upgrades; use replication to reduce warm-up.- Canary scaling changes and measure tail latency.- Maintain observable metrics and runbooks for failover and re-shard procedures.- Run cost vs. latency experiments to pick mix of vertical/horizontal for your workload.

Unlock Full Question Bank

Get access to hundreds of Strategic Technical Decision Making interview questions and detailed answers.

Join thousands of developers preparing for their dream job.