Engineering Decision Making and Leadership Questions

Covers how technical leaders and engineers make architecture and implementation decisions while enabling team autonomy and organizational alignment. Areas include engineering judgment in trade offs between performance, reliability, maintainability, and speed; processes for making and communicating technical decisions; governance models such as centralized versus distributed decision making; handling cross team disagreements; managing technical debt; and establishing accountability for technical outcomes. Candidates should show methods for gathering engineering data, involving relevant stakeholders, and ensuring decisions are followed and revised as needed.

MediumTechnical

97 practiced

You are asked to instrument a new service so that product and engineering can make high-confidence decisions about performance and correctness. Describe which metrics (business and technical), logs, traces, and dashboards you would create, sampling and retention considerations, and how you'd expose those to stakeholders.

Sample Answer

Goal: enable high-confidence decisions about performance and correctness by providing business + technical observability, fast investigations, and actionable dashboards/alerts.

Metrics (business + technical)- Business: API usage (requests/sec), conversion/feature-specific events, revenue/MAU impacted by service, error budget consumption.- Technical: P95/P99 latency, success/error rate, throughput, request size, backend DB/queue latency, resource usage (CPU/mem), circuit-breaker/fallback counts, cache hit ratio.

Logs- Structured JSON logs with request_id, user_id (hashed), trace_id, component, severity, short message, contextual fields.- Log levels: INFO for normal ops, WARN for retries, ERROR with stack + context.- Include sampling for verbose debug logs (~1%) with on-demand burst capability.

Traces- Distributed tracing (W3C traceparent): capture end-to-end spans, annotate spans for DB, external calls, queueing, retries.- Sample 100% for errors; 5–10% for success paths; higher during incidents.

Dashboards & Alerts- Executive dashboard: business KPIs, error budget, SLA.- SLO dashboard: latency and availability against targets with burn rate.- Ops dashboard: real-time p95/p99, throughput, service map, recent traces and logs.- Alerts: SLO burn rate, spike in error rate, latency breach, resource saturation (pager), noisy but actionable (pager for on-call, Slack for less critical).

Sampling & Retention- Metrics: high-cardinality tags limited; retain high-res (<1m) for 30 days, aggregated for 13 months.- Logs: retain error logs 90–365 days; sampled access logs 30 days; allow archive to cheaper storage.- Traces: retain full traces for 7–30 days; store span indexes longer for search.

Exposing to stakeholders- Product: weekly KPI report + executive dashboard (read-only), alerts for feature-impacting regressions.- Engineers: realtime dashboards, alert channels, runbooks, access to raw logs/traces via explorer.- Self-serve: templates for ad-hoc queries and pre-built Grafana dashboards; onboarding docs and training.

Why- Combines business outcomes with technical signals so product can prioritize and engineers can triage quickly. Sampling and retention balance cost vs. investigatability. Alerts tied to SLOs reduce noise and direct action.

EasyTechnical

78 practiced

What should a high-quality RFC contain? Write an outline for an RFC proposing a new distributed caching layer for a web app expected to handle 100,000 QPS. Include sections for motivation, alternatives, cost estimate, rollout, monitoring, and rollback.

Sample Answer

A high-quality RFC clearly states scope, motivation, design, trade-offs, and a concrete implementation and rollout plan so reviewers can make an informed decision. It should include: summary, goals & non-goals, background, detailed design, alternatives & trade-offs, safety/rollback, testing, rollout/ops plan, monitoring/alerting, cost estimates, security & compliance, and clear acceptance criteria.

Outline: "Distributed Caching Layer for Web App — RFC"

1. Title & Authors2. Summary (1–2 paragraphs): propose an LRU/segmented distributed cache (in-memory) in front of app tier to support 100k QPS with <5ms P95 read latency.3. Motivation: current DB CPU/latency, cost of DB scaling, user impact, target SLAs (99.9% availability, P95 latency).4. Goals & Non-goals: reduce DB read load by 80%, support TTL-based invalidation; not responsible for strong consistency for writes.5. Architecture / Design: - Cache tier (Redis Cluster / Memcached / bespoke), sharding, client libraries, key schema, serialization. - Read/write flow diagrams, cache warming, invalidation strategies (write-through, write-back, cache-aside). - Capacity sizing: data items, hot-key handling, eviction policies.6. Alternatives & Trade-offs: - Managed Redis vs self-hosted vs CDN/edge caching — pros, cons, operational complexity.7. Cost Estimate: - Infra (nodes, instance types), network, licensing, operational headcount — rough monthly/yearly numbers and sensitivity.8. Rollout Plan: - Canary on noncritical traffic → perf tests → 10% → 50% → 100% with KPIs gating each step.9. Monitoring & Alerting: - Metrics: cache hit-rate, miss-rate, latency, eviction rate, CPU/memory, network I/O, DB QPS. - Alerts: drop in hit-rate, high evictions, increased DB load, node failures. - Dashboards and SLOs.10. Testing: - Load tests (100k QPS simulation), chaos tests (node failures), data correctness tests.11. Rollback Strategy: - Traffic switch-back to previous path, TTL expiration, automated config toggle, procedure with runbook and expected impact.12. Security & Compliance: auth, encryption, ACLs, secrets handling.13. Migration & Cleanup: key naming, backfills, cache warming scripts, decommission plan.14. Acceptance Criteria & Next Steps: measurable metrics to approve GA.

Appendices: capacity calculations, simulation results, runbooks, cost spreadsheets.

MediumBehavioral

64 practiced

Tell me about a time you pushed back on a business request due to technical risk. Describe the data and arguments you used, how you negotiated scope or timeline, and the eventual outcome and lessons learned.

Sample Answer

Situation: Six months into a major payments feature, PM asked us to accelerate a go-live by two sprints to hit a marketing deadline. The change touched our reconciliation pipeline and had low test coverage.

Task: I needed to assess technical risk and push back if the shortened timeline would materially raise production failure risk.

Action:- I ran quick smoke and load tests against the existing pipeline and reproduced a 12% failure rate on edge-cases (time-zone cutovers, duplicate transactions). I measured test coverage (38% for the critical module) and estimated remediation work: ~3 sprints to add end-to-end tests, fix two flaky deserializers, and harden retries.- I presented the data to the PM and stakeholders: measured failure rates, coverage %, and estimated mean-time-to-detect/resolve (MTTD ~6 hours) and potential customer impact (~0.2% revenue exposure per day under failure scenarios).- Proposed a negotiated plan: (1) ship a limited rollout behind a feature flag for 5% of traffic after one sprint of safety work (fixing the deserializers and improving logging), (2) complete the remaining hardening and tests over the next two sprints, and (3) full rollout only after automated E2E tests pass in CI and a 48-hour canary window.- I volunteered to pair with a backend engineer to prioritize the highest-risk fixes and created a clear acceptance checklist.

Result: Stakeholders accepted the phased approach. The canary caught an unexpected reconciliation edge case, which we fixed before full rollout. We met the marketing milestone with minimal exposure; post-launch incidents were reduced to <0.01% daily impact. The negotiated plan preserved business timing while mitigating production risk.

Lessons learned: Quantify risk with measured data, offer a concrete mitigated path (feature flags + canary + tests), and trade scope/timing rather than simply saying “no.” Clear acceptance criteria and shared ownership accelerate decision-making and reduce surprise incidents.

HardTechnical

68 practiced

Design an organization-wide observability pipeline to support engineering decision-making: ingest logs, metrics, and traces from 1,000 services with 1 TB/day ingest. Requirements: role-based access, retention tiers, rollups, and query latency under 2s for recent data. Describe ingestion, storage tiers, schema strategy, cost-control mechanisms, and access patterns.

Sample Answer

Situation & goals: ingest logs/metrics/traces from 1,000 services at 1 TB/day, provide <2s query latency for recent data, RBAC, retention tiers, rollups and cost control. Proposed end-to-end design below.

Ingestion- Agents: OpenTelemetry Collector + Vector/Fluentd on hosts/k8s to emit structured JSON logs, metrics (Prometheus exposition/remote_write) and spans (OTel).- Ingest plane: Kafka (or kinesis) as durable buffer and burst absorber. Schema-normalizer service consumes from Kafka, validates/enriches (service, env, hostname, trace_id), then fans to downstream stores.- Backpressure & QoS: per-tenant/service rate limiting at collector + token-bucket in Kafka topic partitions.

Storage tiers & components- Hot (0–7 days): low-latency stores for real-time dashboards and debugging. - Metrics: Mimir/Cortex or Thanos Ruler with TSDB shards; use ingesters with local memory + index; store WAL and compacted chunks. - Logs: ClickHouse or Elasticsearch hot cluster with inverted index for last 7d; schema-on-write JSON columns + indexed fields. - Traces: Jaeger/Tempo with local index (index by trace_id, service, span_id) and hot span-cache.- Warm (7–30 days): downsampled metrics and rollups, aggregated logs (per-minute/hour counts) in ClickHouse; traces only sampled (e.g., 1-5%) and stored.- Cold (30–365 days): object storage (S3) with Parquet/ORC for logs and traces; metrics as compressed Prometheus blocks or Parquet aggregates.- Archive (>365 days): Glacier/Deep Archive.

Schema strategy- Enforce structured, typed events: logs in JSON with reserved fields (timestamp, service, env, level, trace_id, span_id, msg) and a free attributes map. Validate at ingress.- Metrics: labels schema and cardinality guardrails. Maintain allowed label keys per service; reject high-cardinality labels or convert to attributes stored only with traces.- Traces: store spans normalized: trace_id, span_id, parent_id, service, op, start_ns, duration_ns, attributes(map). Use append-only columnar store for long-term.

Rollups & precomputation- Real-time rollups: streaming jobs (Flink/Beam) consume Kafka and generate per-service/metric/minute/hour aggregates, error rates, p95/p99 latency histograms; store in hot ClickHouse/TSDB for sub-2s dashboard queries.- Logs rollups: precompute counts by (service, level, reason) per minute and histograms for log volume.- Trace rollups: service-level latency percentiles per minute; full traces retained for sampled set.

Query latency & access patterns- Optimize hot path: queries hit hot clusters with in-memory indexes, precomputed rollups for dashboards => sub-2s.- Ad-hoc/forensics: query warm/cold via async federated query engine (Trino/Presto) that reads Parquet on S3; provide background job to hydrate results into hot cache if repeated.- Access types: - Dashboards/alerts: read from hot rollups (RBAC filtered). - Dev debugging: raw logs/traces in hot for 7d; older via warm sample. - Compliance/audit: cold archive with strict access control and audit logging.

RBAC & multi-tenancy- Centralized authz (OIDC + LDAP) + attribute-based RBAC.- Row-level/field-level masking for sensitive fields.- Tenant scoping: namespace tags and per-tenant Kafka topics, storage quotas, and enforcement at ingestion.

Cost-control mechanisms- Ingestion controls: sampling (adaptive tail-sampling for traces), dynamic rate-limits per service, reject/transform high-cardinality labels, early aggregation at collectors.- Storage controls: aggressive compression (Parquet/ORC, snappy/zstd), TTL-based lifecycle (hot→warm→cold→archive), downsampling windows.- Query cost: precomputed rollups to avoid scanning raw data; query quotas, cost-based query planning and result caching.- Autoscaling & spot instances for batch workers; use object storage for most bytes.

Operational concerns & trade-offs- Reliability: Kafka for durability; replicate hot clusters across AZs. Use consistent schema registry.- Trade-offs: keeping full raw data hot is expensive — we keep 7d raw, sample later. This meets <2s recent queries while controlling long-term cost.- Observability for the observability stack: instrument pipeline, set alerts for ingestion/backpressure, SLOs on query latency.

This design balances rapid developer workflows (fast recent queries, full debugging on recent data) with long-term cost control via sampling, rollups and tiered storage.

HardTechnical

75 practiced

Explain the trade-offs between strong consistency (linearizability) and eventual consistency in distributed systems. Provide three real-world application scenarios where you would prefer one model over the other and explain how the choice affects developer workflows, testing strategy, and deployment patterns.

Sample Answer

Strong consistency (linearizability) guarantees each read reflects the most recent completed write globally — operations appear to occur in a single, real-time order. Eventual consistency allows replicas to diverge temporarily but guarantees they converge given no new writes.

Trade-offs:- Latency vs correctness: Linearizability typically requires coordinator/leader sync or quorum protocols, increasing latency and reducing availability under partitions (per CAP). Eventual consistency yields lower latency and higher availability but shifts complexity to conflict resolution.- Simplicity vs complexity: Linearizability simplifies reasoning and debugging (single source of truth). Eventual consistency forces application-level conflict-handling, compensating actions, or CRDTs.- Scalability: Eventual models scale writes and geo-distribution more easily; linearizable systems often bottleneck on leaders or strong quorum costs.

Three scenarios:1) Financial ledger (bank transfers) — prefer linearizability. - Why: correctness and auditability are paramount. - Developer workflow: synchronous APIs, strict transaction boundaries, fewer but more careful code paths. - Testing: strong focus on correctness tests, formal verification, fault-injection for partitions. - Deployment: careful rolling upgrades, leader election observability, conservative scaling.

2) Social media feed / likes count — prefer eventual consistency. - Why: temporary divergence acceptable; UX tolerates slight staleness. - Developer workflow: design idempotent ops, use append-only logs or CRDT counters, implement reconciliation. - Testing: chaos testing for convergence, property-based tests for CRDTs, user-experience A/B tests. - Deployment: multi-master replication, regional caches, relaxed upgrade constraints.

3) Inventory for e-commerce during checkout — hybrid: linearizable for final commit, eventual for browsing stock levels. - Why: show availability fast (eventual), but reserving/purchasing requires strong correctness. - Developer workflow: separate read-paths and write-paths, implement reservation/compensating transactions. - Testing: integration tests that simulate race conditions, end-to-end checkout flows, rollback scenarios. - Deployment: partitioned services (read replicas + single-master reservation service), feature flags for rollout.

Practical guidance:- Prefer linearizability when invariants cannot be repaired (money, identity). Use eventual consistency where latency/availability and scale matter and divergence is tolerable. For many systems a hybrid design (fast eventually-consistent reads, strong writes/commits) balances UX and correctness.

Unlock Full Question Bank

Get access to hundreds of Engineering Decision Making and Leadership interview questions and detailed answers.

Join thousands of developers preparing for their dream job.