Engineering and Business Outcomes Questions

How engineering work and technical decisions translate into measurable business outcomes and how to demonstrate that linkage. Topics include mapping architecture choices, reliability, performance improvements and developer productivity initiatives to business metrics such as revenue, customer engagement, time to market, cost reduction and customer satisfaction. Candidates should be able to identify engineering metrics to track including latency, availability, error and incident rates, cycle time and deployment frequency, explain instrumentation strategies to capture signals, design measurement plans and experiments to establish causal impact, and attribute observed changes to specific engineering efforts. This topic also covers communicating technical tradeoffs and impact to nontechnical stakeholders, choosing appropriate granularity for measurement, and describing concrete initiatives with their measurement approach and quantified business impact.

HardTechnical

30 practiced

Discuss algorithms for computing p99 latency in a high-throughput streaming pipeline where you cannot store complete request lists. Compare t-digest, DDSketch, and reservoir sampling in terms of mergeability, tail accuracy, memory usage, and numerical stability. Recommend an approach and describe how you would validate it in production.

Sample Answer

Start by clarifying constraints: continuous high-throughput streams, cannot retain full request lists, need accurate p99 (very high quantile) with low memory, mergeable across shards, robust numerically.

Comparative summary- t-digest - Mergeability: Excellent — designed for distributed merges while preserving centroid ordering. - Tail accuracy: Good for moderate tails; accuracy near extremes (p99.9+) can degrade unless tuned with more centroids focused on tails. - Memory: Compact; memory grows with number of centroids (configurable). - Numerical stability: Generally stable but sensitive to centroid compression settings; must avoid pathological merges when data is highly skewed.- DDSketch - Mergeability: Good — sketches are easily merged by per-bucket aggregation. - Tail accuracy: Excellent for relative-error guarantees (relative error bound for each bin), especially suitable when multiplicative error is acceptable for large latencies. - Memory: Controlled by accuracy parameter (alpha); can be larger than t-digest for very tight relative error but predictable. - Numerical stability: Strong — uses logarithmic bucketing to maintain relative error and avoid overflow; careful overflow handling still required.- Reservoir sampling - Mergeability: Poor-to-moderate — merging unbiased samples across shards requires weighted reservoirs or coordinated sampling; naive merges bias the distribution. - Tail accuracy: Weak for rare events — fixed-size reservoir likely misses extreme tail values unless sampling rate is very high. - Memory: Minimal and fixed-size. - Numerical stability: Trivial (stores raw samples) but statistically noisy for p99.

Recommendation- Use DDSketch as primary approach for p99 in high-throughput streaming where p99 is large-value sensitive and relative error guarantees matter. DDSketch gives predictable tail accuracy and easy merging across shards. Augment with a t-digest for complementary absolute-quantile estimates if you need better median or low-percentile accuracy with fewer bins, or if you require CDF reconstruction for visualization.- Avoid plain reservoir sampling for p99 unless you supplement with targeted tail sampling (e.g., sample-all for requests above an adaptive threshold).

Validation plan for production1. Offline ground-truth: Replay historical request logs through your pipeline, compute exact p99 from full data and compare error vs DDSketch/t-digest across workload patterns (uniform, bimodal, heavy-tailed).2. Synthetic injections: Inject synthetic latency spikes at controlled rates/values to test tail capture and merge correctness.3. Shadowing/A-B: Run sketching in parallel with a small exact-collection cluster that stores full lists for short windows to continuously measure error drift.4. Monitoring & Alerts: Track sketch health metrics (bucket growth, compression ratio), divergence from shadow ground-truth, and rate of out-of-range buckets; alert when error exceeds SLO (e.g., >2% relative error at p99).5. Operational practices: Tune DDSketch alpha and t-digest centroid count based on observed error patterns; keep deterministic merge order where possible; add versioning to sketch formats to avoid silent incompatibilities.

This combination provides a mergeable, memory-predictable, tail-accurate solution with an actionable validation pipeline to ensure p99 SLOs are met in production.

MediumSystem Design

40 practiced

Design SLOs and an error budget policy for a multi-tenant platform where tenants have different paid tiers. Describe how you would implement per-tenant SLIs, compute aggregate weighted SLOs for the company, and policies for billing credits or mitigations when a tenant's SLO is violated.

Sample Answer

Requirements & constraints:- Multi-tenant platform with paid tiers (Free, Standard, Premium).- Per-tenant visibility, isolation of metrics, and company-level reliability target.- SLIs must be measurable, low-overhead, and support billing/mitigation automation.- Support realtime enforcement of error budgets and monthly reporting.

High-level design:- Instrumentation layer collects events (requests, success/failure, latency) tagged with tenant_id and tier.- Metrics pipeline (Prometheus/Pushgateway → Kafka → long-term store like Cortex/TSDB) aggregates per-tenant SLIs.- Policy engine reads SLI/SLO state and triggers actions (credits, rate-limits, escalations).

Per-tenant SLIs (examples):- Availability: % of successful API requests per tenant (status 2xx/3xx) measured per minute/hour.- Latency: 95th and 99th percentile tail latency per tenant for key APIs.- Error rate: % requests returning >=5xx or known transient errors.

Per-tenant SLOs by tier:- Free: Availability 99.0%, p95 latency 500ms- Standard: Availability 99.9%, p95 latency 300ms- Premium: Availability 99.95%, p95 latency 100ms

Aggregate weighted SLOs:- For company-wide SLO, compute weighted average where weight = monthly revenue contribution or seat count per tenant.- Formula: Company_SLO = 1 - (sum_over_tenants(weight_t * error_budget_burn_t) / sum_weights) or directly compute weighted availability = sum(weight_t * availability_t) / sum_weights.- Use revenue-weighting for business impact alignment; fallback to equal-weighting for privacy-preserving aggregated views.

Error budget & policy:- Monthly window per tenant. Error budget = 1 - SLO.- Continuous burn-rate monitoring: - Normal (burn < 1x): no action. - Elevated (1x <= burn < 5x): notify tenant account team, increase sampling, create incident P1 watch. - Critical (burn >= 5x or >50% budget consumed in 24h): enforce mitigations and entitlement actions.

Mitigation & billing policies:- Automated mitigations: - Traffic shaping: for noisy tenants whose traffic causes system instability, apply soft rate-limits (tier-aware) after warnings. - Feature gates: temporarily disable non-critical features for that tenant.- Billing credits: - If tenant-level SLO violated for the monthly window and root cause is platform fault (not tenant misuse), grant credits proportional to outage severity and SLA: - Credit = min(monthly_invoice, outage_fraction * credit_factor_by_tier). Example: Premium gets 10x outage_fraction, Standard 5x, Free none. - Credits require validation: incident root cause analysis (RCA) that attributes fault to provider within 7 days.- Escalation & contractual remedies: - For repeated SLA breaches, provide contractual remediation: higher credit caps, dedicated engineering support, or termination clauses per contract.

Implementation details:- Rollup queries in TSDB compute per-tenant p95/p99 and availability with labels.- Policy engine (e.g., custom service or feature in SRE runbook automation) subscribes to metrics via alertmanager/webhooks and runs decision logic.- Audit logs for credit decisions, automated mitigation actions, and tenant notifications for compliance and dispute resolution.

Trade-offs & considerations:- Per-tenant metrics increase cardinality — mitigate with sampling, aggregation windows, cardinality limits, and retention tiers.- Revenue-weighting favors big customers — ensure transparency and contractual alignment.- Automated mitigations must be conservative to avoid false positives; always include human-in-the-loop for billing actions.

Monitoring & reporting:- Tenant portal exposing current SLI/SLO, remaining error budget, past credits, and incident history.- Monthly SLO report and RCA distribution to affected tenants and internal stakeholders.

This design balances technical feasibility, business impact alignment, and fair operational policies while giving sales and legal teams concrete SLO/billing artifacts to include in contracts.

MediumTechnical

37 practiced

You need to detect whether a 0.5 percentage point lift in conversion from 5.0% to 5.5% is statistically significant. Explain how to compute a confidence interval and required sample size for this A/B test, including assumptions and when to use normal approximation versus exact binomial or bootstrap methods.

Sample Answer

Approach summary — compute a CI for the difference in proportions and do a power/sample-size calculation under usual parametric assumptions. For conversion p1=0.050 → p2=0.055 (absolute lift Δ=0.005).

1) Confidence interval (normal approximation)- Estimate difference d = p2hat − p1hat.- Standard error: SE = sqrt( p1hat(1−p1hat)/n1 + p2hat(1−p2hat)/n2 ).- Two-sided 95% CI: d ± z_{0.975} * SE (z_{0.975}=1.96). This is valid when n·p and n·(1−p) for each arm are reasonably large (rule of thumb: ≥5–10).

2) Required sample size (two-arm, equal n, two-sided α=0.05, power 80%)- Use normal-power formula accounting for both sampling variability: n ≈ [ (z_{1−α/2}√(2 p̄ (1−p̄)) + z_{power}√(p1(1−p1)+p2(1−p2)) )^2 ] / Δ^2 where p̄=(p1+p2)/2.- Plugging p1=0.05, p2=0.055, Δ=0.005, z_{1−α/2}=1.96, z_{power}=0.84 gives n ≈ 31,200 per arm (≈62.4k total). This matches intuition: tiny absolute lift requires large n.

3) Assumptions & when to use alternatives- Normal approximation valid when sample sizes are large enough (np, n(1−p) ≥ ~5–10). If not, or if p is near 0 or 1, the normal CI is unreliable.- Exact binomial (Clopper–Pearson) or Newcombe’s method for difference of proportions: use when sample sizes small or you need guaranteed coverage.- Bootstrap: useful when randomization is complex (unequal weights), metrics are derived/aggregated, or the distributional form is unknown. Bootstrap gives empirical CI without parametric assumptions but needs many resamples and careful stratification by user/session to preserve dependency structure.- Other practical notes: choose one-sided vs two-sided based on hypothesis; adjust for multiple testing; account for noncompliance, missing data, or correlated observations (use cluster-robust SEs if users have repeated events).

4) Recommendation for a solutions architect- For quick planning use the normal-power formula above to estimate traffic requirements and timelines.- During implementation, run both asymptotic and bootstrap/Clopper–Pearson checks; if counts are borderline, report exact/bootstrap CIs and pre-specify analysis plan to avoid bias.

EasyTechnical

37 practiced

Explain the difference between SLI, SLO, and SLA. Provide concrete examples for each in the context of a customer-facing payments API, including how you would measure them, what business outcomes they connect to, and what operational actions you'd take if each is violated.

Sample Answer

SLI (Service Level Indicator)- Definition: a measurable metric that reflects service behavior (technical).- Example (payments API): "Authorization success rate" = % of payment requests returning 2xx within 500ms; also p99 latency for authorization.- How to measure: instrument gateway logs + metrics pipeline (Prometheus/Grafana) counting total vs successful authorizations and latency histograms per minute.- Business outcome: maps to transactions completed and revenue; low SLI → lost revenue and churn.- Operational action on violation: trigger on-call, run runbook: check downstream PSPs, DB connection pools, rate-limiting, revert recent deployments, enable failover traffic to backup region.

SLO (Service Level Objective)- Definition: a target/goal for one or more SLIs over a time window (internal reliability commitment).- Example: "Authorization success rate SLO: >= 99.95% per 30-day window; p99 latency <200ms."- How to measure: rolling 30-day aggregation, burn-rate alerts.- Business outcome: sets engineering prioritization and release gates; preserves customer experience target that supports merchant conversion rates.- Operational action on breach/threat: escalate to engineering and product, convene reliability war room, prioritize bugfixes and capacity increases, postpone non-critical releases, schedule postmortem and remediation roadmap.

SLA (Service Level Agreement)- Definition: contractual guarantee with customers; includes remedies (credits) for breaches.- Example: Contract: "99.9% monthly uptime for payments API; financial credits of X% of monthly fee if violated."- How to measure: external uptime check (synthetic probes from multiple regions) + server-side success metrics validated monthly.- Business outcome: legal and commercial exposure; impacts renewals, penalties, and sales negotiations.- Operational action on breach: notify affected customers per contract, apply credits, execute incident RCA, commit timeline for fixes, offer mitigation (e.g., dedicated support, custom SLAs) to retain large customers.

Notes on alignment: design monitoring to derive SLIs reliably, map SLOs to business KPIs (conversion, revenue), and ensure SLA thresholds are stricterly negotiated (and reflected in SLOs) with runbooks, fallback paths, and capacity plans to reduce commercial risk.

MediumSystem Design

36 practiced

You're evaluating an observability backend migration. Propose a retention and rollup policy that balances cost with the need to investigate incidents and support audits. Include retention periods for raw logs, traces, and metrics, rollup windows and granularity, and a plan for ad-hoc rehydration of raw data if required.

Sample Answer

Requirements & constraints:- Support incident investigation (root cause within 30 days, forensic up to 1 year)- Audit/compliance: retention for certain events up to 7 years (configurable)- Control storage cost while preserving useful signal- Allow ad-hoc access to high-fidelity raw data when necessary

Proposed retention policy (default, adjustable per data-class):- Raw logs: keep full-text raw logs 30 days. Indexed metadata (keys, timestamps, log-level, service, trace_id) retained 365 days. Archive critical log streams (security, payments) to cheap object storage for 7 years.- Traces: full-span traces 90 days. Sampled trace index (trace id, span summary, error flags, latency buckets) 2 years. Retain 100% of traces for high-priority services; for low-priority, sample 10–20% after 30 days.- Metrics: high-resolution (1s-15s) for 30 days. Rolled-up granularities: 1m for 365 days, 1h for 3 years, 1d for 7 years.

Rollup windows & granularity:- Logs: rollup to event counts and top-K messages per hour/day; store histograms for error types per service.- Traces: aggregate span-level latency P50/P90/P99 per minute, per service; store error rates per minute then hourly/day summaries.- Metrics: downsample raw to 1m aggregates (min/max/avg/count) after 30 days; hourly aggregates after 1 year.

Ad-hoc rehydration & investigation plan:- Maintain immutable raw data archive in object store (S3/Glacier) for 7 years for critical streams; for general logs/traces, keep backups for 1 year.- Implement a rehydration workflow: index metadata catalog + manifest per day -> quick search to identify candidate blobs -> on-demand restore (hours) with priority path for incidents (1–4 hours SLA using expedited restore) and longer for audits (24–48 hours).- Provide temp higher-fidelity window: allow live short-term retention extension (policy toggle) for services under investigation.

Operational & cost controls:- Apply sampling & dynamic retention: automatic retention extension for periods surrounding detected incidents (e.g., retain raw for affected services +14 days).- Tiered storage: hot (real-time), warm (fast queries), cold (cheap object storage), archive (deep archive).- Governance: tag sensitive/audit-relevant data to override default rollup; run monthly cost reports per data-class and enforce quotas.

Trade-offs & justification:- Balances cost by aggressive rollup for long-term storage while preserving recent raw fidelity for debugging.- Sampling reduces long-term trace/log volume but keeps statistical signal.- Archive + rehydrate meets audit/compliance needs at lower ongoing cost with acceptable restore SLAs.

Unlock Full Question Bank

Get access to hundreds of Engineering and Business Outcomes interview questions and detailed answers.

Join thousands of developers preparing for their dream job.