Decision Making Under Uncertainty Questions

Focuses on the frameworks, heuristics, and judgment used to make timely, defensible choices when information is incomplete, conflicting, or still evolving, in any domain. Covers diagnosing what is genuinely unknown before deciding, setting explicit decision criteria and thresholds, weighing probabilities against impact (expected value and cost benefit thinking), and defining upfront triggers for reversing course, escalating, or waiting for more evidence. Also covers calibrating risk tolerance to the stakes involved, choosing between a small test or pilot versus committing directly to a decision, communicating uncertainty and trade offs to stakeholders in plain terms, and how senior candidates fold organizational constraints (budget, time, politics, precedent) into a call when the fully right answer cannot be known in advance. The underlying judgment applies to any high-stakes decision made with partial information: a hiring call with an incomplete reference check, a budget reallocation with uncertain ROI, a legal or compliance risk judgment, a vendor or partner selection, a go/no-go on a product bet, or a technical rollout. No single domain should dominate the framing.

MediumTechnical

50 practiced

Explain the trade-offs between strong consistency and eventual consistency for a user-facing feature (e.g., profile updates visible across regions). As an analyst, how would you measure and quantify user impact using observational data to inform the consistency choice?

Sample Answer

High-level trade-offs- Strong consistency: users see the same profile everywhere immediately. Pros: predictable UX, fewer support issues, easier reasoning about behavior (e.g., personalization). Cons: higher latency, reduced availability during network partitions, increased infrastructure cost.- Eventual consistency: low latency and higher availability; updates propagate asynchronously so different regions may see different profile states briefly. Cons: possible user confusion, stale reads, consistency-dependent bugs (e.g., wrong recommendations or failed identity checks).

As an analyst: how to measure user impact from observational data1. Instrumentation & data sources- Log-level events: profile_update (user_id, region, timestamp, version), profile_read (user_id, region, timestamp, version_read), request latencies, error codes.- Business metrics: session length, conversions, messages sent, support tickets, churn, NPS/feedback.- Join with user metadata: region, device, locale, user tier.

2. Key metrics- Time-to-consistency (TTC): distribution of (first_read_after_update_time - update_time) per region.- Staleness rate: fraction of reads within X minutes that returned older version.- UX impact metrics: conversion rate, task completion, message send success, average session duration before/after update, and support-ticket rate for profile issues per 1k updates.- Correlated metrics: bounce rate and short sessions following profile edit.

3. Analysis approach- Cohorts by region, device, and user activity. Compute TTC and staleness by cohort.- Pre-post comparisons around deployment windows when consistency mode changed (if available).- Quasi-experimental methods: difference-in-differences using regions with different rollout timing; regression controlling for confounders (time of day, user segment).- Propensity-score matching if rollout was non-random.- Event-level funnel analysis: e.g., profile_update → profile_read_consistent within 30s → conversion; compare conversion lift when read was consistent vs stale.

4. Decision rules and KPIs- Define acceptable thresholds: e.g., 95% of reads consistent within 30s, support-ticket uplift < X%, conversion delta insignificant (p>0.05, or < Y% relative).- Cost-benefit: estimate infrastructure cost delta vs. revenue/retention impact from inconsistent reads.

5. Reporting & monitoring- Dashboards: TTC heatmap by region, staleness trend, business-metric deltas, and support volume.- Alerts when staleness or user-impact metrics breach thresholds.

Example insight- If TTC median is <5s and no measurable lift in support or conversion for inconsistent reads, eventual consistency may be acceptable. If high-stakes flows (billing, authentication) show negative impact, use strong consistency selectively for those paths.

This approach quantifies trade-offs with observational data, uses robust quasi-experimental methods to infer impact, and produces operational KPIs to guide consistency choices.

HardTechnical

54 practiced

Design a metric and analysis plan to quantify 'user trust' impact after introducing eventual consistency for follower counts and profile data. Describe what proxies you would use (e.g., repeat-visit rate, support tickets, social feedback), the longitudinal analyses needed, and how to decide acceptable degradation thresholds.

Sample Answer

Situation: We want to measure whether switching to eventual consistency for follower counts/profile data reduces users’ trust.

Metric design (proxies — primary and secondary)- Primary proxy: Repeat-visit rate within 7/30 days (session return probability) — sensitive to perceived reliability.- Secondary proxies: - Support volume and severity for “profile/follower” tickets per 1k DAUs. - Social feedback sentiment (mentions complaining about inconsistent counts) normalized per 1k impressions. - Engagement on profile-driven actions (follow/unfollow conversions, profile clicks). - Quantitative trust proxy: rate of account-level actions that require trust (e.g., promotional link clicks, ad conversions).- Quality-of-life indicators: time-to-first-resolution on tickets, bot-detected confusion (rapid follow/unfollow loops).

Definitions & instrumentation- Define event-level flags: inconsistent_view_exposed where UI shows stale count vs authoritative. Log exposure cohort.- Compute metrics per user-day and aggregate cohorts (exposed vs unexposed).

Longitudinal analysis plan1. Pre-rollout baseline: collect 4–8 weeks of historical metrics.2. A/B or phased rollout: randomly assign clusters/users where possible; otherwise use stepped rollout.3. Analyses: - Difference-in-differences (DiD) comparing exposed vs control over time to isolate treatment effect. - Interrupted Time Series (ITS) on system-level metrics to detect level/trend shifts post-rollout. - Survival analysis for repeat-visit (time-to-return) to capture retention shifts. - Segmented analyses by device, region, power users vs casuals. - Sentiment time series and ticket-topic modeling (NLP) to quantify complaint volume and tone.4. Statistical approach: - Use non-inferiority tests where null hypothesis is “treatment is worse than control by margin δ.” - For proportions (repeat-visit), use z-tests / logistic regression with cluster-robust SEs; for counts (tickets), use Poisson/negative binomial regression. - Adjust for multiple comparisons (Benjamini–Hochberg) across proxies.

Deciding acceptable degradation thresholds- Translate business/UX tolerance into numeric δ per metric, e.g.: - Repeat-visit rate: non-inferiority margin δ = 1–2% absolute (or 3–5% relative) depending on revenue sensitivity. - Support tickets: acceptable increase ≤ 10% absolute or <= +0.5 tickets/1k DAU. - Sentiment: NPS-equivalent drop ≤ 0.5–1 point.- Set SLOs and alerting tiers: - Soft alert: metric crosses 50% of δ for two consecutive days. - Hard rollback threshold: metric crosses δ with p < 0.05 and sustained for a week.- Determine sample size & power: compute required users to detect δ with 80–90% power before rollout.

Operationalization and decision process- Dashboard tracking cohorted metrics with control bands and ITS trend lines; automated daily checks and weekly deep-dive.- Run root-cause on any flags: correlate exposure logs, client versions, network segments.- Decision rule: if primary proxy fails non-inferiority and secondary proxies corroborate, pause/rollback. If marginal breach but no corroborating signals, implement mitigations (display cache staleness indicators, reduce TTL) and continue monitoring.

Learning & iteration- If acceptable degradation observed, document mitigations and update SLOs.- If degradation concentrated in segments, consider targeted fixes (strong consistency for key flows).This plan links measurable proxies, rigorous longitudinal methods, and clear business-tied thresholds so stakeholders can make data-driven rollout decisions.

EasyTechnical

37 practiced

Define SLO, SLA, and error budget and explain how you would use an error budget to decide whether to proceed with a risky deployment, delay it, or roll it back. As a data analyst, what metrics would you compute from monitoring data to make that recommendation?

HardTechnical

43 practiced

Case: Engineering proposes sharding a service to improve throughput. Sharding increases cross-shard join cost and operational complexity. As a data analyst, design an evidence-gathering plan that includes workload simulation, metrics to estimate cross-shard cost, and concrete go/no-go criteria you would recommend.

Sample Answer

Clarify scope & success criteria- Ask engineering/product: target throughput increase, latency SLOs, peak vs steady traffic, acceptable error rates, allowed operational overhead and rollout window.- Define baseline: current QPS, p50/p95/p99 latencies, join frequencies, data size distribution, current CPU/IO/network utilization, and incident rate.

Workload simulation plan1. Data collection & synthesis - Extract representative query logs (7–30 days): timestamps, SQL, parameters, latency, rows scanned, read/write bytes, user/tenant IDs. - Classify queries: single-shard candidates, cross-shard joins, analytical vs OLTP, fanout patterns. - Create synthetic dataset preserving key distributions (tenant size, hot keys) and anonymize PII.

2. Simulation methods - Offline replay: run historic traffic against a sharded staging cluster (mirror schema) using a traffic replay tool (Tsung/Locust/custom runner) with timing preserved. - Shadow traffic: mirror a % of live requests to sharded service in production-like environment (no writes to master) for safety. - Microbenchmarks: targeted tests for worst-case cross-shard join patterns and fanout scenarios.

Metrics to measure cross-shard cost- Query-level: - Absolute join latency increase (ms) and relative (%) for cross-shard joins. - Additional CPU secs and wall-clock time per query. - Extra rows transmitted and network bytes per join. - Increased p50/p95/p99 for affected endpoints.- System-level: - End-to-end throughput (QPS) vs single-node baseline. - Aggregate network bandwidth and inter-shard RPCs/sec. - Increased tail-latency impact on user-perceived flows (e.g., checkout). - Resource contention (CPU, memory, IO), and queuing/backpressure events.- Operational: - Number of new operational runbooks required, mean time to recover (MTTR) from simulated failures (node loss, reshard, schema change). - Deployment/rebalancing time and estimated maintenance windows.

Analysis & estimation- Compute per-query cost delta and weight by frequency to get expected added system load.- Model worst-case at 95th percentile traffic: project extra CPU/network required to maintain SLOs.- Cost estimation: infra cost (additional instances, network) + expected increase in incident time (FTE hours) × hourly rate.- Sensitivity analysis: vary shard keys, skew scenarios, and tenant growth rates.

Go / No-go criteria (concrete thresholds)Go if all conditions met:- Measured throughput gains ≥ target (e.g., +40% QPS) while keeping end-to-end p95 latency within SLO (+≤10% relative).- Cross-shard join-induced added CPU/network <= budgeted capacity (e.g., <20% extra infra cost) OR ROI > threshold (payback within N months).- Tail-latency for critical user flows not degraded beyond business threshold (e.g., checkout p99 increase ≤50ms).- Operational runbooks validated and MTTR increase acceptable (e.g., MTTR increase ≤20% or compensated by on-call resourcing).No-go if any:- Critical path p95/p99 exceeds SLOs in shadow tests.- Cross-shard join frequency causes >X% of queries to become >2× slower and affects revenue-sensitive endpoints.- Projected infra + operational cost exceeds ROI threshold or exceeds budgeted limits.- Resharding/rebalancing causes unacceptable downtime or complex manual steps.

Recommendations- If borderline, consider hybrid: shard only high-throughput tables, introduce a lookup cache/service to avoid cross-shard joins, or adopt application-layer denormalization for hot paths.- Preserve reproducible tests and dashboards: baseline vs sharded comparisons, drilldowns by query type, and automated alerts for regressions.- Run gradual rollout with kill-switch, 2-week shadow + canary phases, and measure business KPIs (conversion, latency-sensitive transactions) before full cutover.

EasyTechnical

53 practiced

Given tables: deployments(deployment_id, service_id, deployed_at, version), incidents(incident_id, deployment_id, start_ts, end_ts, impact_cost_usd). Write a SQL query to compute expected downtime cost per deployment window (30 days after deployment) and rank deployments by expected cost. Explain assumptions about overlapping incidents and attribution.

Sample Answer

Approach: For each deployment, consider a 30-day window starting at deployed_at. Sum incident impact_cost_usd for incidents whose time overlaps that window. If an incident overlaps multiple deployments’ windows, attribute its full cost to each overlapping deployment (explicit assumption) — alternatively prorate by overlap if desired. Rank deployments by total expected downtime cost.

sql

WITH windowed_deployments AS (
  SELECT
    d.deployment_id,
    d.service_id,
    d.deployed_at,
    d.deployed_at + INTERVAL '30 day' AS window_end
  FROM deployments d
),
incidents_in_window AS (
  SELECT
    w.deployment_id,
    i.incident_id,
    i.start_ts,
    i.end_ts,
    i.impact_cost_usd
  FROM windowed_deployments w
  JOIN incidents i
    -- consider any incident that overlaps [deployed_at, window_end]
    ON i.start_ts <= w.window_end
   AND (i.end_ts IS NULL OR i.end_ts >= w.deployed_at)
)
SELECT
  d.deployment_id,
  d.service_id,
  d.deployed_at,
  SUM(ii.impact_cost_usd) AS expected_downtime_cost_30d
FROM windowed_deployments d
LEFT JOIN incidents_in_window ii
  ON d.deployment_id = ii.deployment_id
GROUP BY d.deployment_id, d.service_id, d.deployed_at
ORDER BY expected_downtime_cost_30d DESC NULLS LAST;

Key points and assumptions:- Overlap rule: an incident overlaps the 30-day window if its start <= window_end and its end >= deployed_at (treat NULL end_ts as ongoing).- Attribution: This query attributes the full incident cost to every deployment whose 30-day window it overlaps. Use this when measuring per-deployment exposure rather than unique cost.- Alternative (prorating): To avoid double-counting, prorate an incident’s cost by the fraction of its duration that falls inside each deployment window; requires computing overlap duration and dividing cost proportionally.Edge cases:- Ongoing incidents (end_ts NULL) handled as overlapping until window_end.- Timezones: ensure deployed_at/start_ts/end_ts are in same timezone or converted.- Large datasets: index on incidents.start_ts and end_ts and deployments.deployed_at for performance.

Unlock Full Question Bank

Get access to hundreds of Decision Making Under Uncertainty interview questions and detailed answers.

Join thousands of developers preparing for their dream job.