Requirements Analysis & Problem Decomposition Questions

Break down complex business requirements into smaller technical components. Identify ambiguities and ask clarifying questions. Prioritize requirements logically. Plan implementation approach step by step. Create technical specifications from business requirements.

MediumTechnical

71 practiced

Stakeholder asks to 'improve recommendation relevance.' Propose a measurable objective with specific offline evaluation metric(s), an online experiment design (A/B or multi-armed), and success thresholds. Explain the trade-offs between using offline metrics (e.g., NDCG) and online metrics (e.g., CTR, revenue).

Sample Answer

Objective: Increase perceived recommendation relevance and business impact by improving ranked relevance and conversion. Measurable objective — within 8 weeks deploy a new recommender that yields +5% relative lift in offline relevance (NDCG@10) and translates to ≥+2% relative lift in online Click-Through Rate (CTR) and ≥+1% lift in Revenue Per Thousand Users (RPM), both statistically significant (p<0.05) with 80% power.

Offline evaluation:- Primary: NDCG@10 (higher weight on top results) and Mean Reciprocal Rank (MRR) for single-click tasks.- Secondary: Hit Rate@k, diversity (ILD) and calibration if applicable.Use cross-validation on holdout users and time-split evaluation to avoid leakage.

Online experiment design:- A/B test (two arms) with random assignment at user-level. If testing multiple algorithms/parameters, use multi-armed bandit or multi-arm A/B with pre-specified contrasts.- Sample size: compute based on baseline CTR and desired MDE (for example, baseline CTR=4%, detect 2% relative lift → MDE ≈ 0.08% absolute; compute N per arm for 80% power).- Run for at least one full business cycle (min 2 weeks) and until sample size/power achieved. Monitor novelty, retention, and safety guardrails (error rates, latency).- Analyze with pre-registered metrics and correction for multiple comparisons if multi-arm.

Success thresholds:- Offline: ≥+5% rel. NDCG@10 improvement on time-split holdout.- Online: ≥+2% relative CTR lift AND ≥+1% RPM lift, p<0.05, 80% power. No degradation (>1%) on retention or negative quality signals.

Trade-offs:- Offline metrics (NDCG) are fast, low-cost, and good for iterative model development and ranking quality, but may not capture exposure bias, UI effects, position bias, or downstream business impact. They can overfit to historical feedback (popularity bias).- Online metrics (CTR, revenue) capture real user behavior, long-run effects (engagement, retention), and system integration but are slower, costlier, and subject to noise and external confounders. Online experiments are the ground truth for impact, so use offline metrics to shortlist candidates and online A/B to validate business outcomes.

EasyTechnical

73 practiced

A stakeholder requests 'real-time scoring' for a fraud pipeline. List the clarifying SLA questions you would ask, covering latency targets, throughput (TPS), acceptable false-positive rate, statefulness, data freshness, auditing, and data retention. Explain how each clarification alters the solution design.

Sample Answer

Clarifying questions (and why each matters for design):

1) Exact latency target (e.g., <50ms, <200ms) and percentiles (p50/p95/p99)?- Impact: <50ms pushes lightweight models (logistic/linear), in-memory feature store, colocated scoring; higher budgets allow more complex models (ensembles, NN) and extra preprocessing.

2) Expected throughput (TPS) and burst profile (steady vs spiky)?- Impact: Determines autoscaling, load-testing, whether we need GPU inference, batching, rate-limiting, or a fronting cache to avoid overload.

3) Acceptable false-positive and false-negative rates (business cost of each)?- Impact: Model thresholding, calibration, need for hierarchical scoring (fast cheap model then expensive confirmatory model), or human-in-loop review for borderline cases.

4) Stateful vs stateless scoring (need user/session history or aggregated features)?- Impact: If stateful, require low-latency feature store or streaming state (Redis, RocksDB, Flink state), windowed aggregations, and more complex consistency guarantees.

5) Data freshness requirements for features (real-time, last-minute, hourly)?- Impact: Real-time freshness drives streaming pipelines (Kafka, ksqlDB), online feature stores; looser freshness allows micro-batch ETL.

6) Auditing, explainability, compliance needs (persist inputs, model version, SHAP, reasons)?- Impact: Must log inputs/outputs, model version, feature values; storage/ETL for audit trail; choose explainable models or integrate explainers with latency budget.

7) Data retention and GDPR/PII constraints?- Impact: Storage size, retention policies, anonymization, encryption, deletion flows; may limit how long historical features are stored.

8) Availability/SLAs (99.9%?), fallback behavior if scoring fails?- Impact: Design for redundancy, health checks, circuit breakers; define safe defaults (e.g., block/allow) and offline batch scoring for recovery.

9) Monitoring and alerting requirements (latency, accuracy drift, data skew)?- Impact: Add telemetry, drift detectors, automated retraining pipelines, and ops playbooks.

Each answer shifts trade-offs between model complexity, infra (streaming vs batch), feature store design, logging, and operational costs.

MediumTechnical

100 practiced

Design a step-by-step implementation plan for a churn prediction project whose goal is to reduce churn by 5% next quarter. Include milestones, data requirements, model candidate types, validation plan, deployment strategy (pilot/rollout), rollback criteria, and how you will measure business impact. Assume data is available from CRM, transactions, and web logs.

Sample Answer

Situation/Goal: Reduce customer churn by 5% next quarter via a predictive model + targeted interventions.

1) Milestones (12 weeks)- Week 0–1: Kickoff, define churn definition (e.g., no purchase in X days or subscription cancel), success metrics, stakeholders, data access.- Week 2–3: Data ingestion & profiling (CRM, transactions, web logs), privacy review, baseline analysis.- Week 4–5: Feature engineering (recency/frequency/monetary, engagement, support tickets, campaign exposure), exploratory analysis.- Week 6: Model prototypes & shortlist.- Week 7: Validation & uplift modeling for treatment strategy.- Week 8: Pilot deployment (2–4 weeks) to a holdout test cohort.- Week 10–11: Analyze pilot, iterate.- Week 12: Full rollout or rollback decision and handoff.

2) Data requirements- CRM: demographics, segment, lifecycle dates, support interactions.- Transactions: purchase history, returns, LTV, frequency, recency.- Web logs: sessions, pageviews, product views, churn-related behavior (cart abandonment).- Labels: churn window, treatment history. Ensure joins by customer_id, timestamp alignment.

3) Model candidates- Classification: Gradient Boosted Trees (XGBoost/LightGBM) for baseline.- Neural net: simple feedforward or embedding layers if high-cardinality categorical features.- Temporal: RNN/Transformer or survival analysis (Cox) for time-to-churn.- Causal/Uplift: Two-model or meta-learners to predict incremental impact of retention offers.

4) Validation plan- Use time-based train/validation/test splits to avoid leakage.- Metrics: Precision@k, ROC-AUC, PR-AUC, calibration, and business KPIs (expected churn reduction, uplift).- Backtest: simulate interventions historically.- Robustness: cohort/subgroup performance, feature importance, SHAP explanations.

5) Deployment strategy- Pilot: Score a random holdout sample (e.g., 5–10% of population). Run treatment (offers/emails) vs control A/B test for 2–4 weeks.- Rollout: Phased increase (25%, 50%, 100%) with monitoring dashboards for model drift, conversion, costs.

6) Rollback criteria- No statistically significant uplift in retention vs control after pilot period.- Increase in cost per retained customer above threshold (e.g., >2x expected).- Adverse business signals (customer complaints spike, KPI regressions).

7) Measure business impact- Primary: Absolute churn rate change vs control (target −5%).- Secondary: CLTV uplift, cost per retained customer, response rate, ROI.- Statistical significance testing, confidence intervals, and lift by segment.- Post-rollout: monitor monthly churn, recalc model refresh cadence (e.g., retrain monthly).

Final notes: Instrument tracking to attribute impact, maintain explainability for business ops, and schedule retraining and data-quality checks.

HardTechnical

99 practiced

You must prioritize a portfolio of ten ML initiatives with limited engineering and compute resources. Define a set of criteria (e.g., revenue impact, time-to-value, legal risk, strategic alignment), a scoring mechanism, and show example scoring and ranking for three projects: 1) high revenue, low complexity; 2) moderate revenue, high legal risk; 3) low revenue, critical ops automation. Explain how interdependencies and sunk costs affect prioritization.

Sample Answer

Framework: clarify goals (maximize business value subject to limited eng/compute), constraints (2 FTE eng, fixed GPU budget), and decision criteria.

Criteria (with weights summing to 100):- Revenue impact (35%)- Time-to-value / implementation speed (20%)- Engineering & compute complexity (inverse: lower is better) (15%)- Legal / compliance risk (inverse) (10%)- Strategic alignment (long-term capabilities) (10%)- Operational risk / reliability impact (5%)- Reusability / platform leverage (5%)

Scoring mechanism:- Score each criterion 0–5 (5 best). Weighted score = weight * (score/5). Total possible = 100.

Example scoring for three projects:

1) High revenue, low complexity (Project A)- Revenue 5 → 35*(5/5)=35- Time-to-value 5 → 20- Complexity 5 (low complexity) → 15- Legal risk 5 (low risk) → 10- Strategic alignment 3 → 6- Ops impact 2 → 2- Reusability 3 → 3Total = 35+20+15+10+6+2+3 = 91

2) Moderate revenue, high legal risk (Project B)- Revenue 3 → 21- Time-to-value 2 → 8- Complexity 2 → 6- Legal risk 1 (high risk) → 2- Strategic alignment 4 → 8- Ops impact 3 → 3- Reusability 4 → 4Total = 52

3) Low revenue, critical ops automation (Project C)- Revenue 1 → 7- Time-to-value 4 → 16- Complexity 4 → 12- Legal risk 5 → 10- Strategic alignment 2 → 4- Ops impact 5 → 5- Reusability 2 → 2Total = 56

Ranking by score: A (91), C (56), B (52). Recommendation: prioritize A immediately, parallel pilot C if a small SRE/engineer can implement (ops risk reduction often multiplies downstream value), deprioritize B until legal mitigations or clearer ROI.

Interdependencies & sunk costs:- Interdependencies: if Project B provides features used by A or C, re-evaluate by adding dependency bonus (increase strategic/reusability weight). Model a dependency graph and compute marginal value of enabling projects.- Sunk costs: ignore sunk development effort in prioritization; focus on incremental cost and marginal benefit. However, account for partially-built assets by reducing remaining implementation cost and increasing reusability score.

Operational suggestions:- Run rapid 2–4 week spikes for top candidates to validate assumptions (especially legal/complexity).- Use staged funding: allocate core team to A, reserve small buffer for C pilot, and gate B on legal sign-off + a secondary ROI review.

MediumTechnical

89 practiced

Given this table schema: orders(order_id PK, user_id INT, sku TEXT, price DECIMAL, status TEXT, created_at TIMESTAMP), write SQL queries to: 1) compute weekly active users (WAU), 2) compute weekly revenue per cohort where cohort is week of first purchase, and 3) write a data-quality query to detect duplicate orders (same user_id, sku, created_at within 1 minute). Include assumptions and brief explanation for each query. Use standard ANSI SQL.

Sample Answer

Assumptions- We're using Postgres/ANSI SQL with DATE_TRUNC and INTERVAL support.- orders contains one row per order event; WAU counts distinct user_id with at least one order that week.- price is revenue per order; NULL prices are treated as 0.- Week buckets use week starting Monday via DATE_TRUNC('week', created_at).

1) Weekly Active Users (WAU)Counts unique users per calendar week.

sql

SELECT
  DATE_TRUNC('week', created_at) AS week_start,
  COUNT(DISTINCT user_id) AS weekly_active_users
FROM orders
GROUP BY 1
ORDER BY 1;

Explanation: DATE_TRUNC('week', ...) buckets events to the week start. COUNT(DISTINCT user_id) yields WAU.

2) Weekly revenue per cohort (cohort = week of first purchase)Compute each user's cohort (week of first purchase), then aggregate revenue by cohort and relative week (week index since cohort).

sql

WITH user_cohorts AS (
  SELECT
    user_id,
    DATE_TRUNC('week', MIN(created_at)) AS cohort_week
  FROM orders
  GROUP BY user_id
),
orders_with_cohort AS (
  SELECT
    o.*,
    uc.cohort_week,
    -- weeks_since_cohort: 0 = cohort week, 1 = one week after, etc.
    FLOOR(EXTRACT(EPOCH FROM (DATE_TRUNC('week', o.created_at) - uc.cohort_week)) / 604800)::INT AS weeks_since_cohort
  FROM orders o
  JOIN user_cohorts uc USING (user_id)
)
SELECT
  cohort_week,
  weeks_since_cohort,
  SUM(COALESCE(price, 0)) AS revenue,
  COUNT(DISTINCT user_id) AS active_users_in_bucket
FROM orders_with_cohort
GROUP BY cohort_week, weeks_since_cohort
ORDER BY cohort_week, weeks_since_cohort;

Explanation: user_cohorts finds each user’s first-order week. For each order we compute weeks_since_cohort by subtracting the cohort_week from the order's week, converting to seconds and dividing by 604800 (seconds/week). Aggregation yields revenue per cohort-week. We also include active user count per bucket.

3) Data-quality query — detect possible duplicate ordersFind orders for same user and sku within 1 minute of each other (could be exact duplicates).

sql

SELECT
  o1.order_id AS order_id_1,
  o2.order_id AS order_id_2,
  o1.user_id,
  o1.sku,
  o1.created_at AS ts1,
  o2.created_at AS ts2,
  ABS(EXTRACT(EPOCH FROM (o1.created_at - o2.created_at))) AS seconds_between
FROM orders o1
JOIN orders o2
  ON o1.user_id = o2.user_id
  AND o1.sku = o2.sku
  AND o1.order_id < o2.order_id  -- avoid self-join duplicates
WHERE ABS(EXTRACT(EPOCH FROM (o1.created_at - o2.created_at))) <= 60
ORDER BY user_id, sku, seconds_between;

Explanation: Self-join on user_id and sku and filter pairs within 60 seconds. Use order_id inequality to prevent mirrored pairs. Review these pairs to decide whether to dedupe (e.g., based on identical price, status, or other flags).

Edge cases / notes- Timezones: ensure created_at is normalized (UTC).- Refunds / cancellations: revenue query might need to exclude status IN ('refunded','cancelled').- Cohort window choice: using calendar weeks; consider using rolling 7-day windows if business requires.- For very large tables, index on (user_id, created_at) and (created_at) and use partitioning for performance.

Unlock Full Question Bank

Get access to hundreds of Requirements Analysis & Problem Decomposition interview questions and detailed answers.

Join thousands of developers preparing for their dream job.