Technical Analysis and Methodology Questions

Focuses on the technical depth and concrete analytical methods you use to produce reliable quantitative results. Interviewers look for how you validate assumptions, stress test key inputs, choose modeling techniques, and apply appropriate tools and processes. This includes building and auditing models, performing sensitivity and scenario analysis, data cleaning and transformation, statistical or econometric methods where relevant, and using software such as advanced spreadsheet techniques, scripting languages, or database queries to manipulate data. Candidates should be able to articulate their preferred tools and methods at a level appropriate to the interview and explain trade offs between model complexity and interpretability.

EasyBehavioral

30 practiced

You must explain uncertainty in model outputs to a product manager who is used to point estimates. Give a concise script and visualization approach to communicate aleatoric vs epistemic uncertainty and how each should influence product decisions.

EasyTechnical

28 practiced

Given a dataset with missing values across multiple numeric and categorical features, outline a systematic methodology to assess the missingness mechanism (MCAR, MAR, MNAR) and decide on imputation strategies. Include how you would validate imputation quality before training models.

Sample Answer

Start with exploratory analysis to characterize patterns:- Missingness matrix/time plots and pairwise heatmaps to see blocks or co-occurrence.- Tabulate missing rate by feature and by groups (target, categorical levels, time).

Assess mechanism (MCAR / MAR / MNAR):- Little’s MCAR test for a global MCAR signal (reject MCAR → not MCAR).- Conditional checks: compute whether missingness indicator correlates with observed features or target (chi-square/KS tests, logistic regression predicting missingness). Significant correlations → MAR likely.- MNAR suspicion: missingness depends on unobserved values or shows patterns not explained by observed variables; use domain knowledge (e.g., labs missing when values abnormal), examine changes over time, or perform follow-up data collection if possible.

Decide imputation strategy:- If MCAR (rare): simple methods acceptable (mean/median for numeric, mode for categorical) or drop rows if small fraction.- If MAR: use model-based imputers that condition on observed data — MICE / IterativeImputer, KNN, or tree-based models. Preserve multivariate relationships.- If MNAR: consider explicit modeling of missingness (joint models), include missingness indicators as features, perform sensitivity analysis and multiple imputation to reflect uncertainty.

Validate imputation quality before modeling:- Mask-and-recover: randomly hide a subset of observed values (stratified by feature) and compare imputations to true values using RMSE, MAE, classification metrics for categorical features.- Distributional checks: compare marginal and conditional distributions (density plots, KS test) of imputed vs observed.- Downstream impact: train models on datasets with different imputation methods and compare cross-validated performance, calibration, and feature importances.- Uncertainty: use multiple imputation to capture variance; propagate uncertainty into model predictions.

Report choices, assumptions, and sensitivity analyses so stakeholders understand risks from missing-data mechanisms.

HardTechnical

27 practiced

You need to detect concept drift in a deployed classifier. Compare three statistical approaches you could use for drift detection, and explain how you would choose thresholds and evaluate detector performance without labeled data.

Sample Answer

Situation: In production you must detect when input–output relationships change (concept drift) but labels are scarce or delayed. Three statistical approaches I’d compare:

1) Distribution tests on input features or model scores (batch tests like Kolmogorov–Smirnov, Chi-square, KL/JS divergence, PSI)- What: Compare recent window vs baseline window distributions per feature or on predicted-probability (score) vector.- Pros: Simple, interpretable, works without labels.- Cons: Detects covariate shift (P(X)) but not always label shift or conditional P(Y|X) drift; multiple tests → multiple comparisons.

2) Sequential change detectors on prediction stream (CUSUM, Page-Hinkley, ADWIN)- What: Treat model outputs (e.g., top predicted class probability, loss surrogate like negative log-prob) as a time series and run online detectors for abrupt/gradual shifts.- Pros: Designed for streaming, control detection delay vs false alarms, adapts to evolving baseline.- Cons: Need good summary statistic; may be sensitive to noise.

3) Two-sample tests on calibrated score distributions or surrogate labels (class-wise KS, energy distance, kernel MMD)- What: Use calibrated confidences or model-derived pseudo-labels and apply stronger multivariate tests (MMD) that capture joint changes across features/scores.- Pros: More power for subtle, high-dim changes; can detect conditional shifts when using class-conditional scores.- Cons: Computational cost, needs kernel/hyperparameter tuning.

Choosing thresholds & operating point without labels- Use bootstrap/permutation on baseline data to get empirical null distributions of your test statistic → pick threshold for desired Type I error (e.g., α = 0.01).- Calibrate sequential detector parameters (window sizes, drift magnitude h) to control average run length to false alarm using historical traffic or simulated no-drift segments.- Use synthetic drift injection: inject controlled shifts into production-like data (feature shift, label-flip, conditional shift) to empirically map detector threshold → (false positive rate, detection delay).- Use delayed labels or occasional labeled sampling: hold out a small stratified label budget to validate alarm quality and adjust thresholds iteratively.

Evaluating detector performance without labels- Proxy metrics: false alarm rate estimated on known-stable periods; detection delay measured on synthetic injected drifts; alarm concordance across independent detectors (ensemble voting increases precision).- Use champion–challenger or shadow deployments where challenger trained on recent data can reveal degradation in decisions (disagreement rate spikes as proxy).- If you can obtain delayed labels, compute true positive rate, precision, and time-to-detect on those labeled intervals and extrapolate.- Track business KPIs (conversion, error rates) as downstream signals that correlate with model degradation.

Practical recipe- Monitor: per-feature PSI and score-distribution KS daily; run ADWIN on probability of chosen class online.- Calibrate thresholds by bootstrap + synthetic injections to set alarms that keep false positives acceptable.- When alarm fires: trigger targeted labeling (active sampling), run class-conditional tests, and if validated, retrain/roll back.- Log detector metrics, detection latency, and subsequent model actions to iterate thresholds and reduce operational cost.

This combination balances simplicity, online responsiveness, and statistical power while providing concrete ways to choose thresholds and evaluate detectors even when labels are limited.

EasyTechnical

29 practiced

You build a binary classifier to detect fraudulent transactions. Which evaluation metrics would you report to the business, why, and how would you choose thresholds given class imbalance and different costs for false positives vs false negatives?

EasyTechnical

34 practiced

Describe the difference between cross validation approaches for i.i.d. data and time series data. Provide concrete guidance on when to use K-fold, stratified K-fold, blocking, or rolling-window validation for model selection and hyperparameter tuning.

Unlock Full Question Bank

Get access to hundreds of Technical Analysis and Methodology interview questions and detailed answers.

Join thousands of developers preparing for their dream job.