MediumTechnical

81 practiced

Implement a scikit-learn-compatible feature transformation pipeline that: imputes missing values, encodes categorical variables with safe handling for unseen categories at inference time, scales numerical features, and can be serialized (joblib) for production. Provide Python code demonstrating fit and transform methods.

Sample Answer

Approach: use scikit-learn Pipelines + ColumnTransformer. Use SimpleImputer for missing values, OneHotEncoder(handle_unknown='ignore') for safe unseen-category handling, StandardScaler for numeric scaling. Wrap as a single Pipeline that's serializable with joblib.

python

from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OneHotEncoder, StandardScaler
import pandas as pd
import joblib

# Example columns
numeric_features = ["age", "income"]
categorical_features = ["country", "device"]

# Build transformers
numeric_transformer = Pipeline([
    ("imputer", SimpleImputer(strategy="median")),
    ("scaler", StandardScaler())
])

categorical_transformer = Pipeline([
    ("imputer", SimpleImputer(strategy="constant", fill_value="missing")),
    ("onehot", OneHotEncoder(handle_unknown="ignore", sparse=False))
])

preprocessor = ColumnTransformer([
    ("num", numeric_transformer, numeric_features),
    ("cat", categorical_transformer, categorical_features)
], remainder="drop")

# Full pipeline (no estimator, just transform)
feature_pipeline = Pipeline([
    ("preprocessor", preprocessor)
])

# Usage
df_train = pd.DataFrame({
    "age": [25, None, 40],
    "income": [50000, 60000, None],
    "country": ["US", "FR", "US"],
    "device": ["mobile", None, "desktop"]
})

# Fit
feature_pipeline.fit(df_train)

# Transform -> numpy array
X_train_transformed = feature_pipeline.transform(df_train)
print("Transformed shape:", X_train_transformed.shape)

# Serialize
joblib.dump(feature_pipeline, "feature_pipeline.joblib")

# Later: load and transform new data (handles unseen categories safely)
loaded = joblib.load("feature_pipeline.joblib")
df_test = pd.DataFrame({
    "age": [30],
    "income": [70000],
    "country": ["DE"],        # unseen country at train time
    "device": ["tablet"]      # unseen device
})
X_test = loaded.transform(df_test)
print("Test transformed:", X_test)

Key points:- OneHotEncoder(handle_unknown="ignore") ensures unseen categories don't error and produce zeros for those columns.- SimpleImputer handles missing numeric/categorical values.- Pipeline + ColumnTransformer produces a single transformable object that's joblib-serializable for production.

MediumTechnical

64 practiced

You must reduce cloud inference costs by 40% for a model serving ~200k predictions/day while maintaining SLAs. Propose concrete strategies covering model-level optimization (quantization, distillation), infrastructure changes (right-sizing, reserved instances, serverless vs container), batching and caching, priority queues, and estimate expected cost reductions and risks for each approach.

Sample Answer

Situation: You must cut inference cloud costs 40% for ~200k predictions/day while keeping SLAs. Below are concrete, prioritized strategies with expected savings and risks.

1) Model-level: Quantization (FP16/INT8)- Action: Post-training quantization or quant-aware training to INT8 on CPU/GPU or using hardware accelerators (e.g., NVIDIA TensorRT).- Est. reduction: 15–30% per-op cost (smaller memory, higher throughput) → end-to-end cost impact ~10–20%.- Risk: Small accuracy drop (0–2%); mitigate with calibration and QA tests.

2) Model Distillation / Smaller architecture- Action: Train a distilled/student model with similar performance but fewer FLOPs.- Est. reduction: 30–60% per-inference cost → end-to-end impact ~15–25%.- Risk: Loss of edge-case fidelity; mitigate with targeted evaluation and fallback to teacher for low-confidence inputs.

3) Batching + Adaptive Batching- Action: Group requests at model server (max latency budget), dynamic batch size based on rate; use asynchronous inference.- Est. reduction: 10–35% (better GPU utilization) → ~8–15% overall.- Risk: Added tail latency; enforce SLA with max-wait thresholds and prioritize small batches for P95/P99-sensitive traffic.

4) Caching & Result Reuse- Action: Cache frequent inputs / embeddings; use bloom filters for fast miss checks.- Est. reduction: If 10–30% cache hit → 10–30% cost reduction proportional to hit rate.- Risk: Staleness/consistency; mitigate TTLs and validation for user-specific contexts.

5) Infra: Right-sizing + Reserved/Savings Plans- Action: Move predictable baseline instances to reserved or committed use; right-size instance types (GPU type or CPU inference optimized like AWS Inf1/GCP A2).- Est. reduction: 20–50% on infra spend for reserved vs on-demand; combined infra-level ~10–25% overall.- Risk: Commitment mismatch; start conservative + ramp schedule.

6) Serverless vs Containers (autoscaling)- Action: Use serverless GPU or containerized autoscaling with target utilization; mix spot instances for non-critical workloads.- Est. reduction: 10–40% when using spot for batch/off-peak + autoscaling to zero.- Risk: Spot interruptions; mitigate by checkpointing and using spots for low-priority tasks only.

7) Priority Queues & Hybrid Routing- Action: Route latency-sensitive requests to optimized low-latency path (smaller model / reserved instances) and bulk/non-urgent to cheaper path (distilled model / spot).- Est. reduction: Improves overall cost by ~5–15% by avoiding over-provisioning.- Risk: Complexity; ensure strict SLA routing rules and observability.

Combined plan (practical rollout): 1) implement quantization + batching + right-sizing + reserved instances (quick wins) → expect 30–45% reduction. 2) add distillation + caching + spot/hybrid routing → additional 10–20% but takes longer. Monitor accuracy, P95/P99 latencies, and cost per prediction. Use canary experiments, A/B test accuracy, and automated rollout with rollback thresholds.

HardTechnical

82 practiced

Case study: Your recommender increased engagement but reduced revenue per user. As the AI Engineer, propose metrics to quantify trade-offs (e.g., ARPU, LTV, engagement per cohort), algorithmic changes (diversity constraints, revenue-aware ranking, contextual bandits), offline evaluation protocols and simulation, and a safe rollout strategy to optimize both engagement and revenue.

Sample Answer

Clarify goal & constraints:- Objective: maximize long-term business value (LTV) while maintaining healthy engagement and retention. Short-term engagement increases are welcome only if they don’t erode revenue per user (ARPU) or future monetization.

Metrics to quantify trade-offs:- Immediate: ARPU (daily/weekly/monthly), Revenue per DAU, conversion rate, avg order value.- Engagement: DAU/MAU, session length, impressions per session, click-through rate (CTR).- Lifetime: cohort LTV at 7/30/90/365 days, retention curves, churn hazard rate.- Efficiency: revenue per impression, revenue per recommendation, engagement-to-revenue ratio.- Risk & fairness: distribution of revenue across items, supplier concentration.

Algorithmic interventions:- Revenue-aware ranking: add a linear combination score = α * predicted_engagement + β * predicted_revenue, learn α,β via business objective.- Constrained optimization: maximize engagement subject to ARPU >= baseline (or vice versa) using Lagrangian methods.- Diversity and novelty constraints: apply determinantal point processes or promote long-tail items to avoid cannibalization.- Contextual bandits / RL: treat revenue as reward signal (multi-objective), use contextual bandits with weighted reward (engagement + λ*revenue) and off-policy correction.- Causal uplift models: estimate revenue lift per recommendation to avoid biases from popularity.

Offline evaluation & simulation:- Counterfactual policy evaluation (IPW, doubly robust) to estimate revenue/engagement for new policies using logged propensities.- Replay simulation with user simulators calibrated on historical transition/transaction models to test long-term LTV effects.- Synthetic A/B scenarios: stress-test worst-case item cannibalization and supplier revenue shifts.

Safe rollout strategy:- Start with offline selection → small holdout/offline-simulated validation.- Canary: expose policy to small user fraction (stratified by cohort/business value), use sequential testing (Bayesian A/B or continuous monitoring) on ARPU, conversion, retention, not just engagement.- Adaptive ramp: if ARPU drops but engagement rises, tune α/β or add constraints; use multi-armed bandit traffic allocation to favor better-performing variants.- Monitoring & kill switches: realtime dashboards, alert thresholds on revenue and retention, automated rollback if metrics breach SLOs.- Post-rollout: periodic cohort LTV analysis and model retraining with updated reward signals.

Key trade-off reasoning: combine direct revenue signals with engagement to avoid short-term proxies; use constrained optimization and causal evaluation to ensure changes improve long-term business value.

MediumTechnical

132 practiced

You are tasked with reducing inference latency for a Transformer-based NLP model delivered to mobile clients. List concrete optimization techniques (quantization, knowledge distillation, pruning, operator fusion, ONNX/TensorRT conversion, caching), describe expected trade-offs in accuracy and memory, and outline how you'd validate improvements with benchmarks.

MediumTechnical

61 practiced

Your offline model metrics improved but online click-through rate (CTR) dropped after rollout. Outline a step-by-step investigation: verify event logging and sampling, ensure instrumentation parity, perform segmented A/B analysis, check user cohorts and confounders, validate traffic assignment, and design follow-up experiments or rollbacks.

Sample Answer

1) Triage & scope: confirm the drop size, timing, affected segments, and statistical significance (compute p-values, confidence intervals, Bayesian credible intervals). Compare daily CTR, impressions, and conversion funnels for treatment vs control.

2) Verify event logging & sampling:- Check raw logs for missing events, duplicate events, or schema changes since rollout.- Validate sampling ratios: compare production sampled stream vs full logs.- Example checks: count(user_id,event_type) by hour; verify no sudden drops.

3) Instrumentation parity:- Ensure online scoring uses identical model and feature code as offline evaluation (same preprocessing, normalization, feature versions).- Run shadow/side-by-side inference on a sample of live requests to compare scores and outputs.

4) Traffic assignment & experiment integrity:- Confirm randomization key, bucketing logic, user overlap, and no leakage between variants.- Verify that started vs exposed vs analyzed cohorts align (no post-exposure exclusions that bias CTR).

5) Segmented A/B analysis & cohorts:- Break down by device, geography, user tenure, traffic source, time of day.- Look for heterogeneous treatment effects (e.g., negative effect only on new users or mobile).- Check confounders: concurrent launches, UI changes, personalization rules, caching/CDN issues.

6) Root-cause validation:- Reproduce the issue in a smaller controlled rollout or with a canary group.- Instrument additional logging for feature distributions, predicted CTR, and downstream ranking changes.

7) Action plan:- If instrumentation/assignment issue found: fix and re-run analysis; consider rolling back until fixed.- If model behavior issue found: rollback to previous model, retrain with production-distribution data, or apply guardrails (thresholding, blending).- Design follow-ups: incremental rollouts, stratified experiments, or multivariate tests to isolate features. Define success metrics and monitoring dashboards; set automated alerts for metric regressions.

Throughout, communicate status to stakeholders, keep experiments reproducible (versioned data, code, seeds), and prioritize user-facing rollback if user experience is harmed.

AI and Machine Learning Background Questions

Sample Answer

Sample Answer

Sample Answer

Sample Answer

Sample Answer

Unlock Full Question Bank