MediumTechnical

26 practiced

You inherit an ML environment where many scientists spin up large experiments without cost controls, causing a sudden infrastructure bill spike. What immediate triage steps would you take to stabilize cost, and what policies or systems would you put in place to prevent recurrence while preserving researcher productivity?

EasyTechnical

22 practiced

Define 'technical debt' specifically for AI systems. List common sources (code, data, experiments, infra), explain why some AI debts are more expensive than others, and propose a lightweight triage rubric you would use to prioritize which debts to fix first.

Sample Answer

Technical debt for AI systems: the accumulated shortcuts, design choices, and neglected maintenance across models, data, experiments, and infrastructure that make the system harder to operate, less reliable, biased, or costly to change over time. It causes hidden runtime costs, degraded model quality, and increased risk.

Common sources- Code: brittle training/eval scripts, duplicated preprocessing, tangled model-serving logic, lack of tests.- Data: label errors, schema drift, undocumented sampling, feature leakage, stale or unbalanced datasets.- Experiments: unreproducible runs, missing hyperparameter provenance, ad-hoc forks, unclear baselines.- Infrastructure: manual pipelines, poor CI/CD for models, insufficient monitoring, tight coupling to hardware or cloud services.

Why some debts cost more- Downstream impact: data issues or label bias can silently degrade many models and require costly relabeling — higher cost than refactoring a single function.- Frequency & reach: debts in serving paths or shared data pipelines affect production traffic continuously.- Detection latency: debts that are hard to detect (silent accuracy drift, fairness problems) accumulate longer and are pricier to fix.- Cross-cutting complexity: infra or experiment reproducibility problems block many engineers and slow iteration.

Lightweight triage rubric (score 1–5; prioritize highest total)1. Impact on user/business (1 low — 5 critical): How much does it affect correctness, revenue, safety?2. Blast radius (1 small — 5 system-wide): How many models/pipelines/users are affected?3. Detectability/time-since-introduced (1 easy/short — 5 silent/long): Is it already causing hidden failures?4. Cost-to-fix (1 cheap — 5 very expensive): Engineering effort, relabeling, infra changes.5. Risk of regression (1 low — 5 high): Likelihood that fixing will cause other failures.

Prioritization: Fix items with high Impact + Blast radius + Detectability and relatively moderate Cost-to-fix first. For very high-cost but very high-impact debts (e.g., systemic label bias), schedule as a prioritized project with stakeholder buy-in and measurement plan. Include quick wins (low cost, medium impact) to reduce immediate risk and free capacity.

MediumTechnical

23 practiced

Compare building an internal model serving platform versus using a managed cloud solution for a mid-size SaaS company handling sensitive PII. Discuss tradeoffs across cost, latency, security/compliance, team skills required, time-to-market, and vendor lock-in, and give a recommendation with rationale.

Sample Answer

Requirements/context: mid-size SaaS company processing sensitive PII; must balance cost, latency, security/compliance, team skills, time-to-market, and vendor lock-in.

Comparison (tradeoffs):

- Cost - Managed cloud: predictable OPEX, pay-for-use; can be expensive at scale (inference hours, storage, egress). - Internal: higher upfront CAPEX (HW, infra), lower marginal cost per inference long-term if highly utilized.

- Latency - Managed cloud: regional endpoints can be low-latency but cross-region egress or multi-hop adds delay. - Internal: colocated GPU servers or on-prem reduces latency and gives tighter control.

- Security & Compliance - Managed cloud: strong baseline controls, compliance certifications (SOC2, HIPAA BAA options) but you must verify data residency and contract terms for PII. - Internal: maximum control over data flows and encryption, easier to meet strict residency/inspection demands—but requires rigorous internal security ops.

- Team skills required - Managed cloud: fewer infra specialists; still need ML ops and secure integration skills. - Internal: needs SRE/GPU ops, hardware procurement, capacity planning, security engineers—higher hiring/training burden.

- Time-to-market - Managed cloud: fastest—prebuilt endpoints, autoscaling, monitoring. - Internal: longer—procurement, deployment, hardening.

- Vendor lock-in - Managed cloud: risk increases with proprietary APIs and model formats; mitigated by using standard containers, ONNX, or hybrid designs. - Internal: low vendor lock-in but higher operational burden.

Recommendation:For a mid-size SaaS handling sensitive PII, start with a managed cloud solution that supports strict compliance (choose a provider with HIPAA/SOC2 and region-specific data residency, and sign necessary BAAs). Use managed for rapid time-to-market, auditability, and reduced operational risk. Architect to minimize lock-in: keep models in open formats, containerize inference (Docker/KServe), and build an abstraction layer for routing so you can later migrate or run a hybrid model. Plan a phased path: benchmark costs and latency; if steady-state volume and latency/security needs justify it, evaluate a private on-prem/GPU cluster in year 2–3. This balances speed, security, and long-term cost control.

MediumTechnical

21 practiced

Design a monitoring and alerting strategy for a production LLM-based API that must maintain 99.95% availability and detect model drift, increasing hallucination rate, latency spikes, and cost anomalies at hourly granularity. Specify SLOs, metrics, sampling strategy, and automated remediation steps.

Sample Answer

Requirements & SLOs- Availability SLO: 99.95% (<= 4.38 minutes downtime/month). Error budget: 0.05% per month.- Latency SLO: p95 ≤ 300ms, p99 ≤ 800ms (hourly evaluation).- Quality SLOs (hourly): hallucination rate ≤ 0.5% (detected), model-drift score ≤ threshold (see below).- Cost SLO: cost-per-1k-requests under target; hourly anomaly budget defined.

Key metrics (hourly, with alerts on hourly window)- Infra/Service: success_rate, error_rate (4xx/5xx), availability, request_rate (RPS).- Latency: p50/p95/p99, tail latency spikes.- Resource: GPU/CPU utilization, memory, queue length.- Cost: cost_per_hour, cost_per_request, reserved vs on-demand spend.- Model quality: hallucination_rate (auto-checks), factuality_score, BLEU/ROUGE on synthetic probes, semantic-drift distance (embedding drift KL/CCD), confidence calibration, token_repetition_rate.- Business: key-customer failure rate, SLA-violations.

Sampling strategy- Synthetic probes: hourly scheduled queries (canaries) covering core intents, known facts, adversarial prompts.- Live-sampling: stratified random sampling across customers/endpoints; min 1% of requests or 1k requests/hour whichever larger; prioritize high-value tenants.- Ground-truth verification: for sampled responses run fast automated checks: knowledge-base lookup, fact-checking via retrieval-augmented verification, entailment classifier, and embedding distance to training centroids.- Drift windows: compute embedding drift and output-distribution drift hourly using sampled set.

Detection logic & thresholds- Hallucination detection: ensemble of (1) retrieval-match score < T1, (2) entailment model flags contradiction/probability < T2, (3) low confidence + high novelty. Alert if hallucination_rate > 0.5% (warning) or >1.5% (critical) in hour.- Model drift: embedding centroid shift or distributional KL > historical baseline + 3σ => warning; >5σ => critical.- Latency spikes: p99 > SLA threshold or sustained p95 increases >2x baseline.- Cost anomalies: rolling-hour cost > baseline + 4σ or sudden increase >30% hour-over-hour.

Alerting & routing- Two-tier alerts: warning (paged to dashboard + Slack) and critical (PagerDuty on-call).- Attach context: hour window, sample IDs, example requests/responses, diffs, infra metrics.

Automated remediation (playbook)1. Auto-scale: if latency/queue/backpressure, scale replicas/GPU pool and increase autoscaling limits. If scaling fails, trigger circuit-breaker.2. Degrade gracefully: route new traffic to smaller/faster model variant or cached responses (fallback model) preserving availability.3. Throttle/rate-limit: apply per-tenant throttles to protect shared capacity.4. Rollback: if drift/hallucination spike correlates with model rollout, auto-switch traffic to previous stable model (weighted rollback).5. Quarantine & shadow-test: divert suspect requests to a quarantine lane for increased sampling and human review.6. Cost controls: temporarily disable non-essential workloads (batch fine-tuning, heavy logging), enforce spend caps and notify finance.7. Retrain trigger: if drift persists beyond X hours (e.g., 6 hours critical), create automated dataset snapshot + alert MLOps to start fine-tune pipeline; mark candidate samples for labeling.8. Post-mortem automation: log events to incident system, consume error budget, auto-create ticket with diagnostics.

Instrumentation & tooling- Metrics: Prometheus + Grafana for infra, OpenTelemetry for tracing.- Feature/embedding store: record embeddings & metadata (time, tenant, prompt hash).- ML checks: lightweight fact-checker, entailment model, and drift library (e.g., River, NannyML) running hourly.- Observability: correlate traces (request→model→inference) and attach sample payloads in secure storage.- Audit & privacy: redact sensitive data; store hashes for repro.

Operational rules & SLAs- Error budget policy: if error budget exhausted, prioritize availability—reduce non-critical features, increase throttling, pause experiments.- Human-in-loop: automatic alert escalation to ML engineer + data annotators for critical hallucination/drift incidents.- Runbooks: maintain documented runbooks for each alert type with exact thresholds, checks, and remediation commands.

Why this works- Mix of synthetic probes + stratified live sampling gives coverage and reduces blind spots.- Automated fast checks detect likely hallucinations and drift at hourly granularity.- Automated remediation preserves availability (99.95%) while providing controlled paths for quality recovery and cost containment.

HardTechnical

38 practiced

Discuss the tradeoffs between model interpretability and raw predictive performance in high-stakes domains (finance, healthcare). Propose a decision framework for when to prefer interpretable models, and list compensating controls (human review, monitoring, thresholds) you would require when using black-box models.

Sample Answer

Interpretability vs predictive performance is a tradeoff between transparency (why a model made a decision) and optimized accuracy from complex models (ensembles, deep nets). In high‑stakes domains that affect rights, safety, or money, interpretability supports auditing, error diagnosis, regulatory compliance, and trust; black‑box models can improve raw metrics but increase risks: hidden biases, brittle failure modes, and harder remediation.

Decision framework (stepwise):1. Classify risk: map use-case to impact (safety/financial/regulatory). High impact → strong preference for interpretable models.2. Required actionability: if model outputs must be defensible to customers/regulators or directly drive irreversible actions, prefer interpretable models.3. Performance gap analysis: quantify delta between simple interpretable model (logistic regression, decision tree, GAM) and best black box. If gap small (< business threshold) choose interpretable.4. Mitigation cost/benefit: if black box yields large gains, evaluate compensating controls (cost, latency, governance).5. Hybrid approach: use interpretable front‑end for gating and black‑box for ranking where appropriate; use model distillation to produce simpler surrogates.

Compensating controls for black boxes:- Human‑in‑the‑loop reviews for high‑risk decisions and overrides.- Clear decision thresholds and conservative defaults; require escalation on borderline scores.- Rigorous pre‑deployment testing: adversarial, subgroup fairness, and stress tests; holdout tests mirroring production distribution.- Continuous monitoring: concept drift, calibration, disparity metrics, input distribution alerts.- Explainability tools (SHAP, LIME, counterfactuals) for case-level rationales and audit logs.- Versioning, reproducible training pipelines, and documented data lineage.- Red team audits and external regulatory reviews.- Recovery and rollback procedures plus periodic re‑training triggers.Use this framework to balance practical gains with ethical, legal, and safety obligations.

Strategic Technical Leadership Questions

Sample Answer

Sample Answer

Sample Answer

Sample Answer

Sample Answer

Unlock Full Question Bank