Company Technology and Strategic Direction Questions

Situation: My org needed to move dozens of production ML services from a heterogeneous legacy platform to a new Meta-wide ML platform. Risks: downtime, model regressions, lost productivity, and political resistance across infra, data, research, and product teams.

Task: As lead AI Engineer, I had to define and run a migration strategy that minimized risk, aligned stakeholders, and drove adoption within 9 months.

Action:

• Stakeholder alignment: Convened a steering group (Infra, Data, ML Platform, Research, Product, Security, Legal). Mapped pain points, success criteria, and constraints. Established RACI and bi-weekly governance reviews.
• Pilot selection: Chose 3 pilots representing orthogonal risk vectors: a low-latency CV inference pipeline, a large-batch training workload, and an online recommendation model with strict privacy needs. Criteria: business impact, team bandwidth, technical representativeness, rollback feasibility.
• Phased migration: Phases: discovery (inventory, dependency graph, SLAs), pilot (validation + metrics parity), scale (bulk migrations in waves grouped by similarity), cutover (final traffic shifts), and optimization (cost/perf tuning).
• Data & model migration: Implemented reproducible artifacts (containerized training recipes, model cards, versioned datasets), schema adapters, and data validation (TFT/Great Expectations). Used model export standards (ONNX/TorchScript) plus a translation layer for platform-specific runtimes.
• Backward compatibility: Built a proxy layer to route requests to legacy or new runtime; A/B and canary traffic splits with shadow testing to compare outputs without user impact. Maintained API contracts and gradual deprecation timelines.
• Training & enablement: Created role-based training (hands-on workshops, migration playbooks, migration CLI), office hours, and a migration “runbook” with templates. Paired each migrating team with a Platform engineer for the first two migrations.
• Incentives: Introduced migration SLO credits (reduced infra billing for early adopters), recognition in quarterly reviews, and a “migration sprint” fund for engineering time.
• KPIs: Tracked parity metrics (accuracy, latency, throughput), operational metrics (MTTR, deployment frequency), adoption (number of migrated models, percent of traffic), business impact (revenue lift or cost reduction), and developer experience (time-to-deploy, satisfaction scores). Set targets and automatic alerts for regressions.

Result: Pilots achieved metric parity within two sprints; canary tests caught one compatibility bug before user impact. Within 6 months, 70% of target services migrated, latency improved 15% on average, infra cost per training job dropped 18%, and developer satisfaction rose from 6.4 to 8.1. The structured governance, shadow testing, and pairing model were key learnings—technical migration succeeds only with clear incentives, strong support, and measurable safety nets.

Requirements & constraints:

• Real-time product surfaces (chat, comments, image uploads) plus async review. Latency SLAs differ: inline blocking <200ms, flagging async allowed minutes–hours. Safety bar: minimize high-risk false negatives; business/UX: minimize false positives.

High-level architecture:

1. Input router: surface-specific preprocessor (text normalization, OCR, feature extraction).
2. Ensemble of specialized models:
   • Fast lightweight classifiers (distil-BERT / MobileNet) for low-latency screening.
   • Stronger contextual models (RoBERTa/Large, multimodal transformers) for deeper judgment.
   • Heuristic/rule-based filters for policy hard rules (e.g., child sexual content).
3. Cascading pipeline with thresholds:
   • Stage A (fast): if score_high >= TA -> immediate block/label; if score_low <= TB -> allow; if ambiguous between TB and TA -> escalate to Stage B.
   • Stage B (expensive): ensemble voting + calibrated probability + uncertainty estimation (MC-dropout or temperature scaling). Compute risk score = weighted sum(policy severity, model confidence, user history, content virality).
   • If risk score > T_block -> block; if between T_human and T_block -> send to human-in-the-loop queue; else allow with soft actions (warning, reduced distribution).
4. Human-in-the-loop (HITL):
   • Prioritized queues sorted by risk score, recency, influencer reach.
   • UI captures annotator decision, rationale tags, and edge-case flags.
   • SLA tiers: <5 min for high-risk, <24h for moderate.
5. Feedback loop & retraining:
   • Log model inputs, predictions, human labels, downstream outcomes (appeals, reinstatements).
   • Active learning sampling: prioritize examples with high model uncertainty, model disagreement, and high product impact for labeling.
   • Regular retrain cadence: continuous small-batch updates for calibration + periodic full retrain monthly with drift detection.
6. Reducing FP/FN:
   • Calibrate thresholds by cost-sensitive optimization (weight FN higher for safety categories).
   • Use ensemble diversity (architectures, pretraining data) to reduce correlated errors.
   • Implement rejection/abstain option where model uncertainty is high to defer to humans.
   • Post-process using contextual signals: user history, temporal patterns, provenance, metadata.
   • A/B test threshold adjustments and monitor business/UX metrics and safety KPIs.
7. Scaling strategies:
   • Autoscale fast models on GPUs/TPUs; cache repeated predictions; batching for expensive models.
   • Move heavy inference to async worker for non-blocking surfaces.
   • Use feature-store and RL/online learning only where safe and validated.
8. Monitoring & governance:
   • Real-time dashboards: precision/recall by category, latency, queue backlog, appeals rate, demographic fairness metrics.
   • Drift alerts, PSI for input distribution, and kill-switch to roll back thresholds/models.
   • Audit logs, explainability snippets for humans and appeals.

Why this works:

• Cascading saves compute while ensuring expensive models and humans handle uncertainty.
• Risk scoring combines model confidence with contextual impact to prioritize human effort.
• Active learning and calibrated ensembles reduce long-term FP/FN while scaling safely through automation and strong monitoring.

Requirements & constraints:

• Attribute GPU, storage, network, and persistent disk costs to teams, projects, models; support multi-cloud + on-prem GPUs; near real-time telemetry; enforceable policies; scale to hundreds of teams.

High-level architecture:

• Instrumentation → Cost-aggregation → Analysis & Recommendations → Policy Engine → UI/Alerts
• Components: Telemetry collectors (agents, cloud billing exports), Cost attribution service, Rightsizing & spot recommender (ML service), Storage lifecycle manager, Policy enforcement layer (IAM + Orchestration hooks), Dashboard + Alerting.

Telemetry & attribution:

• Ingest cloud billing (AWS Cost & Usage, GCP Billing) + Kubernetes/GPU exporter (nvidia-smi + DCGM metrics) + model-serving logs.
• Normalize to canonical resources (gpu-hour, GiB-month, GB-egress).
• Tag propagation: enforce standardized tags (team, project, model, env). Where tags missing, use workload fingerprinting: container image + owner mapping + commit metadata + scheduler labels — attribute probabilistically with confidence scores.
• Store in time-series DB (Prometheus / Influx) for metrics and data warehouse (BigQuery/S3 + Parquet) for cost accounting.

Automated rightsizing:

• Aggregate utilization (GPU utilization, memory, CPU, I/O) per job over windows.
• Train rules + ML model to suggest smaller instance types or batch sizing (min viable GPU, mixed-precision, gradient-accumulation). Provide candidate configs with expected cost and training time delta, and confidence.
• Example: Job using 40% GPU utilization for sustained periods → recommend moving from V100 to T4 or split into multi-job with smaller batch.

Spot vs on-demand recommendations:

• Use historical spot interruption rates, job tolerance (checkpointing frequency, remaining runtime), and SLA to compute expected cost and risk. Score: expected cost saving vs probability-weighted interruption cost.
• Recommend spot for checkpointed, restartable jobs; on-demand for long, non-checkpointed critical training.

Warm vs cold model storage:

• Tiering policy: warm (SSD / hot object store) for models with access frequency > X/day or used by production endpoints; cold (S3-IA / Glacier) for >30 days no accesses.
• Automate based on access traces and model importance tags; allow pre-warm windows before scheduled runs.

Alerting & runaway spend:

• Real-time anomaly detection on spend velocity per team (CUSUM + seasonal decomposition) + threshold alerts (absolute and % over baseline).
• Automated actions for high-confidence runaways: throttle new job submissions, suspend non-critical experiments, notify owners and approvers via Slack/email with one-click remediation.

Policy enforcement:

• Central policy engine integrated with Kubernetes admission controllers, cloud IAM, and orchestration (Airflow, Kubeflow). Policies: max GPU quota per team, allowed instance types (spot-eligible), required checkpointing for spot jobs, mandatory tags.
• Enforcement modes: dry-run (audit), warn, block. Maintain audit trail and escalation workflow.

Metrics & UX:

• Key metrics: cost by team/model, $/GPU-hour, utilization%, % of spend on spot, model storage cost, number of runaways.
• Provide interactive dashboard with “Accept recommendation” flow (automated rollout via PR to infra-as-code), simulated savings, and A/B pilot.

Operational considerations & trade-offs:

• Accuracy of attribution depends on tagging; invest in developer workflows to auto-tag.
• Conservative recommender initially (human-in-loop) to build trust; gradually enable automated enforcement.
• Balance risk: aggressive spot usage saves cost but adds interruption complexity; mitigate with robust checkpointing and shorter job shards.

This framework reduces costs while preserving agility: telemetry for visibility, ML-driven recommendations for rightsizing and spot decisions, lifecycle policies for storage, and enforcement to prevent runaway spend.

Requirements & constraints:

• Functional: generate concise, engaging vacation captions for a user’s Photos library.
• Non-functional: low latency for interactive UX, privacy-first (sensitive/personal data), offline capability, scalable cloud fallback, strong safety (no hallucinations, no personal data leakage), user control and transparency.

High-level architecture:

• Client (iPhone): lightweight caption generation pipeline using a distilled multimodal on-device model (quantized Transformer + vision encoder) for common cases; local metadata extractor (EXIF, location, timestamps), privacy classifier, and filter chain.
• Cloud: full-capacity generative model (larger multimodal LLM + vision model) accessed when: user opts in, on-device confidence low, heavy aggregation across many photos needed, or resource constraints.
• Decision service (on-device policy + optional cloud policy): inputs — on-device confidence score, privacy risk score, network state, user preferences — outputs: "serve on-device", "send minimal features to cloud", or "send image". Use feature-hash or embeddings for sending minimal data when possible.
• Safety stack:
  • Personal-data detector: face recognition opt-in, PII detectors (names, addresses, license plates) run locally; if PII detected, either redact or keep strictly on-device.
  • Hallucination guard: calibration of model confidence, grounding step that verifies generated facts against image-extracted metadata (location/geotags, timestamps, recognized landmarks) or returns a generic phrasing when unsupported.
  • Toxicity & content filter: local lightweight filters + cloud-side classifiers for deeper checks; human-in-the-loop reporting and rapid rollback.
  • Differential data handling: only upload embeddings or masked thumbnails unless explicit consent; ephemeral keys; end-to-end encryption.
• User controls & UX:
  • Clear opt-in/out for cloud generation, per-album privacy settings.
  • "Explain caption" toggle showing which evidence (face/landmark/metadata) supports each phrase.
  • Edit, regenerate, or disable generative captions; feedback button to improve models.
• Observability & offline MLOps:
  • Telemetry limited to opt-in aggregated metrics and lossless privacy-preserving telemetry (differential privacy, k-anonymity).

Evaluation metrics:

• Quality: BLEU/ROUGE for constrained tests, but primarily human-rated relevance, fluency, and usefulness; caption diversity and succinctness.
• Grounding & factuality: hallucination rate (fraction of claims not supported by image/metadata), verification recall (supported claims detected).
• Privacy & safety: PII leakage rate (measure via red-team tests), toxic content incidence, user opt-out rate.
• UX: latency (p95), success rate of on-device serving, battery/CPU impact, user satisfaction (NPS), edit/regenerate frequency.
• Operational: cloud cost per caption, failed generation rate, and time-to-mitigate flagged issues.

Trade-offs and rationale:

• Favor on-device by default to minimize privacy risk and latency; cloud used selectively to improve quality when user consents or when on-device confidence low.
• Grounding verification reduces hallucinations at expense of conservative output; present explainability to retain user trust.

Implementation notes:

• Use model distillation, quantization, CoreML/accelerator tooling for on-device models; secure APIs and federated learning or differentially private aggregation for continuous improvement.

Centralized analytics (core platform team):

• Excels when: company-wide standards, reusable data models, and strong governance are needed. Benefits: consistent metrics, economies of scale, shared tooling, and security/compliance.
• Fails when: product teams need rapid experimentation or domain-specific context; central bottlenecks slow delivery.
  Decentralized analytics (embedded in product teams):
• Excels when: fast iterations, deep product knowledge, localized KPIs and ownership are required. Teams can iterate on features and metrics rapidly.
• Fails when: leads to metric divergence, duplicated infra, and inconsistent governance; harder to enforce privacy and cost controls.
  Hybrid model recommendation: central platform provides core data pipelines, shared metrics, and tooling (catalogs, access controls), while product teams own domain-specific derived metrics and experiments. Use federated governance, a metrics registry, and SLAs for the platform to balance speed and consistency.