Spotify Junior AI Engineer Interview Preparation Guide

Direct answer
Both ReLU and Leaky ReLU, and their gradients, are simple elementwise comparisons that vectorize cleanly over an array of any shape, with the only real design decision being what gradient value to assign at exactly zero.

Structured elaboration

import numpy as np

def relu(x):
    x = np.asarray(x)
    return np.maximum(0, x)

def relu_grad(x):
    x = np.asarray(x)
    return (x > 0).astype(x.dtype)

def leaky_relu(x, alpha=0.01):
    x = np.asarray(x)
    return np.where(x > 0, x, alpha * x)

def leaky_relu_grad(x, alpha=0.01):
    x = np.asarray(x)
    return np.where(x > 0, np.array(1, dtype=x.dtype), np.array(alpha, dtype=x.dtype))

Both relu and leaky_relu use np.maximum/np.where, which broadcast over arrays of any shape with no explicit Python-level loop, so the same code handles a 1-D vector or a 4-D convolutional feature map identically. At exactly $x=0$, this implementation defines the ReLU gradient as 0 and the Leaky ReLU gradient as $\alpha$ (both consistent with treating $x=0$ as falling into the "non-positive" branch); this is one valid, defensible convention among a couple of equally reasonable choices, and what matters in practice is being explicit and consistent about it, since $x$ landing on EXACTLY zero is a measure-zero event for continuous inputs anyway.

Worked example
Run directly: for test_in = [-1.0, 0.0, 2.0], relu(test_in) returns [0.0, 0.0, 2.0] and relu_grad(test_in) returns [0.0, 0.0, 1.0], both matching the expected unit-test values exactly. For leaky_relu(test_in, alpha=0.01), the result is [-0.01, 0.0, 2.0] (the negative input is scaled by 0.01 rather than zeroed), and leaky_relu_grad(test_in, alpha=0.01) returns [0.01, 0.01, 1.0] (note $x=0$ falls into the non-positive branch here, giving $\alpha$ rather than 1). Both functions were also confirmed to preserve shape correctly on a random 3-dimensional (3,4,5) array, confirming genuine N-dimensional support, not just 1-D correctness.

Trade-offs & pitfalls
Time complexity is $O(n)$ for $n$ total elements (a single pass, fully vectorized), and space is $O(n)$ for the returned array (or $O(1)$ extra if you overwrite the input array in place instead, which these implementations deliberately do not do, to avoid surprising a caller who still needs the original values). A common bug when implementing this from scratch is using Python-level conditionals or explicit loops (if x[i] > 0: ...) instead of vectorized NumPy operations, which is both far slower and defeats the entire purpose of the exercise; a second common bug is forgetting to cast the boolean mask back to the input's dtype in relu_grad, which can silently produce a different (and sometimes incompatible) dtype than the caller expects when this gradient is later multiplied against an upstream gradient array.

Direct answer

Pick a project that genuinely needed more than one function, and be specific about two things: what YOU owned (not what 'the team' did), and the one concrete mechanism that determined whether the collaboration worked, such as a shared definition of done, a clear handoff point, or clarity on who decided what when opinions differed. Vague answers ('we communicated well') sound rehearsed; specific answers sound lived-in.

What the story needs to show

Your specific contribution. Interviewers are listening for what you personally decided or built, distinct from what your collaborators did. If every sentence is 'we', the interviewer cannot tell what you'd do differently on the next team.

A mechanism-level explanation. Organize the story around one of three lenses:

• Shared goal: did every function agree on what 'done' looked like and how success would be measured, or was each function quietly optimizing for its own definition?
• Interface or handoff: was there a clear point where work crossed from one function to another, and was that point actually defined, or did people guess?
• Decision rights: when functions disagreed, was it clear whose call it was, or did disagreement just stall until someone got tired of arguing?

Honesty if it's a struggle story. The question explicitly allows 'succeed or struggle'. A good struggle story ends on what you changed about the collaboration, not on who was at fault.

Worked example

Situation: [your team] needed to deliver [a feature or initiative] that required real work from [Team A, for example a design or research function] and [Team B, for example a data or infra function], against a fixed external date.
Task: your role was the one connecting the three groups, for example owning the shape of the interface between design and engineering, or owning how data requirements got translated into a schema.
Action: early on, each function had a different idea of what 'done' meant for their piece, which caused rework when the pieces met. You wrote a short one-page agreement naming the shared definition of done and who would sign off on each handoff, and used it to resolve the next two disagreements without a meeting.
Result: the project shipped on the revised date, and the agreement itself became something the group reused on the next cross-functional piece of work, which is the real marker of a story about redesigning the collaboration rather than just pushing through it.

To make that skeleton concrete rather than a fill-in-the-blank: picture a checkout redesign that needed real work from the design function and the payments engineering function, against a fixed external date tied to a promotional campaign launch. The specific disagreement was about what 'done' meant for the new payment-method selector: design considered the screen done once every state (loading, error, empty) matched the approved mockups pixel-for-pixel, while payments engineering considered it done once the integration correctly handled every payment-provider response code, even ones with no mockup drawn yet. That mismatch caused two rounds of rework when a payment-provider error state shipped without a design pass. The one-page agreement that resolved it included this line: 'A screen is done when it matches an approved mockup for every state the payments API can return, and any new state discovered after mockups are drawn triggers a joint 15-minute review before either side builds it.' That single sentence is what let the two functions stop re-litigating 'done' every time a new edge case appeared, and both sides signed off on it before the next round of work began.

Trade-offs and pitfalls

• A generic 'we all communicated well' answer with no mechanism is the single most common weak version of this story, avoid it.
• Over-crediting the team at the expense of your own specific contribution leaves the interviewer unable to evaluate you.
• If you pick a struggle story, resist framing it as the other function's fault. The senior version of this answer explains what you changed about how the groups worked together, not who dropped the ball.
• The strongest answers show you redesigning a structure (a handoff, a shared definition, a decision rule), not just working harder inside a broken one.

Requirements & goals:

• Restore trust after biased LLM output; equip PMs and legal with practical understanding of bias sources, mitigation levers, risk assessment, and operational guardrails.
• Audience: product managers (feature owners) + legal/compliance (policy & risk owners).
• Outcome: shared mental model, concrete mitigation playbook, templates and follow-up cadence.

One-day workshop agenda (8 hours)

• 09:00–09:20 — Welcome & objectives; incident brief (factual, non-blaming)
• 09:20–10:00 — Foundations: how LLMs work, where bias arises (data, pretraining, prompts, fine-tuning, decoding)
• 10:00–10:45 — Legal risks & regulatory landscape (privacy, discrimination, disclosure obligations)
• 10:45–11:00 — Break
• 11:00–12:30 — Hands-on lab 1: reproducible bias sources (see exercises)
• 12:30–13:30 — Lunch
• 13:30–14:15 — Mitigations: system-level controls (pre/post-filters, red-teaming, guardrails), product trade-offs
• 14:15–15:30 — Hands-on lab 2: build & evaluate mitigation chain
• 15:30–15:45 — Break
• 15:45–16:30 — Policy workshop: approval flow, labeling, user notices, escalation playbooks
• 16:30–17:00 — Roadmap & responsibilities: owner matrix, measurement plan, immediate action items
• 17:00–17:15 — Wrap-up & feedback

Pre-work for attendees (one week prior)

• 15-minute explainer video: LLM basics + short glossary
• Read: sanitized write-up of the incident (facts + outputs) and current product flow
• Questionnaire: role-specific concerns and three real use-cases they own
• Mini lab (optional): run a supplied notebook that queries a small model to see variability with prompts

Hands-on exercises (concrete)

1. Bias source demo (45–60 min)

• Provide a small open-source model + dataset. Tasks:
  • Show how changing prompt framing produces biased completions (e.g., occupational suggestions by gendered prompt).
  • Show training-data contamination: swap/omit protected-group examples and observe output shifts.
  • Use temperature/decoding changes to show hallucination-vs-bias tradeoffs.
• Deliverable: short incident-style write-up explaining which source(s) caused bias.

2. Mitigation chain build (75–90 min)

• Teams implement: input sanitizer, prompt templates with constraints, output classifier (bias detector), and a fallback response policy.
• Measure: run a benchmark of targeted prompts before/after and record false-positive/false-negative trade-offs.
• Deliverable: mitigation playbook card per product use-case.

Follow-up artifacts & sustained guardrails

• Bias Playbook: decision tree for triage, severity rubric, mitigation checklist, escalation matrix, required approvals.
• Templates: incident report, user-facing disclosure language, model-card and dataset-card templates.
• Automated tests: unit tests that run bias-detection heuristics on PRs and nightly regression suites.
• Governance: monthly red-team exercises, quarterly legal-product sync, mandatory sign-off for high-risk features, and SLAs for incident response.
• Metrics dashboard: bias-rate, false-positive/negative rates for detectors, user-reported harm, time-to-mitigation.
• Training: recorded workshop, short role-based micro-modules, and a certification checklist for PMs before shipping AI features.

Why this works

• Cross-functional: combines legal risk framing with technical demos so both speak the same language.
• Practical: hands-on exercises reveal causal links and trade-offs.
• Durable: artifacts + automation + governance ensure learning becomes operational practice.

Direct answer

CI checks catch problems BEFORE a model artifact exists or is deployed (code and data issues); CD gates catch problems AFTER a candidate is built but BEFORE it fully serves traffic (deployment and runtime issues): conflating the two, or skipping either, is what lets a bad model reach production.

Structured elaboration

CI (belongs in the code/data pipeline, runs on every commit or data update):

• Unit tests for feature-transformation code: pure code correctness, no data or model involved.
• Data-schema checks: does the new training data match the expected schema, before training even starts.
• Distributional tests: does the new training data's distribution look sane relative to historical norms (a basic sanity check, distinct from full drift monitoring which runs continuously in production).
• Reproducibility validation: can this exact pipeline configuration be re-run and produce a comparably consistent result, catching non-determinism bugs before they reach a real training run.

CD (belongs in the deployment pipeline, runs on a specific candidate model artifact):

• Canary health checks: infra-level (latency, error rate) AND model-quality-level (prediction distribution, confidence) checks on the canary's live traffic.
• Post-deploy shadow comparisons: the candidate's shadow-traffic predictions compared against the currently-serving model's, BEFORE the canary even starts taking real traffic.

Automated vs. manual gates: CI checks are almost always fully automated (they're fast, deterministic, and blocking a commit on a failed unit test is uncontroversial). CD gates are more often a MIX: canary health-check thresholds can be automated (a clear latency or error-rate breach automatically halts the rollout), but the final full-promotion decision after a successful canary often keeps a human approval step, at least until a specific model class has built enough of a track record to earn full automation: the same graduated-trust principle that applies to error-budget-triggered retraining.

Worked example

A concrete failure this taxonomy prevents: a model whose training code has a subtle bug (say, a leaked future feature) can still pass every CD gate perfectly: it serves fast, doesn't error, and even looks statistically fine in a canary, because the leak makes it LOOK unrealistically good, not obviously broken. Only a CI-stage check (a specific automated leakage-detection test on the training pipeline) catches this, since by the CD stage the bad model is already built and "working" by every deployment-level signal.

Trade-offs & pitfalls

The trap is treating CI and CD gates as redundant or interchangeable ("we'll catch it in canary if the training data was bad"): they genuinely check different failure classes, and a bug that only a CI-stage data-schema check would catch can sail straight through every CD gate looking completely healthy, precisely because CD gates are checking DEPLOYMENT health, not data or code correctness.

Approach: balance memory/compute costs vs. latency and ranking quality by applying a layered strategy: reduce table size (quantization/pruning/distillation), reduce access cost (caching/hot rows/asynchronous fetch), and right-size infra (autoscaling + async pipelines). Measure cost vs. quality/latency trade-offs with specific metrics.

Plan:

1. Quantization

• Use 8-bit or 4-bit quantization (e.g., QAT for minimal accuracy loss). Expect ~2–4x memory reduction at <1% NDCG loss for many embeddings.
• Validate with A/B: track delta NDCG@10 and recall@k.

2. Pruning & sparse/dense hybrid

• Identify low-usage or low-contribution dimensions via L1/L2 importance or sensitivity; remove or sparsify them.
• Target 10–30% dimension pruning with retrain/fine-tune to keep quality.

3. Model distillation

• Train a smaller embedding model (student) to mimic teacher. Reduce embedding size (e.g., 512→128) to cut storage and compute by ~4x; measure end-to-end ranking loss.

4. Caching hot rows

• LRU or LFU cache in RAM/SHM for top X% of lookups (e.g., top 1% rows produce 50–80% of hits). Store quantized but fast-to-dequantize representations.
• Metric: cache hit rate, p50/p95 fetch latency, reduced egress/storage ops.

5. Asynchronous fetch & batching

• Pre-fetch embeddings for predicted candidates; non-blocking background loads for cold rows; batch RPCs to storage (coalescing).
• Metric: tail latency reduction, queue lengths, miss penalty.

6. Autoscaling policies

• Scale based on combined signals: request QPS, cache miss rate, and CPU/GPU utilization. Use predictive autoscaling (time-series + recent traffic) to avoid cold starts.
• Policy example: maintain p95 latency < 50ms; scale out when miss rate > 5% and CPU>70%.

Trade-offs & validation:

• For each knob (quant bits, embedding size, cache size), run gated experiments and plot cost per QPS vs. NDCG loss and p95 latency.
• Expected targets: reduce storage/serving cost 2–6x with <1–2% relative ranking degradation and p95 latency increase <10ms.

Concrete metrics to monitor:

• Cost metrics: $/M requests, storage $/GB-month, compute $/hour
• Quality: NDCG@10, recall@k, model calibration drift
• Performance: p50/p95/p99 fetch & end-to-end latency, cache hit rate, cold-miss penalty
• Utilization/operational: CPU/GPU utilization, memory usage, RPC latency, queue lengths
• Autoscaling signals: scale events, time-to-scale, scale failures
• Business impact: CTR, conversion lift from A/B tests

Implementation cadence:

• Phase 1: measure baseline and top-k hot rows; add caching + async fetch.
• Phase 2: apply 8-bit quantization with QAT; monitor quality.
• Phase 3: pruning/distillation and predictive autoscaling.
• Run continuous A/B and rollback thresholds tied to NDCG/CTR and latency SLOs.

Direct answer. SIMD (Single Instruction, Multiple Data) speeds up compute kernels by executing one instruction across several data elements packed into one wide register, instead of issuing one instruction per element. For a 512-bit vector unit operating on single-precision (32-bit) floats, the register holds $512 / 32 = 16$ elements, so the purely lane-count-driven theoretical speedup for $1\times 10^9$ scalar operations is:

i.e. up to 16x fewer instructions for the same arithmetic work, if every operation is independent, perfectly aligned, and the instruction stream is the bottleneck.

Structured elaboration: why the real speedup falls short of 16x.

• Alignment. Vector loads/stores are fastest when the data starts at an address that is a multiple of the vector width in bytes (64 bytes for a 512-bit register); misaligned accesses can require an extra instruction or a slower code path.
• Data layout. SIMD wants contiguous, uniformly-strided data (structure-of-arrays); if the data is laid out as an array-of-structures with unrelated fields interleaved, the vector unit either sits idle waiting for a gather/scatter operation or cannot be used at all without a costly repacking step first.
• Memory bandwidth saturation (the memory bus running at its maximum sustainable
  transfer rate, so extra demand queues instead of moving faster). Like the scalar
  case, if the kernel moves more bytes per FLOP (floating-point operation) than the
  memory bus can sustain, 16 lanes computing in parallel just wait longer for the next load; vectorizing a memory-bound kernel mostly speeds up the compute portion that was never the bottleneck.
• Loop-carried dependencies. If iteration $i+1$ needs the result of iteration $i$ (e.g. a running product), the lanes cannot be filled with independent work and the vector unit degrades toward scalar-equivalent throughput regardless of its width.

Worked example. Tensor libraries allocate buffers aligned to 32- or 64-byte boundaries specifically so every row of a tensor starts on a vector-friendly address, avoiding the misalignment penalty above by construction. Separately, consider a numeric pipeline capped at 10 GB/s of achievable memory bandwidth, where the kernel moves roughly 4 bytes per FLOP-pair (one multiply and one add) processed. Carry that through rather than asserting the conclusion, and make sure both sides of the comparison are the same kind of quantity: a FLOP rate cannot be compared against the dimensionless 16x lane ratio, so convert the bandwidth into the FLOP rate it can sustain. The memory system feeds $10\times10^{9}/4 = 2.5\times10^{9}$ FLOP-pairs per second, which is $5$ GFLOP/s. A 512-bit unit clocked at 3 GHz with two fused-multiply-add ports peaks near $16 \times 2 \times 2 \times 3\times10^{9} = 192$ GFLOP/s (16 lanes, 2 FLOPs per fused multiply-add, 2 ports; substitute your own hardware's port count and clock). The memory ceiling therefore sits roughly 38x below the compute ceiling, which means the scalar kernel was already waiting on memory before any vectorization and the realized speedup from widening the registers is close to 1x, not 16x. Wider SIMD registers cannot lift throughput past what the 10 GB/s bus delivers; the fix in that regime is reducing bytes moved per FLOP (lower precision, better locality, prefetching) rather than a wider vector unit. The general form is worth carrying into an interview: with $C$ the scalar compute time and $M$ the memory time, vectorizing takes the runtime from $\max(C, M)$ to $\max(C/16, M)$, so the achievable speedup is $\max(C, M) / \max(C/16, M)$, which collapses to 1x as soon as $M \ge C$ and reaches the full 16x only when $M \le C/16$.

Trade-offs & pitfalls. Quoting a vector-width-based number like 16x as "the speedup" without naming the four limits above is a common interview mistake; a correct answer states the theoretical ceiling and then names which limiter (alignment, layout, bandwidth, or dependencies) is realistically expected to bind for the kernel in question, since in most production kernels one of these keeps realized speedup well under the register-width ceiling.

Direct answer

Estimating total GPU-hours to train a large transformer starts from the standard approximation that training FLOPs are roughly $6 \times \text{parameters} \times \text{tokens}$ (accounting for both forward and backward passes), then divides by the achievable FLOPS per GPU (peak FLOPS times realistic utilization, not peak alone) to get total GPU-seconds, converted to GPU-hours.

Structured elaboration

• Total training FLOPs: for P parameters and T training tokens, the widely-used approximation is $\text{FLOPs} \approx 6PT$, where the factor of 6 accounts for roughly 2 FLOPs per parameter per token for the forward pass and roughly 4 more for the backward pass (which needs to compute gradients with respect to both the activations and the parameters).
• Effective per-GPU throughput: peak advertised FLOPS for a given GPU (e.g. a stated bf16 TFLOPS figure) is a theoretical ceiling; realistic sustained utilization for a well-tuned large-model training run (model-FLOPs-utilization, MFU) is typically 30-50% of peak, not 100%, due to communication overhead, memory-bound operations, and pipeline bubbles, so effective per-GPU FLOPS = peak FLOPS x MFU.
• Total GPU-hours: total GPU-seconds = total FLOPs / effective per-GPU FLOPS, then GPU-hours = GPU-seconds / 3600; this is the total across all GPUs used (i.e. it's independent of how many GPUs you actually use in parallel, since GPU-hours is a measure of total compute consumed, not wall-clock time, which DOES depend on GPU count).

Worked example

For a 100B-parameter model trained on 300B tokens: $\text{FLOPs} = 6 \times 100\times10^9 \times 300\times10^9 = 1.8\times10^{23}$ FLOPs. Using an A100 with a stated bf16 peak of 312 TFLOPS ($3.12\times10^{14}$ FLOPS/sec) at an assumed 40% MFU: effective per-GPU FLOPS = $3.12\times10^{14} \times 0.4 = 1.248\times10^{14}$ FLOPS/sec. Total GPU-seconds = $\frac{1.8\times10^{23}}{1.248\times10^{14}} \approx 1.4423\times10^9$ seconds. Converting to GPU-hours: $\frac{1.4423\times10^9}{3600} \approx 4.006\times10^5$ GPU-hours, i.e. roughly 400,600 GPU-hours.

Trade-offs & pitfalls

The single largest source of estimation error in this kind of calculation is MFU: assuming 100% (peak) utilization when real large-model training typically achieves 30-50% overstates achievable throughput by 2-3x, correspondingly UNDERSTATING required GPU-hours (and therefore cost and time) by a similar factor; always state the assumed MFU explicitly as part of any such estimate, and validate it against measured throughput from an actual small-scale test run on the target hardware/model combination rather than assuming a textbook figure applies unchanged.

Direct answer

A feature-store validation system for online inference needs to guarantee bounded freshness (features are not stale beyond an acceptable window) and online/offline parity (the value served at inference time matches what training saw for the same feature and same logical point in time), with explicit Service-Level Indicators for both, and a cost-aware strategy that combines exact validation on a small set of highest-priority features with cheaper probabilistic sketches for the long tail.

Structured elaboration

• Freshness SLI: the latency between when the underlying event that a feature depends on occurred and when that feature's updated value is available to the online store, measured as a percentile (p95, p99) rather than an average, since a small tail of very stale features can matter more than the typical case for a model that depends on recency.
• Online/offline parity SLI: the rate at which a feature's online-served value matches what the offline (training) pipeline would have computed for the same entity and timestamp; a parity gap here is one of the most damaging and hardest-to-detect classes of ML data-quality bug, since it produces no error, just a model quietly seeing different data than it was trained on.
• Validating thousands of features under a strict compute budget: rather than exact validation on every feature continuously, which is prohibitively expensive at that scale, run exact validation continuously on a small, deliberately-chosen set of highest-business-impact features, and use probabilistic sketches (compact summaries that approximate distributional statistics with bounded error) for the remaining long tail, escalating a feature from sketch-based to exact validation if the sketch itself signals something anomalous.

Worked example

A feature store serving 2,000 features for a real-time recommendation model designates the 50 features with the largest measured impact on model output for continuous, exact freshness and parity validation, while the remaining 1,950 lower-impact features are validated via lightweight statistical sketches computed hourly, with any sketch showing an anomalous shift automatically triggering a temporary escalation to full, exact validation for that specific feature until the anomaly is explained or resolved.

Trade-offs and pitfalls

The risk of the tiered approach is that "low business impact today" is not a permanently fixed property, a feature currently deemed low-priority can become critical after a model retraining that weights it more heavily, and if the priority tiering is not periodically re-evaluated against the current model's actual feature-importance profile, the validation system's coverage silently drifts out of alignment with what the model actually depends on most.

Approach & notation

Given a 3×3 confusion matrix C where C[i][j] = true class i predicted as j (rows = true, cols = pred). For class k (k=1..3):

• True Positives (TP_k) = C[k][k]
• False Positives (FP_k) = sum over i != k of C[i][k]
• False Negatives (FN_k) = sum over j != k of C[k][j]

Per-class formulas

Precision_k:

precision_k = TP_k / (TP_k + FP_k)

Recall_k:

recall_k = TP_k / (TP_k + FN_k)

F1_k (per-class):

f1_k = 2 * precision_k * recall_k / (precision_k + recall_k)

Macro vs Micro F1

• Macro-F1: average of per-class F1s

macro_f1 = (f1_1 + f1_2 + f1_3) / 3

• Micro-F1: compute global TP, FP, FN (sum over classes) then F1 from aggregated precision/recall

micro_precision = sum_k TP_k / (sum_k TP_k + sum_k FP_k)
micro_recall    = sum_k TP_k / (sum_k TP_k + sum_k FN_k)
micro_f1 = 2 * micro_precision * micro_recall / (micro_precision + micro_recall)

Worked example (skewed customer-support intents)

Take a simplified 3-intent slice of a support classifier (billing_question, cancel_subscription, technical_issue) where billing_question dominates traffic, the kind of skew a real 10-intent classifier shows:

Confusion matrix C (rows = true, cols = predicted):
                pred_billing  pred_cancel  pred_technical   row total
true_billing         760           25            15            800
true_cancel            8           10             2             20
true_technical         5            2             8             15
column total         773           37            25            835

Per-class precision/recall/F1:

billing:    TP=760, FP=13, FN=40  -> precision=760/773=0.983, recall=760/800=0.950, f1=0.966
cancel:     TP=10,  FP=27, FN=10  -> precision=10/37=0.270,  recall=10/20=0.500,  f1=0.351
technical:  TP=8,   FP=17, FN=7   -> precision=8/25=0.320,   recall=8/15=0.533,   f1=0.400

macro_f1 = (0.966 + 0.351 + 0.400) / 3 = 0.572

micro: sum TP=778, sum FP=57, sum FN=57, so micro_precision = micro_recall = 778/835 = 0.932, micro_f1 = 0.932

The two averages diverge by 0.36: micro-F1 (0.932) is dragged almost entirely by the large billing_question class the model already handles well, while macro-F1 (0.572) exposes that the model is mediocre on the two rare, business-important intents (cancel_subscription and technical_issue). For a stakeholder report I would present macro-F1 here, since a headline 0.93 would hide that the model is failing on the rare intents a support team most needs correctly routed, and I would name the two weak per-class F1 scores explicitly rather than only the macro average.

When prefer Macro-F1

Use macro-F1 when class balance matters and you want equal weight per class: e.g., detection of rare but critical classes (fraud, disease). Macro-F1 penalizes poor performance on minority classes; micro-F1 can be dominated by large classes and hide failures on rare but important classes.

Direct answer

The policy should default teams toward external large language model (LLM) APIs for anything non-differentiating and low-sensitivity, and require an explicit exception process (with named criteria) for hosting in-house, since in-house LLM hosting carries a cost and operational burden that's rarely justified until a team hits a specific, provable limit of the API path.

Structured elaboration

• Decision criteria for defaulting to an external API: acceptable latency for the use case, no requirement to fine-tune on sensitive proprietary data that can't leave your infrastructure, and a cost profile where usage-based pricing is cheaper than the fixed cost of hosting at your expected volume.
• Criteria that justify an exception (in-house hosting): a hard data-residency or privacy requirement that rules out sending data to a third party, a customization need beyond what the API's fine-tuning options support, or a security requirement (a fully air-gapped environment) the vendor can't meet.
• Governance. All API usage should go through a small number of approved, centrally-procured accounts (never ad hoc credit-card signups), with access control scoped per team and a review of what data each integration sends externally.
• Monitoring. Track spend per team against budget, latency and error rates from the vendor, and a recurring review of whether any team's usage has grown into "exception" territory (heavy customization, sensitive data) that should trigger a re-evaluation.

Worked example

A support-ticket summarization feature has no sensitive customer PII in its inputs after redaction, tolerates a few hundred milliseconds of latency, and needs no fine-tuning: this defaults cleanly to an external API. A clinical-notes assistant handling protected health information, by contrast, hits the data-residency exception immediately, and the policy routes it to an in-house or private-hosting evaluation from day one rather than starting on the general-purpose API and having to migrate later under regulatory pressure.

Trade-offs and pitfalls

The risk of a too-permissive default is teams sending sensitive data through unreviewed API integrations before anyone notices; the risk of a too-restrictive default is teams building unnecessary in-house infrastructure for use cases that never needed it, burning engineering time that could have shipped features. The policy earns its keep by making the exception criteria concrete enough that teams can self-assess, rather than needing a slow central review for every single integration.