Meta AI Engineer Entry-Level Interview Preparation Guide

Direct answer: Big-O gives an asymptotic upper bound (the algorithm never does worse than this), Big-Omega gives an asymptotic lower bound (it never does better), and Big-Theta gives a tight bound (both at once, up to constant factors). Most everyday usage of "O(n)" is sloppy shorthand for Theta(n) - people usually mean the tight bound even when they write O.

Structured elaboration

Formally, for functions of $n$:

The intuition: O is "at most this fast-growing", Omega is "at least this fast-growing", Theta is "grows at exactly this rate" (sandwiched between two constant multiples of $g(n)$).

A common confusion: Big-O does not mean "this is the worst case" - it's a growth-rate bound that can describe best-case, average-case, or worst-case behavior depending on which function you plug in as $f(n)$. "Worst-case" and "O()" are independent axes; you can (and often should) say "the worst-case time is $\Theta(n^2)$."

Worked example

Let $f(n) = 3n^2 + 5n$ and $g(n) = n^2$.

• $f(n) = O(n^2)$: pick $c = 8$, $n_0 = 1$. For $n \ge 1$, $3n^2 + 5n \le 3n^2 + 5n^2 = 8n^2$. Holds.
• $f(n) = \Omega(n^2)$: pick $c = 3$, $n_0 = 1$. For $n \ge 1$, $3n^2 + 5n \ge 3n^2$. Holds.
• Since both hold, $f(n) = \Theta(n^2)$.

Now the O-but-not-Theta example the question asks for: let $f(n) = n$ and $g(n) = n^2$. Then $f(n) = O(n^2)$ (pick $c=1, n_0=1$: $n \le n^2$ for $n \ge 1$), but $f(n) \ne \Theta(n^2)$, because there is no $c > 0$ with $n \ge c \cdot n^2$ for all large $n$ (the ratio $n / n^2 = 1/n \to 0$, so no positive constant lower-bounds it). $f$ is O(g) but grows strictly slower, so it is not Omega(g), hence not Theta(g).

Trade-offs & pitfalls

• Interviewers usually accept "O(n)" when you mean the tight bound - but if asked to be precise (as here), know the distinction and use Theta when you mean it.
• A frequent mistake: quoting O() for an average-case argument as if it were a worst-case guarantee (e.g. calling hash-table lookup "O(1)" without qualifying "average case, assuming a good hash function").
• Omega is the least commonly used in casual conversation but matters when you need to argue a lower bound is unavoidable (e.g. proving comparison sorts need $\Omega(n \log n)$ comparisons).

Direct answer
The riskiest boundary in padding/truncation logic sits exactly where special tokens meet the length budget: if an implementation truncates content to max_seq_len and then adds [CLS]/[SEP] on top, every sequence quietly overshoots the model's real input limit by 2 tokens. The four required test lengths (0, 127, 128, 129) look like they probe "at, just under, just over max_seq_len," but the actual content budget is max_seq_len - 2 = 126, so all of 127, 128, and 129 exceed it, and a correct implementation must truncate content identically (to 126 tokens) for all three while a buggy one will not.

Structured elaboration
Using a BERT-style scheme with a [CLS] token at the start and a [SEP] token at the end: content_budget = max_seq_len - 2. The correct sequence is built as [CLS] + content[:content_budget] + [SEP], then padded with a pad token up to max_seq_len if shorter. The attention mask must be 1 for every real token including [CLS] and [SEP] (not just the content tokens), and 0 for every padding position.

The off-by-one risk is specifically in how content_budget is derived: a naive implementation that checks if len(tokens) <= max_seq_len and only then decides whether to truncate, without ever subtracting the 2 reserved slots, will treat an input of length 127 as "fits" (since 127 <= 128) when it actually still needs to lose at least 1 token to leave room for both special tokens. A second, subtler risk is off-by-one in the subtraction itself (using max_seq_len - 1 instead of -2, accounting for only one special token), which silently drops one extra content token from every sequence.

Worked example (executed; the original draft shipped only the two function definitions with the results table narrated in prose and no driver code actually producing it, and the function bodies were flattened to a single indent level, both fixed below with a real driver loop whose printed output matches the table)

CLS_ID, SEP_ID, PAD_ID, MAX_SEQ_LEN = 101, 102, 0, 128

def pad_truncate_fixed(token_ids, max_seq_len=MAX_SEQ_LEN):
    content_budget = max_seq_len - 2
    content = token_ids[:content_budget]
    seq = [CLS_ID] + content + [SEP_ID]
    attention_mask = [1] * len(seq)
    pad_needed = max_seq_len - len(seq)
    seq = seq + [PAD_ID] * pad_needed
    attention_mask = attention_mask + [0] * pad_needed
    return seq, attention_mask

def pad_truncate_buggy(token_ids, max_seq_len=MAX_SEQ_LEN):
    # BUG: truncates to max_seq_len BEFORE adding the 2 special tokens
    content = token_ids[:max_seq_len]
    seq = [CLS_ID] + content + [SEP_ID]
    return seq, [1] * len(seq)

for length in [0, 127, 128, 129]:
    tokens = list(range(1000, 1000 + length))
    fseq, fmask = pad_truncate_fixed(tokens)
    bseq, bmask = pad_truncate_buggy(tokens)
    content_kept = min(length, MAX_SEQ_LEN - 2)
    content_dropped = length - content_kept
    print(f"len={length}: fixed_total={len(fseq)} fixed_mask_sum={sum(fmask)} "
          f"kept/dropped={content_kept}/{content_dropped} buggy_total={len(bseq)}")

Running both implementations against input lengths 0, 127, 128, and 129 (via the driver loop above) produced:

The fixed implementation always produces exactly 128 tokens with [CLS] at position 0, [SEP] immediately after the last real content token, and the attention mask matching real-token count exactly (2 for the empty input, since only [CLS]/[SEP] are real; 128 for the other three, since the content budget is fully used and no padding remains). The buggy implementation produces a 2-token sequence for empty input (itself fine) but overshoots max_seq_len by 1 token at input length 127 and by 2 tokens at 128 and 129, exactly the off-by-one/off-by-two the reserved-slots bug predicts, and exactly what the four chosen test lengths are designed to expose: the length-127 case alone already proves the bug, since a correct implementation caps at 128 total regardless.

Trade-offs & pitfalls
Teams often special-case the empty-input case inconsistently, for example skipping [CLS]/[SEP] entirely when there is no content, which then breaks any downstream code that assumes position 0 is always [CLS]; the test at input length 0 exists specifically to pin down that special tokens are still present even with zero real content, as shown above (total length 128, mask sum 2, not 0). A second pitfall is testing only the attention mask's SUM (a scalar) rather than its exact positions: a mask that has the right total count of 1s but marks the wrong positions (for example, marking padding as attended while missing [SEP]) would pass a sum-only assertion while still being wrong; the test in the worked example checks that [SEP] sits at the boundary between real tokens and padding, not just that the mask has the right count. Finally, truncation direction is a real design decision, not a given: dropping from the end of the content (as shown here) is standard for tasks where early context matters most, but a task where the END of a long document matters more (for example, a document's conclusion) would need the opposite truncation direction, and that choice deserves its own explicit test rather than being assumed.

Direct answer
A strong answer to "tell me about a time you failed" or "a time you fell short of one of our values" does three things: it names the failure honestly without over-apologizing or explaining it away, it ties the failure to a specific principle or value rather than a vague "I learned to work harder," and it spends more time on the concrete change made afterward than on the failure itself.

Structured elaboration

• Situation and task: set up briefly; this should not be the bulk of the answer.
• The failure itself: describe plainly what happened, and own your specific part in it ("I failed to X," not "the team failed").
• The principle reflection: name which principle or value, in hindsight, you underweighted in the moment. For example, you may have optimized for looking on-track when the situation called for earlier transparency, or vice versa.
• Result and change: the concrete thing you actually changed (a process, a habit, a communication pattern), and how you know it held up, ideally with a later situation where the new behavior was tested.

Worked example
A candidate had committed to a two-week delivery timeline for a partner team without validating a key dependency first. The dependency slipped, and the candidate didn't flag the risk until the deadline itself, leaving the partner team no time to re-plan. In hindsight, they had underweighted early, uncertain communication in favor of appearing on-track. Afterward, they changed their habit: the moment any dependency looks uncertain, they send a short "this is at risk" note rather than waiting for certainty. Two commitments since then have both surfaced early warnings, giving the receiving team time to adjust rather than being surprised at the deadline.

Trade-offs and pitfalls
A common miss is choosing a "failure" that is actually a humble-brag, a failure that reads as impressive; interviewers notice this quickly, and it undermines the self-awareness the question is testing. Spending most of the answer narrating the failure and only a sentence on the change inverts what the question actually tests; the change and the evidence it worked should take up the majority of the answer. A lesson stated too generically ("I learned to communicate more") is weaker than naming the specific behavioral change that resulted.

Direct answer
Model capacity is, informally, how large and flexible the space of functions a model can represent is; for a neural network the concrete drivers are depth, width, and total parameter count, and for a tree-based model they're max depth, number of leaves, and number of trees. In practice you tell whether current capacity is too high or too low by watching the train/validation gap, the shape of the learning curve, and how performance responds to small capacity changes.

Structured elaboration

• Concrete capacity drivers (neural nets): depth (more layers means more compositions of non-linear transformations, so richer functions), width (more units per layer means more basis functions to combine), and total parameter count (a rough proxy that combines both, and correlates with how much of the training data the model can effectively memorize).
• Concrete capacity drivers (trees): max depth (deeper trees can carve the input space into more, smaller regions), minimum samples per leaf (smaller minimums allow finer, higher-variance splits), and ensemble size (more trees in a random forest averages down variance without raising per-tree bias, which is a slightly different capacity story than a single deep tree).
• Three signals for too-high vs. too-low capacity: (1) the train/validation gap: near-zero training error with a large gap to validation error signals capacity is too high (overfitting); both errors high and close together signals capacity is too low (underfitting). (2) Sensitivity to a small capacity change: if adding a bit more capacity (one more layer, a slightly deeper tree) meaningfully improves validation performance, you were likely under capacity; if it does nothing or makes validation worse, you've likely got enough or too much already. (3) Learning-curve shape: an underfitting model's validation curve plateaus early and low even with lots of data; an overfitting model's validation curve is still sensitive to how much data you have, often improving further if you added more.

Worked example
Two candidate CNN architectures for the same image task: architecture A (5 conv layers, 2M parameters) and architecture B (20 conv layers, 40M parameters), both trained on 10,000 images. If A shows train accuracy 85% and val accuracy 83% (small gap, both mediocre), A is under capacity for this data. If B shows train accuracy 99.9% and val accuracy 78%, B is over capacity for the amount of data available, even though B is the "better" architecture on paper for large-data regimes.

Trade-offs & pitfalls
Capacity is relative to how much data (and how much genuine signal) you have, not an absolute property of an architecture; a huge model with heavy regularization and enough data can behave like a low-effective-capacity model (this is the deeper story behind why large over-parameterized networks can still generalize, an active area with its own dedicated theory beyond simple capacity counting).

Situation: You need to choose between (A) pretraining a much larger model from scratch or (B) buying/collecting more supervised fine‑tuning data for a smaller base model. Here’s a concise trade-off framed by business and technical metrics, plus rules of thumb and a quick estimation method.

High-level trade-off (one line)

• Pretrain large model: higher fixed cost and lead time, broader capabilities and transfer; better long‑tail generalization.
• Invest in fine‑tuning data: lower upfront cost, faster time‑to‑market, higher task-specific accuracy per dollar for narrow tasks.

Business metrics

• Cost: Pretraining scales as compute_hours × hardware_rate; often 5–50× more expensive than fine‑tuning. Fine‑tuning cost ≈ labeling_cost + modest compute.
• Time‑to‑market: Pretraining months; fine‑tuning weeks.
• Flexibility / future value: Large pretrained models amortize across products; pay once, reuse many times.

Technical metrics

• Generalization: Larger pretraining improves out‑of‑domain performance and unsupervised capabilities. Fine‑tuning excels on labeled target distribution but may overfit and not generalize beyond labels.
• Label cost & quality: High-quality labels are expensive and subject-matter constrained; diminishing returns after a point.
• Risk: Pretraining reduces dependency on labels but increases infra/ops risk; fine‑tuning depends on label pipelines and governance.

Rules of thumb

• If target tasks are narrow and high label quality exists: prefer more supervised data on a smaller model.
• If you need broad capabilities, multimodal or long‑tail reasoning, or anticipate many downstream tasks: invest in larger pretraining.
• Break‑even heuristic: when expected reuse_count × value_per_task > pretraining_cost / (fine_tune_cost per task), pretraining pays off.

Quick estimation approach (5‑minute back‑of‑envelope)

1. Estimate costs:
   • C_pretrain = GPU_hours_pretrain × $/GPU_hour + infra/dev
   • C_finetune_per_task = labeling_cost + GPU_hours_finetune
2. Estimate business value:
   • V_task = revenue or cost_saved per successful task deployment
   • Reuse_count = number of tasks/products benefiting from pretraining
3. Compare: Choose pretraining if Reuse_count × V_task − C_pretrain > Σ (V_task_i − C_finetune_per_task_i) across tasks; otherwise iterate with more labeled data.

Practical recommendations

• Start with a small pretrained base + active learning to reduce label needs; simulate gains with few-shot validation.
• If organizational horizon >2 years and multiple products planned, favor pretraining; if 0–6 month launch for a single high‑value task, favor supervised investment.
• Monitor marginal improvement per dollar: plot performance vs dollars for both paths and pick the steeper ROI.

This gives executives clear cost/time tradeoffs, where pretraining is a high‑risk/high‑long‑term‑reward bet and fine‑tuning is fast, predictable, and cost‑efficient for focused needs.

Direct answer

Data leakage in preprocessing means information from outside the training fold, most commonly from the validation or test set, influences a transform the model then learns from. Fitting a scaler, encoder, or imputer on the full dataset before splitting is the most common way this happens, because the fitted statistics (mean, category frequencies, encoding values) end up quietly informed by data the model should never see until evaluation.

Structured elaboration

Two concrete examples: (1) fitting a StandardScaler on the combined train+test set means the scaled training values are shifted very slightly by the test set's own mean and variance, an effect that's small per-feature but systematically biases every downstream metric optimistic; (2) using a feature like "days until cancellation" to predict churn is target leakage in a starker form, since that feature literally cannot be known at prediction time for a customer who hasn't churned yet, it's derived from the very outcome you're trying to predict.

The structural fix is to make the leakage-safe order the ONLY possible order, not just the intended one:

from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.model_selection import cross_val_score
from sklearn.linear_model import LogisticRegression

preprocess = ColumnTransformer([
    ("num", Pipeline([("impute", SimpleImputer(strategy="median")),
                       ("scale", StandardScaler())]), numeric_cols),
    ("cat", Pipeline([("impute", SimpleImputer(strategy="most_frequent")),
                       ("encode", OneHotEncoder(handle_unknown="ignore"))]), categorical_cols),
])
pipeline = Pipeline([("preprocess", preprocess), ("model", LogisticRegression(max_iter=1000))])
scores = cross_val_score(pipeline, X, y, cv=5, scoring="roc_auc")

Because preprocess lives INSIDE the Pipeline, cross_val_score refits it fresh on each fold's training portion only; computing means, encodings, or imputation values on the whole dataset up front, before this pipeline ever sees a fold boundary, is the mistake this structure makes structurally impossible.

Worked example

If you instead ran preprocess.fit_transform(X) once on the full dataset and only THEN passed the transformed X into cross_val_score, every fold's "held-out" validation rows would have already contributed to the scaler's mean and the encoder's category vocabulary before training even begins, silently violating the fold boundary the cross-validation was supposed to enforce.

Trade-offs and pitfalls

Keeping the SAME imputation logic consistent between training and inference matters as much as getting the fold boundary right: if training imputes missing values with a median computed at fit time but the production serving code recomputes a fresh median from whatever data happens to be available at request time, the model sees systematically different feature values in production than it did during training, a subtler leakage-adjacent bug with the same root cause (statistics computed at the wrong time).

Direct answer

An experiment-tracking system for ML needs to record run metadata (hyperparameters, code version, environment), metrics over time (training/validation loss and other tracked quantities), and artifacts (model checkpoints, logs, plots), because each of these three categories supports a different but essential need: reproducing a run exactly, comparing runs to each other, and recovering a specific trained model.

Structured elaboration

• Hyperparameters and configuration: every tunable setting used for the run (learning rate, batch size, model architecture choices, random seed); this is what lets you answer "what configuration produced this result" and compare configurations across runs to understand what mattered.
• Code version: the exact git commit hash (or equivalent) of the training code used, since the same hyperparameters with different code can produce very different results, and without this, a run's result can't be trusted to be reproducible even with everything else recorded.
• Environment: the software environment (container image tag, or a dependency lockfile snapshot), since, as discussed for reproducibility more broadly, different library versions can change numerical behavior even with identical code and hyperparameters.
• Metrics over time: not just the final metric value but the full time series (loss/accuracy per step or epoch), which supports diagnosing training dynamics (did it diverge briefly and recover, was convergence unusually slow), not achievable from a final-value-only record.
• Artifacts: model checkpoints (so the trained model itself is recoverable, not just knowledge that a run happened), and any generated plots/logs useful for later review without needing to re-run the experiment.
• How this supports reproducibility, collaboration, and governance: reproducibility is directly served by the hyperparameters/code/environment triad (everything needed to exactly redo the run); collaboration is served by making all of this discoverable and comparable across a team (rather than trapped in one person's local notes or terminal history); governance (audit, compliance, model lineage in a production environment) is served by the combination of all of the above providing a complete, traceable record of exactly how any deployed model was produced.

Worked example

A logged run record: {run_id: "r-2847", commit: "a3f92e1", hyperparameters: {lr: 3e-4, batch_size: 256, seed: 17}, environment: "training:v2.3.1-cuda12.1", metrics: [{step: 100, loss: 2.31}, {step: 200, loss: 1.87}, ...], artifacts: {checkpoint: "s3://bucket/r-2847/model.ckpt", plots: ["s3://bucket/r-2847/loss_curve.png"]}}; from this single record, another team member could exactly reproduce the run's environment and configuration, understand its training dynamics from the metric time series, and directly load the resulting trained model from the artifact reference, without needing to ask the original author anything.

Trade-offs & pitfalls

Tracking systems that only log the final metric value (not the full time series) or only the hyperparameters (not the code commit and environment) provide a false sense of completeness; a run record missing even one of the three categories (config, metrics-over-time, artifacts) leaves a real gap in either reproducibility, diagnosability, or recoverability that surfaces painfully later, usually when someone actually needs the missing piece.

Approach

Since the elements are guaranteed hashable (hashable means it can be used as a dict key or set member: numbers, strings, and tuples made entirely of hashable values all qualify; lists, dicts, and sets themselves do not, since they can change after creation), a set gives $O(1)$ average membership checks, so the whole pass stays linear: walk the input once, keep a set of everything already seen, and only append an item to the output the first time it appears.

Code (Python 3.12)

def unique_preserve_order(seq):
    """Return items from seq in first-seen order with duplicates removed.
    Assumes every element of seq is hashable.
    """
    seen = set()
    out = []
    for item in seq:
        if item not in seen:
            seen.add(item)
            out.append(item)
    return out

print(unique_preserve_order([3, 1, 3, 2, 1, 4]))
# [3, 1, 2, 4]
print(unique_preserve_order(["b", "a", "b", "c"]))
# ['b', 'a', 'c']
print(unique_preserve_order([]))
# []

The identical logic also dedupes an already-sorted input (a "unique_sorted" ask is the same function; sortedness only changes what the output looks like, not the algorithm) and works the same way on any hashable payload, such as deployment IDs:

deployment_ids = ["d-9", "d-3", "d-3", "d-11", "d-9", "d-2"]
print(unique_preserve_order(deployment_ids))
# ['d-9', 'd-3', 'd-11', 'd-2']

Key points

• The set membership check is what keeps this linear; checking membership against the growing output list instead (if item not in out) would silently work but degrade to $O(n^2)$, since list membership is a linear scan.
• Order is preserved because out is only ever appended to, never reordered; the set is used purely for the O(1) "have I seen this" check and never influences output order.

Complexity

Time: $O(n)$ average, one hash-table lookup and possible insert per element. Space: $O(n)$ worst case (all elements unique), for both seen and out.

Edge cases

• Empty input returns [].
• All-duplicate input returns a single-element list.
• Unhashable elements (a list or dict inside seq) raise TypeError from item not in seen / seen.add(item); per the problem statement this function is only contracted to handle hashable elements, so that is expected and correct behavior, not a bug to work around here.

Adapting for input too large to hold a full result list in memory

The seen set still has to track every distinct value you've encountered (there is no way to know something is a duplicate without remembering it), so the real constraint to relax is holding the full result in memory at once, not the full seen-set. Turn the function into a generator that yields as it goes, so a caller can stream results to a sink (a file, a socket, a downstream queue) instead of building out as a list:

def unique_preserve_order_stream(iterable):
    seen = set()
    for item in iterable:
        if item not in seen:
            seen.add(item)
            yield item

print(list(unique_preserve_order_stream([3, 1, 3, 2, 1, 4])))
# [3, 1, 2, 4]

This removes the $O(n)$ out list (memory now scales only with the number of distinct items seen so far, in seen, plus whatever the consumer of the generator chooses to retain), and it lets a downstream writer start consuming the first unique item immediately instead of waiting for the whole input to be processed. If even seen itself cannot fit in memory (billions of distinct values), the exact-uniqueness guarantee has to be traded for an approximate one (a probabilistic structure such as a Bloom filter, which can have false positives), which is a genuinely different problem from what is asked here and worth naming as the next escalation rather than silently downgrading correctness.

Direct answer. Rename for the reader who has never seen this code: name the thing by what it represents, not by its type or position, and reserve comments for the why a name alone can't carry.

Before

def p(x, l):
    return [i for i, v in enumerate(x) if v == l]

p tells you nothing; x and l are typeless placeholders; there's no docstring, so a reader has to trace the body to learn this finds positions of a target value.

After

from typing import Sequence, TypeVar

T = TypeVar("T")

def find_indices_matching(values: Sequence[T], target: T) -> list[int]:
    """Return the positions in `values` whose element equals `target`."""
    return [index for index, value in enumerate(values) if value == target]

Behavior is unchanged (verified: find_indices_matching([3, 7, 3, 9], 3) == [0, 2], matching the original).

Naming choices, explained

• find_indices_matching names the ACTION and the RESULT shape (plural 'indices') so a caller knows it returns a list, not a single index.
• values and target name the ROLE each parameter plays, which also makes call sites self-documenting: find_indices_matching(prices, 0) reads naturally at the call site even without the definition open.
• Added type hints do double duty as documentation and as a static-analysis safety net; they make the previously-implicit contract (a sequence of comparable things) explicit.
• The one-line docstring exists because 'returns positions matching a target' is a legitimate thing to state up front rather than force every reader to parse a comprehension.

Other tiny refactors worth making

• If this function is called with == semantics that later need to become 'contains' or 'starts with', extracting a predicate callback parameter now (rather than later) avoids a second, diverging function being created under time pressure.
• If values is large and only the first match matters, consider a find_first_index_matching sibling that short-circuits instead of scanning fully, so intent AND cost stay honest together.

Trade-offs and pitfalls

Don't over-invest in perfect names for something that will be deleted next sprint; renaming has a real (if small) review-and-merge cost. But for anything with more than one caller or more than a few weeks of expected lifetime, the rename pays for itself the first time someone other than the author has to touch it.

Requirements:

• Safely evaluate candidate (challenger) models vs current (champion) without degrading user experience.
• Support incremental traffic routing, strong logging, statistical confidence, automated promotion/rollback hooks.
• Low latency, reproducibility, and auditability.

High-level architecture:

• Model Serving Layer: champion and challenger endpoints behind a router/load‑balancer.
• Traffic Router: configurable strategies (percent-based, user-split, session-stickiness, feature-gated).
• Logging & Event Pipeline: capture inputs, decisions, scores, labels/rewards, timestamps, and metadata → streaming system (Kafka) → analysis store (Parquet on S3 / OLAP).
• Evaluation Service: computes online metrics, runs significance tests, triggers CI/CD promotion or rollback.
• Dashboard + Alerting: real-time metrics, statistical summaries, and runbook links.

Traffic routing strategies:

• Shadow (forking): send live traffic to challenger(s) for inference only; responses not served to users: for safety and cold evaluation.
• Percent rollout (canary): route X% of live requests to challenger and serve its output to users when safe.
• User-based split: route based on user-id hash to ensure consistent assignment and avoid contamination.
• Session-stickiness: ensure same session sees same model for coherent UX.
• Feature-gated cohorts: test on specific segments (geography, device, new users).

Logging required:

• Input features (hashed/anonymized), model id/version, model output (scores, classes, probabilities), decision/treatment, served output, latency, request metadata (user-id, session-id, timestamp), ground-truth label or downstream signal when available (click, conversion, refund), experiment id, routing strategy.
• Maintain deterministic seeds and model artifacts for reproducibility.
• Ensure privacy: PII removal, encryption at rest/in transit.

Metrics to monitor for promotion decisions:

• Primary business metric (e.g., conversion, revenue per user, click-through).
• Model-quality: accuracy, AUC/ROC, precision/recall, calibration, log-loss.
• Safety and fairness: error rate by cohort, distributional shifts.
• Latency, resource usage, error rates/exceptions.
• Downstream impact: retention, refunds, churn.
• Secondary guardrails: spike detection, prediction distributions, feature drift.

Statistical tests & methodology:

• Predefine hypothesis and minimum detectable effect (MDE), significance level (α, e.g., 0.05), power (1-β, e.g., 0.8).
• Use A/B testing framework with:
  • Frequentist tests: two-sample t-test or proportion z-test for means/rates, with bootstrap CIs when assumptions fail.
  • Non-parametric bootstrap for skewed metrics (revenue).
  • Correction for multiple comparisons (Bonferroni or Benjamini-Hochberg) if testing many segments/models.
  • Sequential testing (alpha spending or Bayesian methods) if evaluating continuously.
  • Bayesian alternatives: compute posterior probability challenger > champion and decision thresholds (e.g., >95%).
• Evaluate heterogeneity: stratified analysis and uplift by cohort.
• Require pre-specified sample size or sequential stopping rules to avoid peeking bias.

Promotion policy:

• Predefined success criteria: statistically significant improvement on primary metric AND no regressions on guardrail metrics; stability over evaluation window; resource and latency within SLA.
• Manual review gate for model interpretability, fairness checks, and artifact audit before automatic promotion to production.

Rollback criteria:

• Automated rollback triggers: statistically significant degradation on primary metric or any guardrail breach (e.g., latency spike > SLO, error rate increase > threshold, fairness violation).
• Time-based safety net: immediate rollback if severe (critical errors) or progressive rollback (reduce traffic) if moderate impact.
• Post-rollback: run root-cause analysis, compare logs, revert model version, notify stakeholders, and run postmortem.

Operational best practices:

• Keep experiment definitions, MDEs, and decision rules in code/config stored in version control.
• Canary small, monitor closely, increase traffic only after passing checks.
• Retain raw logs and model artifacts for reproducibility and auditing.
• Automate alerts for drift and performance regressions; maintain runbooks for escalation.

This framework balances safety, statistical rigor, and operational practicality for reliable model promotion in production.