Microsoft Machine Learning Engineer Interview Preparation Guide - Mid Level

Direct answer

Build the splitter around scikit-learn's grouping and stratification machinery, generating candidate fold assignments that respect group boundaries first, then greedily adjusting group-to-fold assignment to bring each fold's label distribution as close as possible to the overall distribution, since satisfying both constraints EXACTLY is not always possible.

Structured elaboration

import numpy as np
from collections import defaultdict

class SimpleStratifiedGroupKFold:
    """API: split(X, y, groups) -> yields (train_idx, test_idx).
    Greedily assigns each GROUP (not each row) to whichever fold currently has the
    lowest positive-rate deficit, keeping label balance close across folds while
    guaranteeing no group is split across folds."""
    def __init__(self, n_splits=5, random_state=None):
        self.n_splits = n_splits
        self.random_state = random_state

    def split(self, X, y, groups):
        y = np.asarray(y)
        groups = np.asarray(groups)
        rng = np.random.default_rng(self.random_state)

        # aggregate label counts per group
        unique_groups = np.unique(groups)
        rng.shuffle(unique_groups)  # process in random order to avoid systematic bias
        total_size = len(y)
        target_share = total_size / self.n_splits
        group_pos_count = {g: int(y[groups == g].sum()) for g in unique_groups}
        group_size = {g: int((groups == g).sum()) for g in unique_groups}

        fold_pos = np.zeros(self.n_splits)
        fold_size = np.zeros(self.n_splits)
        group_to_fold = {}

        for g in unique_groups:  # greedy: assign each group to the fold it helps balance most
            candidate_scores = []
            for f in range(self.n_splits):
                new_pos = fold_pos[f] + group_pos_count[g]
                new_size = fold_size[f] + group_size[g]
                label_penalty = abs(new_pos / max(new_size, 1) - y.mean())
                # without a size term, the greedy rule can chase label balance right into an
                # empty or tiny fold; penalizing distance from the target fold SIZE too keeps
                # fold sizes from collapsing while still preferring the label-closest fold
                size_penalty = abs(new_size - target_share) / total_size
                candidate_scores.append(label_penalty + size_penalty)
            best_fold = int(np.argmin(candidate_scores))
            group_to_fold[g] = best_fold
            fold_pos[best_fold] += group_pos_count[g]
            fold_size[best_fold] += group_size[g]

        row_fold = np.array([group_to_fold[g] for g in groups])
        for f in range(self.n_splits):
            test_idx = np.where(row_fold == f)[0]
            train_idx = np.where(row_fold != f)[0]
            yield train_idx, test_idx

groups = np.array([1,1,1,2,2,3,3,3,3,4,4,5,5,5])
y = np.array([0,0,1,0,1,0,0,1,0,1,0,0,1,0])
skf = SimpleStratifiedGroupKFold(n_splits=3, random_state=0)
for i, (train_idx, test_idx) in enumerate(skf.split(None, y, groups)):
    print(f"fold {i}: test groups={sorted(set(groups[test_idx]))}, test pos rate={y[test_idx].mean():.2f}")

The greedy algorithm: process groups in a randomized order, and for each group, assign it to whichever fold minimizes a COMBINATION of two penalties, how far that fold's positive rate would land from the overall dataset's positive rate, and how far that fold's resulting size would land from its equal target share of the data. The size term is not optional: a purely label-rate-driven greedy rule can walk straight into a degenerate solution (an empty or near-empty fold) whenever that happens to minimize label-rate deviation locally, which is exactly the kind of quietly-wrong behavior that only shows up by actually running the code, not by reading the label-balancing logic in isolation. This is a greedy, not globally optimal, heuristic, since finding the truly optimal group-to-fold assignment is a combinatorial problem, but it works well in practice and runs in roughly linear time in the number of groups.

Limitations of plain GroupKFold: it guarantees no group is split across folds but makes NO effort to balance label proportions across folds at all, so with a skewed label distribution and unevenly-sized groups, some folds can end up with meaningfully different positive rates than others purely by which groups happened to land where, exactly the gap this custom implementation exists to close.

Worked example

Running the code above on 5 groups (sizes 3, 2, 4, 2, 3) with a mix of label rates over an overall positive rate of $5/14\approx 0.357$, the 3-fold split produces test sizes of 4, 5, and 5 rows (no empty or near-empty fold) with per-fold positive rates of 0.25, 0.40, and 0.40, all within a reasonable band around the true 0.357 rate rather than either collapsing to a degenerate empty fold or drifting to an extreme rate the way an unweighted, size-blind greedy rule did before the size penalty was added.

Trade-offs and pitfalls

When groups vary wildly in size (one enormous group, several tiny ones), even this improved greedy heuristic can struggle to achieve good stratification, since assigning the one enormous group anywhere dominates both the size and label balance of whichever fold it lands in, regardless of how the smaller groups are subsequently balanced around it; in that specific situation, it's worth checking the group-size distribution before trusting the stratification quality, rather than assuming the greedy algorithm always closes the gap.

Direct answer
Depth and width are not interchangeable ways to add capacity: depth buys compositional, hierarchical representational power efficiently but makes optimization harder, while width is easier to optimize and parallelize but needs disproportionately more parameters to match what depth can express for structured tasks.

Structured elaboration
Representational capacity: a wide enough single hidden layer can approximate any continuous function (the universal approximation theorem), but doing so for a genuinely compositional function can require exponentially many units, whereas a deep network can represent the same function with exponentially fewer parameters by reusing intermediate features across layers.
Optimization difficulty: deeper networks are harder to train, since a deeper stack pushes gradients through more repeated Jacobian multiplications (vanishing/exploding gradients) and a rougher loss landscape; residual connections, normalization layers, and careful initialization are what make depth beyond roughly 20 layers practically trainable at all. Very wide networks, by contrast, behave closer to a well-conditioned, near-convex optimization problem (the Neural Tangent Kernel regime), which is part of why they can be easier to train stably, at the cost of needing far more memory and compute per layer.
Parameter efficiency: for tasks with real hierarchical structure (most perception and language tasks), depth is the more parameter-efficient way to add capacity; for a task with no real compositional structure, adding width may reach the same accuracy with less optimization difficulty even if it costs more parameters.
Generalization: this connects directly to the double-descent phenomenon, where increasing model size (whether depth or width) past the point of exactly fitting the training data can, counterintuitively, continue to IMPROVE validation performance rather than overfitting further; this means "more capacity" is not automatically bad for generalization the way classical bias-variance intuition alone would suggest, though the practical implication is still to validate on held-out data rather than reasoning from capacity alone.

Worked example
A concrete illustration of the parameter-efficiency claim: representing a function that is the composition of $k$ simple pairwise interactions can require on the order of $2^k$ hidden units in a single wide layer to capture directly, but only $O(k)$ units spread across $k$ depth layers if each layer can build on the previous one's output, precisely because depth lets you REUSE the same small set of primitive features at every stage rather than needing a distinct unit for every possible combination.

Trade-offs & pitfalls
Practical guidance under compute and latency constraints: prefer deeper architectures WITH residual connections when the task has real hierarchical structure and you can afford the training-time complexity of tuning a deep network; prefer wider (but shallower) architectures when training stability, parallelism, or a hard latency ceiling on sequential compute matters more than squeezing out maximum parameter efficiency (a wide network's operations parallelize better on modern accelerators, since there is less sequential dependency between layers). A common mistake is treating "add more layers" as a free capacity increase; past a certain depth without residuals and normalization, additional layers can actively HURT training by making optimization harder without giving any additional representational benefit that the optimizer can actually reach.

Direct answer

For a 100k-token context under a strict p95 (95th percentile) latency target, start with full attention over a retrieved subset: a retriever picks the roughly 8k tokens most relevant to the request and the unmodified dense model attends fully over those. It keeps the pretrained model's behaviour, bounds latency by the retrieved budget instead of the document length, and its recall is directly measurable. Choose sparse local attention when the task needs the whole document read in order and the model can be adapted, and choose linear attention only if quality has been validated after training or distillation, because it changes the function the model computes. What would flip the recommendation: tasks whose answer is spread across the whole document with no localized evidence (retrieval cannot select it), or a retrieval recall that evaluation shows is too low.

What each option does to cost

Terms: sparse local attention lets each token attend only to a nearby window (here the previous 4096 tokens); linear attention replaces the softmax by a feature-map form that keeps a fixed-size running summary instead of every token's keys and values; a dense checkpoint is a saved model that uses ordinary full attention. A FLOP is one floating-point operation (one add or multiply) and a TFLOP is a trillion of them. Time-to-first-token is the delay from sending a request to receiving the first output token, mostly the prefill; concurrency is how many requests are served at the same time. Prefill is processing the prompt once; decode is generating one token at a time, reading the stored keys and values (the KV cache) each step; GQA (grouped-query attention) shares each key/value head across several query heads, which shrinks the cache.

The program does the arithmetic for an example shape (32 layers, 32 query heads, 8 key/value heads, head dimension 128, 16-bit values; an assumption, not a statement about any product). FLOPs are counted as 4 times queries times keys times heads times head dimension per layer (two matrix multiplies). It also simulates how often a window covers a fact placed uniformly at random, and builds a TF-IDF (term-frequency weighted by rarity) retriever as a stand-in for a real embedding retriever. Recall@k is the fraction of questions for which the correct chunk appears among the top k retrieved chunks.

import numpy as np

# Example model shape (an assumption for the arithmetic, not a claim about any product)
L, d_model, n_q_heads, kv_heads, d_head, bytes_el = 32, 4096, 32, 8, 128, 2
n = 100_000
w, retrieved, chunk = 4096, 8_000, 1000          # local window, tokens kept after retrieval

def attn_flops_per_layer(n_q, n_k):               # QK^T and AV: 2 matmuls, 2 FLOPs per multiply-add
    return 4 * n_q * n_k * n_q_heads * d_head

kv_per_tok = 2 * L * kv_heads * d_head * bytes_el
print("KV cache bytes per token:", kv_per_tok)
rows = [("full attention, 100k tokens", n, n),
        ("sliding window w=4096", n, w),
        ("full attention over 8k retrieved tokens", retrieved, retrieved)]
print("\nprefill (process the prompt once): attention score+mix FLOPs, all layers")
for name, nq, nk in rows:
    print(f"{name:42s} {attn_flops_per_layer(nq, nk) * L / 1e12:8.2f} TFLOPs")
# linear attention: per token, state update + read = 2 * (2 * d_head * d_head) per head
lin = n * (4 * d_head * d_head) * n_q_heads * L
print(f"{'linear attention (feature dim = d_head)':42s} {lin / 1e12:8.2f} TFLOPs")

print("\ndecode (one new token): GiB of KV cache (or state) read, all layers")
for name, toks in [("full attention", n), ("sliding window", w), ("retrieved 8k", retrieved)]:
    print(f"{name:20s} {toks * kv_per_tok / 2**30:7.3f} GiB")
state = L * n_q_heads * d_head * d_head * bytes_el
print(f"{'linear attention':20s} {state / 2**30:7.3f} GiB (fixed state, independent of n)")

# Coverage: a question at the end, the needed fact at a uniformly random earlier position
rng = np.random.default_rng(0)
pos = rng.integers(0, n, 100_000)
dist = n - pos
print("\nfraction of facts within the sliding window of the question:", round(float((dist <= w).mean()), 3))

# Retrieval stand-in: TF-IDF over 100 chunks of 60 random word ids from a 300-word vocabulary; the question is 3 words found in the target chunk
V, n_chunks, trials = 300, 100, 200
def trial(seed):
    r = np.random.default_rng(seed)
    docs = r.integers(0, V, (n_chunks, 60))                         # 60 random word ids per chunk, so words recur across chunks
    t = r.integers(0, n_chunks)
    q_words = r.choice(V, 3, replace=False)                      # the question uses 3 ordinary words
    docs[t, :3] = q_words                                       # the target chunk contains all three
    tf = np.zeros((n_chunks, V)); [np.add.at(tf[i], docs[i], 1) for i in range(n_chunks)]
    idf = np.log(n_chunks / (1 + (tf > 0).sum(0)))
    d_vec = tf * idf; d_vec /= np.linalg.norm(d_vec, axis=1, keepdims=True) + 1e-9
    qv = np.zeros(V); np.add.at(qv, q_words, 1); qv *= idf; qv /= np.linalg.norm(qv) + 1e-9
    rank = np.argsort(-(d_vec @ qv))
    return int(np.where(rank == t)[0][0])
ranks = np.array([trial(s) for s in range(trials)])
print("\nTF-IDF stand-in retriever: recall@k that the correct chunk is in the top k of 100 (200 trials)")
for k in [1, 4, 8]:
    print(f"k={k}: {(ranks < k).mean():.3f}")

Output:

KV cache bytes per token: 131072

prefill (process the prompt once): attention score+mix FLOPs, all layers
full attention, 100k tokens                 5242.88 TFLOPs
sliding window w=4096                        214.75 TFLOPs
full attention over 8k retrieved tokens       33.55 TFLOPs
linear attention (feature dim = d_head)        6.71 TFLOPs

decode (one new token): GiB of KV cache (or state) read, all layers
full attention        12.207 GiB
sliding window         0.500 GiB
retrieved 8k           0.977 GiB
linear attention       0.031 GiB (fixed state, independent of n)

fraction of facts within the sliding window of the question: 0.041

TF-IDF stand-in retriever: recall@k that the correct chunk is in the top k of 100 (200 trials)
k=1: 0.560
k=4: 0.990
k=8: 1.000

These are operation and byte counts, not latencies; latency also depends on hardware, batching and kernel quality, which is why the evaluation below measures it. The retrieval row excludes the retriever's own cost.

Turning the counts into seconds

To feel the table, divide by an assumed speed. The program below assumes, purely as an illustration and not as a statement about any hardware, a sustained 200 TFLOP/s for attention work and 2 TB/s of memory bandwidth for reading the cache. The results are lower bounds: they leave out the feed-forward layers, the projections, the retriever and all overheads, so real times are longer. They still show the shape of the decision: at these assumed speeds, full attention over 100k tokens needs at least 26 seconds of prefill attention work against 0.17 seconds for the 8k retrieved budget, so with an example p95 target of a couple of seconds only the last three options are even candidates.

The 8k retrieved budget is a design choice, not a constant. It is 8 chunks of 1000 tokens. Prefill attention cost grows with the square of the budget, so 16k tokens costs four times as much as 8k (0.67 s against 0.17 s here), while 4k halves the room for evidence. The budget to pick is the largest one that still meets the p95 target with margin, then check recall at that many chunks on real questions.

# Illustrative hardware numbers (assumptions, not a claim about any product):
flops_per_s = 200e12          # sustained 200 TFLOP/s of useful attention work
bytes_per_s = 2e12            # 2 TB/s of memory bandwidth for reading the KV cache
L, n_q_heads, kv_heads, d_head, bytes_el = 32, 32, 8, 128, 2
kv_per_tok = 2 * L * kv_heads * d_head * bytes_el

def prefill_tflops(n_q, n_k):
    return 4 * n_q * n_k * n_q_heads * d_head * L / 1e12

print("prefill attention time lower bound at 200 TFLOP/s (illustrative), seconds")
for name, nq, nk in [("full attention, 100k", 100_000, 100_000), ("sliding window 4096", 100_000, 4096),
                     ("full attention over 8k retrieved", 8000, 8000)]:
    t = prefill_tflops(nq, nk)
    print(f"{name:34s} {t:9.2f} TFLOPs -> {t * 1e12 / flops_per_s:7.2f} s")
lin = 100_000 * 4 * d_head * d_head * n_q_heads * L / 1e12
print(f"{'linear attention':34s} {lin:9.2f} TFLOPs -> {lin * 1e12 / flops_per_s:7.2f} s")

print("\ndecode per-token cache read time at 2 TB/s (illustrative), milliseconds")
for name, toks in [("full attention, 100k", 100_000), ("sliding window 4096", 4096), ("8k retrieved", 8000)]:
    print(f"{name:34s} {toks * kv_per_tok / bytes_per_s * 1e3:7.2f} ms")

print("\nwhy not a different retrieved budget: prefill attention cost grows with the square of the budget")
for budget in [4000, 8000, 16000]:
    t = prefill_tflops(budget, budget)
    print(f"budget {budget:6d} tokens = {budget // 1000} chunks of 1000: {t:7.2f} TFLOPs -> {t * 1e12 / flops_per_s:5.2f} s")

Output:

prefill attention time lower bound at 200 TFLOP/s (illustrative), seconds
full attention, 100k                 5242.88 TFLOPs ->   26.21 s
sliding window 4096                   214.75 TFLOPs ->    1.07 s
full attention over 8k retrieved       33.55 TFLOPs ->    0.17 s
linear attention                        6.71 TFLOPs ->    0.03 s

decode per-token cache read time at 2 TB/s (illustrative), milliseconds
full attention, 100k                  6.55 ms
sliding window 4096                   0.27 ms
8k retrieved                          0.52 ms

why not a different retrieved budget: prefill attention cost grows with the square of the budget
budget   4000 tokens = 4 chunks of 1000:    8.39 TFLOPs ->  0.04 s
budget   8000 tokens = 8 chunks of 1000:   33.55 TFLOPs ->  0.17 s
budget  16000 tokens = 16 chunks of 1000:  134.22 TFLOPs ->  0.67 s

Weaknesses that matter for a decision

• Sparse local attention. In the simulation a 4096 window around the question covers only 0.041 of uniformly placed facts of a 100k context. Layers stacked on top propagate information a window at a time, but lossy, so distant dependencies are the weak point. Starting from a dense checkpoint (a saved model with ordinary full attention) is feasible because the weights are the same and only the mask changes, but the model was pretrained with full visibility, so it needs fine-tuning at the new pattern to recover quality.
• Linear attention. Replaces the softmax by a feature-map kernel and a fixed-size state, so it is the cheapest, but the state is a lossy summary of 100k tokens and exact recall of one token degrades. A dense checkpoint cannot simply be switched over; it needs conversion training, meaning extra training that teaches the model to work with the changed attention, often by distillation (training the changed model to imitate the original model's outputs).
• Retrieval plus dense attention. Cost depends on the retrieved budget, not the document. In the stand-in, the correct chunk is ranked first 0.560 of the time, within the top 4 in 0.990 and within the top 8 in 1.000 (200 trials, synthetic words), so recall climbs steeply with k on this toy. Treat that as a best case: the toy plants all three question words inside the target chunk, so finding it is nearly guaranteed by construction, and its chunks hold 60 word ids as a stand-in for 1000-token chunks. Do not read 1.000 at k=8 as the recall to expect; a real retriever on real questions, where the question often shares few words with the evidence, needs its own measurement. It fails on queries that need aggregation across the whole document ("what are the recurring themes?").

How to evaluate

1. Latency: measure p95 time-to-first-token and time per output token under production-like concurrency and the real distribution of prompt lengths, not the average and not one 100k prompt. Pick candidates that meet the target, then compare quality.
2. Distant-dependency recall: place a fact at controlled depths (start, middle, end) and ask for it, scoring exact retrieval; add multi-fact variants and multi-hop variants (questions whose answer needs combining facts from several places in the document).
3. Whole-document tasks: summarization and cross-section questions on real documents, scored by humans or a model grader (a language model asked to score the answers) whose agreement with humans has been checked.
4. Retrieval quality: recall@k of the evidence chunk, tracked separately so a generation failure is distinguishable from a retrieval failure.
5. Reference arm: run full attention at 100k offline on a sample as the quality ceiling, and report each option's gap to it with confidence intervals (a range the true gap plausibly lies in, given the sample size), never from a handful of examples.

Common mistakes

• Choosing on asymptotic complexity alone: linear attention's constant factors and quality loss can outweigh its O(n).
• Measuring only average latency when the requirement is p95.
• Evaluating retrieval by final-answer accuracy alone, which hides whether the retriever or the model failed.

Training/validation/test splits: training fits model, validation tunes hyperparameters, test estimates final generalization. For 100k examples: common split 70/15/15 (70k train, 15k val, 15k test). If time-series: use chronological splits (e.g., first 80% train, next 10% val, final 10% test) to avoid leakage and mimic production. For class imbalance: ensure stratified splits so class proportions are preserved across sets; if minority class is tiny, consider oversampling/SMOTE on training only or use larger validation/test sets for reliable estimates. Also use cross-validation or repeated stratified folds when data is limited.

Direct answer

K-means only moves centroids (cluster centres, each the mean of its points) downhill from where they start, so it finds the local optimum nearest to the start, not the best one. A bad start, for example two centroids inside one natural group and none in another, gets stuck there with a high total squared distance (inertia). Smarter seeding (k-means++) picks starting centroids that are spread apart, and running several starts and keeping the lowest inertia protects against the rest.

Why a bad start hurts

Each iteration assigns points to the nearest centroid and then moves the centroid to its members' mean. A centroid never jumps across empty space to a far-away group, because points far from it are claimed by other centroids. Suppose the data has three groups and two initial centroids land in group A while one sits between B and C. The two in A split A in two, the third stays between B and C and averages them into one cluster. Every step is an improvement, no step can fix the mistake, and the loop stops at a fixed point (a state where one more round changes nothing) with higher inertia than the correct three-group solution. The result also changes from seed to seed, which is a reproducibility problem on its own.

Here is that failure with coordinates, on nine points on a line: group A at 0, 1, 2, group B at 20, 21, 22 and group C at 40, 41, 42. The bad start puts two centroids in A (at 0 and 2) and one at 31, between B and C. The good start puts one centroid in each group.

import numpy as np

# three groups on a line: A around 1, B around 21, C around 41
pts = np.array([0.0, 1.0, 2.0, 20.0, 21.0, 22.0, 40.0, 41.0, 42.0])

def run(centres):
    c = np.array(centres, dtype=float)
    for it in range(1, 10):
        labels = np.abs(pts[:, None] - c[None, :]).argmin(axis=1)    # nearest centroid
        new = np.array([pts[labels == j].mean() if (labels == j).any() else c[j] for j in range(3)])
        inertia = ((pts - new[labels]) ** 2).sum()
        print(f"  iteration {it}: centroids {np.round(new, 1).tolist()}, inertia {inertia:.1f}")
        if np.allclose(new, c):
            break
        c = new

print("bad start: two centroids in group A, one between B and C")
run([0.0, 2.0, 31.0])
print("good start: one centroid in each group")
run([1.0, 21.0, 41.0])

Output:

bad start: two centroids in group A, one between B and C
  iteration 1: centroids [0.5, 2.0, 31.0], inertia 604.5
  iteration 2: centroids [0.5, 2.0, 31.0], inertia 604.5
good start: one centroid in each group
  iteration 1: centroids [1.0, 21.0, 41.0], inertia 6.0

The bad start stops after one update at inertia 604.5, because the centroid at 31 is the mean of B and C together (each of those six points is about 10 away from it), while the two centroids in A split three points that were already tight (point 1 is exactly as close to both, and the tie goes to the first). Nothing in the loop can move a centroid from A across to separate B from C. The good start reaches inertia 6.0 immediately.

Initialisation strategies

k-means++ in implementable steps

1. Choose the first centroid uniformly at random from the data.
2. For every point, compute $D(x)^2$: the squared distance to its nearest already chosen centroid.
3. Choose the next centroid from the data with probability $D(x)^2 / \sum_{x'} D(x')^2$. Far points are much more likely, points already close to a centroid almost never.
4. Repeat steps 2 and 3 until $k$ centroids exist, then run ordinary k-means.

In practice, k-means++ followed by several restarts is the standard default (k-means++ is scikit-learn's default init); farthest-first and informed seeds are specialised choices. Squaring matters: it makes the chance grow faster than distance, so a point twice as far is four times as likely.

Worked example, and measured effect

The program prints the hand-sized step, then clusters 600 points from six blobs 100 times with a single start per seed, once with init="random" and once with init="k-means++" (scikit-learn 1.5.2), and compares each run's inertia with the best found in 50 restarts.

import numpy as np
from sklearn.cluster import KMeans
from sklearn.datasets import make_blobs

# 1) One k-means++ seeding step by hand on six points on a line
pts = np.array([0.0, 1.0, 2.0, 10.0, 11.0, 20.0])
first = 0.0                                    # suppose the first centre is the point at 0
d2 = (pts - first) ** 2                        # squared distance to the nearest chosen centre
prob = d2 / d2.sum()
print("points      :", pts.tolist())
print("D^2         :", d2.tolist())
print("probability :", np.round(prob, 3).tolist())

# 2) Many single starts: random vs k-means++ on six blobs, k = 6
X, _ = make_blobs(n_samples=600, centers=6, cluster_std=0.9, random_state=3)
best = KMeans(6, n_init=50, random_state=0).fit(X).inertia_
print("best inertia found with 50 restarts:", round(best, 1))
for init in ["random", "k-means++"]:
    inertias = np.array([KMeans(6, init=init, n_init=1, random_state=s).fit(X).inertia_
                         for s in range(100)])
    near_best = (inertias <= 1.01 * best).mean()
    print(f"{init:9s} n_init=1 over 100 seeds: share within 1% of best = {near_best:.2f}, "
          f"median = {np.median(inertias):.1f}, worst = {inertias.max():.1f}")

Output:

points      : [0.0, 1.0, 2.0, 10.0, 11.0, 20.0]
D^2         : [0.0, 1.0, 4.0, 100.0, 121.0, 400.0]
probability : [0.0, 0.002, 0.006, 0.16, 0.193, 0.639]
best inertia found with 50 restarts: 866.8
random    n_init=1 over 100 seeds: share within 1% of best = 0.27, median = 935.6, worst = 2760.6
k-means++ n_init=1 over 100 seeds: share within 1% of best = 0.45, median = 884.4, worst = 948.4

On the six points, squared distances are 0, 1, 4, 100, 121 and 400, giving selection probabilities 0, 0.002, 0.006, 0.16, 0.193 and 0.639: the second centroid is most likely the point at 20, and almost never one near 0. On the six-blob data, with one start each, random initialisation landed within 1% of the best inertia in 27 of 100 seeds, k-means++ in 45 of 100. The median inertia was 935.6 for random and 884.4 for k-means++, and the worst run was 2760.6 against 948.4. So on this dataset k-means++ shrinks the damage of an unlucky start a lot (worst case) but does not remove it (only 45 of 100 near best), which is why restarts remain standard. These numbers come from one dataset. On other datasets of the same kind k-means++ puts more of its single starts near the best inertia, but how much the worst run improves varies from a large gap to almost none, so take away the direction (more good starts, fewer terrible ones, never zero bad ones), not the exact shares.

Pitfalls

• k-means++ spreads centroids by distance, so with outliers it can seed on an outlier. Remove or down-weight extreme points first.
• Seeding quality does not fix a wrong $k$ or unscaled features.
• Report the seed and number of restarts so the segmentation is reproducible.

Direct answer

The evaluation plan needs three layers before a clinician ever sees a prediction: rigorous offline validation on data that looks like the deployment population, a period where the model runs silently alongside clinicians so you measure real-world performance without it influencing care, and a staged, monitored rollout with pre-agreed stopping rules. The single organizing principle: in triage, a false negative (missed urgent case) and a false positive (unnecessary alarm) have very different costs, so the plan has to be built around that asymmetry rather than around a single aggregate accuracy number.

Structured elaboration

Evaluation-to-rollout pipeline

flowchart TD
A[Offline evaluation on multi-site holdout] --> B[Silent shadow deployment]
B --> C{Safety and fairness gates}
C -->|Fail| D[Return to model or data team]
C -->|Pass| E[Staged pilot: single site, assistive mode]
E --> F[Prospective RCT or stepped-wedge trial]
F --> G{Non-inferiority and safety met}
G -->|No| D
G -->|Yes| H[Full clinical rollout]
H --> I[Continuous drift and subgroup monitoring]
I --> J[Periodic re-validation]
J --> C

1. Offline evaluation, before anything touches a clinician

• Hold out data by time and by site (not a random split), so the estimate reflects performance on a hospital or population the model has not seen, catching cases where the model quietly learned a site-specific artifact instead of the clinical signal.
• Report sensitivity (recall) on the urgent-case class as the headline metric, not overall accuracy, because urgent cases are usually a small fraction of volume and accuracy can look excellent while missing most of them.
• Report calibration (does a predicted probability of 0.8 correspond to roughly 80% of those cases actually being urgent) separately from discrimination, because a clinician needs to trust the number, not just the ranking.
• Break every metric out by demographic subgroup (age, sex, site, language) rather than reporting one pooled number, since a pooled metric can hide a subgroup where the model is unsafe.

2. Silent (shadow) deployment
The model runs on live cases in real time, its predictions are logged, but clinicians never see them and never act on them. This is the step that catches "worked in offline eval, breaks in production" failures: label leakage in the historical data, a preprocessing mismatch between the training pipeline and the live feature pipeline, or a shift in the patient population since the training data was collected. Compare shadow-period sensitivity and calibration against the offline estimate before proceeding.

3. Staged, human-in-the-loop clinical validation

• Assistive mode first: the model's output (score plus a plain-language rationale) is shown to the clinician, who retains the decision; nothing is automated.
• A single site or unit before a multi-site rollout, so you learn how clinicians actually use the tool (do they defer to it, ignore it, use it only for edge cases) before that behavior is baked in everywhere.
• A prospective trial (randomized or stepped-wedge) with endpoints defined ahead of time: time-to-treatment for urgent cases, rate of missed critical events, and a pre-specified non-inferiority margin, so "did it help" is answered by a design that was agreed on before you saw the data, not by a post-hoc read of favorable-looking numbers.
• Stopping rules for harm: if the missed-urgent-case rate crosses a pre-agreed threshold during the trial, the trial halts. This has to be decided before the trial starts, not negotiated after a bad week.

4. Governance and monitoring that outlives the launch

• A model card and data sheet documenting training population, known limitations, and intended use, reviewed by a model risk or clinical safety committee before go-live.
• Post-deployment: real-time dashboards for sensitivity/FNR (false negative rate, the fraction of true urgent cases the model misses) by subgroup, calibration drift, and feature drift, with automatic alerts and a rollback path if any breach the pre-agreed threshold.
• A retraining and re-validation cadence: the plan doesn't end at launch, it specifies how often the model gets re-evaluated against fresh holdout data as the patient population and clinical practice shift.

Worked example

Say the target patient population has an urgent-case prevalence of 3%, and the triage system processes 5,000 cases a day.

At a false-negative rate (FNR, the fraction of true urgent cases the model misses) of 3%, that is:

which gives 4.5 missed cases/day at FNR = 3%, versus 15/day at FNR = 10%. That gap is why the acceptance threshold is negotiated on sensitivity, not overall accuracy: a model that is 97% accurate overall but has a 10% FNR on the urgent subgroup is missing three times as many critical cases as one with a 3% FNR.

Now suppose the shadow deployment observes 1,500 confirmed urgent cases over its run and the model misses 40 of them:

So the observed FNR is about 2.67%, with a 95% confidence interval of roughly 1.85% to 3.48%. That interval is wide even with 1,500 urgent-case events, which is the practical argument for why shadow periods for rare, high-stakes subgroups need to run long enough (often across multiple sites, over weeks) to pin the estimate down tightly enough to make a go/no-go call with confidence, rather than reading too much into a single week of data.

Trade-offs & pitfalls

• The most common wrong turn: optimizing and reporting a single aggregate metric (AUC, accuracy) instead of the sensitivity/FNR split by subgroup that actually reflects clinical harm. A senior answer leads with the asymmetric-cost framing, not with a generic "we'll do an 80/20 train/test split."
• Skipping the silent/shadow stage and going straight from offline metrics to a clinician-facing pilot is the fastest way to discover a training-serving skew (systematic difference between how features were computed in training versus in the live serving path) as a patient-safety incident instead of a dashboard alert.
• Treating the prospective trial as a formality after the model is already "approved" internally, rather than as the actual gate: the stopping rules and non-inferiority margin must be able to kill the rollout, or they are theater.
• Over-indexing on model performance while under-specifying the human factors: if the UI doesn't clearly communicate uncertainty and the clinician silently starts rubber-stamping the model's suggestion (automation bias), the safety plan has a gap no amount of offline validation catches.
• Regulatory and compliance scope (for example FDA guidance on software as a medical device, or HIPAA for patient data handling) needs to be identified early, since it can change what evidence the trial has to produce, not bolted on after the pilot is already running.

Direct answer
Favor simple heuristics or manual tuning when interpretability of the tuning process itself matters (regulatory or team-trust reasons), when time-to-production is tight and a well-known reasonable default is likely close enough, when the model or dataset is cheap and small enough that a quick manual pass is genuinely faster than setting up an automated search, or when the team lacks the maintenance capacity to own an AutoML pipeline long-term.

Structured elaboration
Interpretability: a manually-chosen, well-understood set of hyperparameters is easier to defend and explain (to auditors, to a new team member, to yourself six months later) than "whatever an automated Bayesian search happened to converge on," which matters more in regulated or high-stakes settings. Time-to-production: for a first version of a model under real deadline pressure, a few well-chosen manual configurations informed by domain experience or a quick heuristic (a known-good default learning rate, a standard tree depth for this kind of tabular problem) often gets you 90% of the way there in a fraction of the time an automated search would take. Compute cost: automated search, especially Bayesian optimization or multi-fidelity methods, has real setup and infrastructure overhead; for a one-off, small model, that overhead can exceed the time a knowledgeable person would spend manually trying 3-4 sensible configurations. Long-term maintainability: an AutoML pipeline itself needs to be maintained, monitored, and occasionally debugged; a team without the capacity to own that infrastructure may be better served by simpler, more transparent manual tuning, at least until the model's importance justifies the investment.

Worked example
A small internal reporting model needed within a week, with a modest dataset and low stakes if it's not perfectly tuned: a manual pass over 3-4 sensible configurations based on known defaults for this model family is likely the right call, versus standing up a Bayesian-optimization pipeline whose setup time alone could exceed the entire project's timeline.

Trade-offs & pitfalls
The risk of defaulting to manual tuning out of habit, even once a model becomes important enough (high business value, frequent retraining, many hyperparameters) that automated search's efficiency gains clearly outweigh its setup cost, is leaving real performance on the table; revisit this choice as the model's stakes and retraining frequency grow, rather than treating the initial "manual is fine for now" decision as permanent.

Requirements:

• Convert a trained model into optimized variants: quantized (INT8), pruned, and vendor-compiled (TensorRT/ONNX Runtime/AOT).
• Run correctness (unit + E2E) and performance (latency, throughput, memory) tests.
• Promote only models meeting accuracy and latency SLOs; provide gate criteria and automated rollback.

High-level architecture:
CI (commit/tag) → Training artifact registry → Optimization pipeline (CD) → Test harness + benchmarker → Promotion/orchestration → Model registry & deployment

Core components and responsibilities:

1. Artifact store: store trained checkpoints + metadata (git commit, dataset version, baseline metrics).
   Tech: S3/GCS + metadata DB (Postgres/ML Metadata).
2. Optimization pipeline (runner: Airflow/GitHub Actions/Jenkins/XFlow):
   • Steps: load checkpoint → convert to ONNX → apply pruning schedule → quantize (post-training or QAT (quantization-aware training)) → compile with target runtimes.
   • Produce artifacts with tags: variant type, target hardware, binary.
3. Test harness:
   • Unit tests: model shape, deterministic outputs on unit inputs.
   • Regression tests: compare outputs to baseline on holdout dataset; compute accuracy drop.
   • E2E tests: run inference through serving stack (containerized) on representative data path.
   • Benchmarks: run p95 latency, throughput, memory, and energy on target infra (CPU/GPU/edge).
     Tech: pytest + custom inference harness; perf measurement with Locust/jMeter or custom runner; use reproducible infra via infra-as-code (Terraform) and containers.
4. Policy engine (gatekeeper):
   • Gate criteria example:
     • Accuracy drop <= 0.5% absolute (or relative) vs baseline OR within statistically equivalent test (paired t-test, 95% CI).
     • p95 latency <= SLO (service-level objective) (e.g., 50ms) and 99th <= 2× p95 baseline.
     • Memory footprint <= limit for target device.
     • No functional test failures.
     • Artifact passes security/static checks (size limits, banned ops).
   • If multiple variants pass, rank by cost/latency/accuracy tradeoff and auto-select top for promotion.
5. Promotion & deployment:
   • On pass: write entry to Model Registry (versioned), create immutable deployment bundle (container image + config), trigger canary deployment (e.g., 5% traffic).
6. Monitoring & rollback:
   • Observability: collect accuracy via shadow traffic/labels, latency, error rate, resource metrics (Prometheus + Grafana).
   • Rollback triggers:
     • Accuracy regression in production beyond rolling threshold (e.g., >1% drop on labeled sampling).
     • Latency SLO violation sustained > N minutes or increased error rate.
   • Automated rollback: orchestrator (Argo Rollouts / Kubernetes) shifts traffic back to previous stable model and marks failed artifact in registry.
   • Postmortem: snapshot inputs that caused failure, alerting, and create ticket with reproducible test runs.

Data flow:

• Training produces checkpoint → artifact store → optimization jobs spawn variants → each variant runs test suite and benchmarks on test infra → results stored in metadata DB → policy engine evaluates gates and updates registry → deployment orchestrator handles rollout and monitors real traffic.

Scalability & repeatability:

• Parallelize optimization per target hardware/container using Kubernetes jobs.
• Use cached intermediate artifacts (ONNX) to avoid recompute.
• Parameterize pipelines for dataset slice, calibration data, and quantization config.
• Use reproducible containers and pinned framework versions to avoid drift.
  Trade-offs:
• Conservative gates reduce risk but may block useful optimizations. Allow "experimental" channels for aggressive variants.
• QAT yields better accuracy than post-training quant but costs more compute and human tuning.
• Running full E2E + production-like benchmarks is costly; use stratified sampling: quick unit/regression first, expensive hardware benchmarks only for candidates that pass light gates.

Example gate configuration (concrete):

• Accuracy delta <= -0.5% AND p95 latency <= 50ms AND memory <= 300MB → promote to canary.
• If canary stable for 60 minutes (no SLO breaches, labeled accuracy stable) → full release.
• Else auto-rollback and notify owners.

This pipeline ensures optimized models are validated for correctness and performance, only promoted when meeting SLOs, and supports safe automated rollback with observability and audit trails.

Direct answer: Recursive Feature Elimination (RFE) repeatedly fits a model, ranks features by the model's own importance measure (coefficient magnitude for a linear model, split importance for a tree), removes the weakest, and refits, continuing until a target feature count remains; wrapping this in cross-validation for the SELECTION step (not just the final model evaluation) is what prevents the selected subset from being optimistically biased toward one particular train/validation split.

Structured elaboration:
The core RFE loop: fit the estimator on the current feature set, rank features by the fitted model's own importance signal, drop the lowest-ranked feature (or a batch of step features), and repeat until n_features_to_select remain. Each iteration requires a full model refit, which is what makes RFE a genuine wrapper method (as opposed to a filter, which never touches the model) and also what makes its cost scale with both the number of features and the cost of a single fit.

Cross-validating the selection means running this whole elimination process across multiple folds and confirming the selected subset (or the performance at each subset size) is stable across folds, not just picking whatever RFE happened to select on one arbitrary split, which can otherwise substantially overstate how good the selected subset actually is when it's later evaluated on genuinely held-out data.

Worked example: Verified with a synthetic dataset where only two of six features actually carry signal: implementing RFE with a logistic regression estimator, iteratively removing the feature with the smallest absolute coefficient magnitude at each step, correctly converges on exactly the two true signal features, having eliminated the four pure-noise features across the elimination steps. This confirms the implementation correctly identifies the true underlying support, which is the property you'd want to validate on any RFE implementation before trusting it on real data where the ground truth isn't known.

Trade-offs and pitfalls: RFE's biggest practical limitation is its cost: at $p$ features, a naive RFE that removes one feature per step requires roughly $p$ full model refits, which becomes prohibitively slow for a slow-to-fit model and a large candidate pool; removing a larger batch per step (a bigger step value) trades some selection quality for a large speedup, and is the standard practical compromise once the candidate pool gets into the hundreds or thousands.

Direct answer

To derive an SLO from a business KPI, work backward through the causal chain: identify what the business actually cares about, find the model-controllable signal that's the most direct proxy for it, and set the SLO target using historical data on what level of that signal has historically correlated with acceptable business outcomes.

Structured elaboration

1. Start with the business KPI: for a search-ranking model, this might be "revenue per search session."
2. Trace the causal chain to something measurable in near-real-time: revenue per session is downstream of click-through rate on the top results, which is downstream of ranking quality. CTR is measurable within minutes; revenue attribution often lags (purchases complete later, get refunded, etc.), so CTR is the better SLI candidate: closer to real-time, still tightly correlated with the outcome you actually care about.
3. Set the SLO threshold from historical correlation, not intuition: look at historical periods where CTR dipped by various amounts and check how revenue per session moved in each case; pick an SLO threshold at the CTR level below which you have historical evidence of meaningful revenue impact, not an arbitrary round number.
4. Validate the proxy periodically: the CTR-to-revenue relationship can itself drift (a pricing change, a new monetization model): revisit whether your chosen SLI is still a faithful stand-in for the business KPI on a regular cadence, not just set it once and forget it.

Worked example

Two example SLOs for a search-ranking model: (1) top-3-result CTR stays within 3 percentage points of its trailing 90-day baseline, measured daily: chosen because historical analysis showed CTR drops beyond this range preceded measurable revenue-per-session declines within the following week; (2) p95 time-to-first-result stays under 150ms, because historical A/B tests on THIS product showed conversion drops measurably once latency crosses roughly this threshold, making it a load-bearing, evidence-based number rather than an industry-average guess.

Trade-offs & pitfalls

The trap is picking an SLI that's easy to measure rather than one that's tightly causally linked to the business outcome: a metric like "average prediction confidence" is trivial to compute but has no established relationship to revenue, so an SLO built on it can hold steady while the business actually suffers, or breach constantly while nothing real is wrong. The discipline is validating the SLI-to-KPI link with historical data BEFORE building an SLO and alerting infrastructure around it, not after.