Microsoft AI Engineer - Mid-Level Interview Preparation Guide

Direct answer: Separate chaining stores colliding keys in a linked list (or small array/tree) per bucket; open addressing (linear/quadratic probing, double hashing) stores every key directly in the table itself, probing to the next slot on collision. Both give average-case O(1) get/put/delete under a good hash function and bounded load factor, but an attacker who can choose (or predict) keys that all hash to the same bucket can force every operation to O(n) - a real denial-of-service vector, not just a theoretical curiosity, mitigated by randomized hash seeding at process startup and hash functions resistant to seed-independent collision construction.

Structured elaboration

• Separate chaining: each bucket holds a small collection (commonly a linked list, or - in some modern implementations like Java 8+'s HashMap - a balanced tree once a bucket grows large) of all keys hashing there. Lookup cost is O(1 + chain length); with load factor kept bounded, expected chain length is O(1).
• Open addressing: on collision, probe subsequent slots (linear: next slot; quadratic: quadratically-increasing offsets; double hashing: a second hash function determines the probe sequence) until an empty slot is found. Avoids the pointer-chasing overhead of chaining (better cache locality, since probing stays within the contiguous backing array), but degrades faster as load factor approaches 1 (must keep load factor well below 1, commonly under 0.7, whereas chaining degrades more gracefully).
• The attack: if the hash function is known (or its output is predictable, e.g. a naive non-cryptographic hash with a fixed seed) and an attacker controls input keys (e.g. HTTP form-field names, JSON keys), they can construct a large set of keys that all collide into the same bucket(s), degrading every operation from O(1) to O(n) and creating an algorithmic denial-of-service - a single request with thousands of colliding keys can pin a server's CPU.

Worked example

This is not hypothetical: this exact vulnerability class ("hash flooding") was responsibly disclosed and patched across essentially every major web-application language runtime around 2011-2012, after researchers demonstrated that a small POST request with carefully-crafted colliding form-field names could consume many CPU-seconds parsing what should have been a millisecond-scale request. The fix adopted across languages was randomizing the hash seed per-process at startup (so the attacker can't predict the collision-inducing keys ahead of time without knowing the runtime's random seed) and, in some cases, switching to collision-resistant hash functions (like SipHash) specifically designed to make seed-independent collision construction computationally infeasible even if an attacker can observe hash outputs.

Trade-offs & pitfalls

• "Average-case O(1)" is a claim about TYPICAL inputs, not a security guarantee - any system that hashes attacker-controlled keys needs the worst-case-resistant mitigations (randomized seeding, or a cryptographically-motivated hash), not just a "good enough in practice" hash function.
• Randomized per-process seeding means hash order becomes non-deterministic across process restarts - code that accidentally depends on hash-iteration order (a common latent bug) will surface intermittently once seeding is randomized.
• Open addressing's degradation is sharper (clustering effects can compound near a full table) than chaining's, so open-addressing implementations typically resize more aggressively (lower load-factor threshold) to stay safely away from the cliff.

Approach: split into two delivery channels — fast, low-cost inference-time explanations for users and richer, higher-fidelity offline reports for auditors/data scientists.

Inference-time techniques (low latency, user-friendly)

• Sparse feature attributions: pre-computed or model-provided scores for top-K signals (e.g., “Recommended because you viewed X”) — very fast, minimal compute.
• Model-side attention/feature masks: expose attention weights or learned gating for short explanations when using Transformer-like or attention models.
• Example-based explanations: return 1–3 nearest-neighbor examples from user/item embeddings (cheap with ANN index).
• Counterfactual snippets via rule templates: simple “If you hadn’t clicked Y, we wouldn’t have recommended Z” generated from decision rules.
• Budgeting: limit to top-1–3 explanation items to cap latency and UI complexity.

Offline/reporting techniques (high fidelity)

• SHAP or Integrated Gradients computed on batched data for global and local attributions.
• Surrogate models: train interpretable trees or rule lists to approximate complex model behavior for audits.
• Counterfactual search at scale (more compute) to show minimal changes to alter a recommendation.
• Aggregate metrics: feature importance distributions, fairness and calibration reports, stability analyses (how often explanations change).
• Store full inference traces/logs with seeds for reproducibility.

Engineering tradeoffs

• Latency vs fidelity: real-time SHAP is too slow; use approximations or pre-compute. Trade simpler model outputs or ANN lookups for sub-100ms UX.
• Accuracy vs interpretability: simpler surrogate models/feature-reduction increase interpretability but may misrepresent complex model decisions — mark fidelity scores.
• Model complexity vs explainability: deep models give best accuracy but require heavy offline explainability tooling; incorporate hybrid architecture (complex ranker + interpretable re-ranker) to balance both.
• Cost vs coverage: batch offline compute increases cloud costs but enables regulator-grade reports; mitigate with sampling and incremental updates.
• Privacy/compliance: explanations must avoid leaking other users’ data — prefer aggregated features or synthetic nearest examples.

Operational best practices

• Define SLAs for explanation latency and fidelity; surface fidelity/confidence with each explanation.
• Instrument and log explanations and model inputs to enable audits and rollback.
• User test explanation UX to ensure clarity and reduce cognitive load.

Direct answer: Selecting from a huge candidate pool under a compute budget is a staged-filtering problem, not a single-algorithm problem: cheap, fast filters eliminate the bulk of obviously-useless candidates first, and progressively more expensive methods are only applied to the much smaller survivor set, so you never pay wrapper-level cost on the full candidate pool.

Structured elaboration:
A practical staged pipeline:

1. Free/near-free filters first: drop near-zero-variance or quasi-constant columns (a column where one value dominates almost every row carries almost no signal), and drop anything failing a basic missingness or schema sanity check. This alone often eliminates a large fraction of a huge automatically-generated candidate pool at negligible cost.
2. Cheap univariate scoring (mutual information, correlation, a fast statistical test) on what remains, run in parallel/distributed since it's embarrassingly parallel per feature.
3. Redundancy removal: cluster features by pairwise correlation and keep one representative per cluster, since two near-duplicate features waste selection budget without adding signal.
4. Sampled or lightweight model-based importance on the reduced set (a fast model on a data sample, not the full expensive model on full data) to get a final ranked shortlist.
5. Only at this much-reduced scale would a genuinely expensive wrapper search (or the production model itself) be used to make the final cut.

Preserving interpretability and reproducibility means logging which stage eliminated each candidate and why, so the final set can be explained and the pipeline re-run deterministically, not just treated as a black box that spits out a list.

Worked example: Consider a specific cost-constrained variant of the same problem: a model already meets its accuracy SLA (service-level agreement) but feature computation is too expensive. The search here is inverted: instead of maximizing accuracy under a feature-count budget, you're minimizing feature-computation cost subject to NOT breaking the accuracy SLA, using greedy ablation (drop the least-important feature, retrain, check the SLA still holds, repeat) or importance-based pruning from the bottom up, with a held-out set to make sure you're not overfitting the pruning search itself to one data split.

Trade-offs and pitfalls: The variance-threshold-style fast filters have a real failure mode worth knowing: a rare-but-highly-predictive one-hot column (present in only 0.1% of rows but perfectly separating a rare, high-value class) looks like "near-zero variance" to a naive threshold and gets dropped, even though it may be the single most valuable feature in the set. A blind variance cutoff should always be checked against target-correlation, not applied in isolation.

The idea in plain words

Every learnable number in a PyTorch model is a parameter, and each parameter has a flag, requires_grad. When the flag is False, backpropagation (the computation of how the loss changes with each parameter) does not compute a gradient for it, so an optimiser (the object, such as SGD, that changes parameters using their gradients) leaves it alone: it is frozen. Transfer learning with a frozen backbone is therefore three moves: (1) load the pretrained network, (2) set requires_grad = False on every existing parameter, (3) replace the last fully connected layer (fc, the layer that maps the 2048 features ResNet-50 computes for an image to class scores) with a fresh nn.Linear. A freshly created layer has requires_grad = True by default, so the order matters: freeze first, replace second.

The code you would write

This is the whole answer to the question as one block. It needs torchvision installed, and the first call downloads the pretrained weights, so it needs network access; the same three moves are run below on a stand-in that needs none. The argument that selects pretrained weights has changed between torchvision releases (an older pretrained=True form and a newer weights= form), so check the documentation of the version you have.

import torch
from torch import nn
from torchvision.models import resnet50

num_classes = 5
model = resnet50(weights="DEFAULT")          # 1. load pretrained ImageNet weights
for p in model.parameters():                 # 2. freeze every parameter that exists now
    p.requires_grad = False
model.fc = nn.Linear(model.fc.in_features, num_classes)   # 3. new head, created after freezing, so it trains

# confirm only the new head requires gradients: list every trainable tensor (name, shape)
for name, p in model.named_parameters():     # named_parameters yields (name, tensor) pairs
    if p.requires_grad:
        print(name, tuple(p.shape))          # prints fc.weight (5, 2048) and fc.bias (5,)

The same three moves, run

To run them without torchvision and without downloading weights, the program below first builds a ResNet-50 from scratch with the same module names torchvision uses (conv1, bn1, layer1 to layer4, fc), randomly initialised. It is a stand-in for the pretrained model, and freeze_and_replace is exactly the code you would apply to the torchvision model. How to read the stand-in:

• Bottleneck is one residual block: three convolutions, each followed by a BatchNorm layer, and a skip connection that adds the block's input to its output (idt). When the shape changes, downsample is a 1 by 1 convolution that reshapes the input so the addition is possible.
• ResNet50 stacks 3, 4, 6 and 3 of those blocks into layer1 to layer4. Each of these four groups is a residual stage; deeper stages hold more channels and smaller feature maps. Then it averages each feature map to one number (2048 of them) and applies fc.
• Below the stand-in, freeze_and_replace and trainable are the parts the question asks for. Everything after them is checking: the assert raises an error if its condition is false, so a silent mistake becomes a loud one.

import torch
from torch import nn
torch.set_num_threads(1)

# ---- stand-in: a ResNet-50 built here with the same module names as torchvision's (conv1, bn1, layer1..layer4, fc),
# ---- randomly initialised, so that this example runs with no downloads.
class Bottleneck(nn.Module):
    def __init__(self, cin, planes, stride):
        super().__init__()
        cout = planes * 4
        self.conv1 = nn.Conv2d(cin, planes, 1, bias=False); self.bn1 = nn.BatchNorm2d(planes)
        self.conv2 = nn.Conv2d(planes, planes, 3, stride, 1, bias=False); self.bn2 = nn.BatchNorm2d(planes)
        self.conv3 = nn.Conv2d(planes, cout, 1, bias=False); self.bn3 = nn.BatchNorm2d(cout)
        self.relu = nn.ReLU(inplace=True)
        self.downsample = None
        if stride != 1 or cin != cout:
            self.downsample = nn.Sequential(nn.Conv2d(cin, cout, 1, stride, bias=False), nn.BatchNorm2d(cout))
    def forward(self, x):
        idt = x if self.downsample is None else self.downsample(x)
        out = self.relu(self.bn1(self.conv1(x)))
        out = self.relu(self.bn2(self.conv2(out)))
        return self.relu(self.bn3(self.conv3(out)) + idt)

class ResNet50(nn.Module):
    def __init__(self, num_classes=1000):
        super().__init__()
        self.conv1 = nn.Conv2d(3, 64, 7, 2, 3, bias=False); self.bn1 = nn.BatchNorm2d(64)
        self.relu = nn.ReLU(inplace=True); self.maxpool = nn.MaxPool2d(3, 2, 1)
        cin, layers = 64, []
        for planes, n, stride in ((64, 3, 1), (128, 4, 2), (256, 6, 2), (512, 3, 2)):
            blocks = []
            for i in range(n):
                blocks.append(Bottleneck(cin, planes, stride if i == 0 else 1)); cin = planes * 4
            layers.append(nn.Sequential(*blocks))
        self.layer1, self.layer2, self.layer3, self.layer4 = layers
        self.avgpool = nn.AdaptiveAvgPool2d(1)
        self.fc = nn.Linear(2048, num_classes)
    def forward(self, x):
        x = self.maxpool(self.relu(self.bn1(self.conv1(x))))
        x = self.layer4(self.layer3(self.layer2(self.layer1(x))))
        return self.fc(torch.flatten(self.avgpool(x), 1))

# ---- the code the question asks for: works unchanged on any model whose classifier is the attribute `fc` ----------
def freeze_and_replace(model, num_classes):
    for p in model.parameters():                       # 1. freeze everything that exists now
        p.requires_grad = False
    model.fc = nn.Linear(model.fc.in_features, num_classes)   # 2. new layer: created AFTER freezing, so it trains
    return model

def trainable(model):
    return [(n, p.numel()) for n, p in model.named_parameters() if p.requires_grad]

num_classes = 5
model = ResNet50()                                     # with torchvision: model = resnet50(weights=...) instead
print("total parameters before:", sum(p.numel() for p in model.parameters()))
model = freeze_and_replace(model, num_classes)
print("trainable parameter tensors:", trainable(model))
n_train = sum(p.numel() for p in model.parameters() if p.requires_grad)
n_total = sum(p.numel() for p in model.parameters())
print(f"trainable {n_train} of {n_total} ({100 * n_train / n_total:.3f} percent); expected fc size 2048*{num_classes}+{num_classes} = {2048 * num_classes + num_classes}")
assert [n for n, _ in trainable(model)] == ["fc.weight", "fc.bias"]

# the same check applied to the wrong order (replace first, then freeze everything) must fail
wrong = ResNet50(); wrong.fc = nn.Linear(2048, num_classes)
for p in wrong.parameters():
    p.requires_grad = False
print("wrong order -> trainable tensors:", len(trainable(wrong)), "(the new head would never learn)")

# the optimiser gets only the trainable parameters
opt = torch.optim.SGD([p for p in model.parameters() if p.requires_grad], lr=1e-2, momentum=0.9)
print("optimiser parameter tensors:", sum(len(g["params"]) for g in opt.param_groups))

# requires_grad=False does not stop BatchNorm from updating its running statistics in train mode
x = torch.randn(2, 3, 64, 64)
before = model.bn1.running_mean.clone()
with torch.no_grad():
    batch_mean = model.conv1(x).mean(dim=(0, 2, 3))     # what bn1 sees: the per-channel mean of this batch
model.train(); model(x)
print("frozen bn1 running_mean changed in train():", not torch.equal(before, model.bn1.running_mean))
expected = (1 - model.bn1.momentum) * before + model.bn1.momentum * batch_mean   # momentum is 0.1 by default
print("matches 0.9 * old + 0.1 * batch mean:", torch.allclose(expected, model.bn1.running_mean, atol=1e-6))
def train_mode_head_only(m):
    m.eval(); m.fc.train()                             # frozen backbone (incl. BatchNorm) in eval mode
before = model.bn1.running_mean.clone()
train_mode_head_only(model); model(x)
print("after backbone.eval(): changed =", not torch.equal(before, model.bn1.running_mean))

Output:

total parameters before: 25557032
trainable parameter tensors: [('fc.weight', 10240), ('fc.bias', 5)]
trainable 10245 of 23518277 (0.044 percent); expected fc size 2048*5+5 = 10245
wrong order -> trainable tensors: 0 (the new head would never learn)
optimiser parameter tensors: 2
frozen bn1 running_mean changed in train(): True
matches 0.9 * old + 0.1 * batch mean: True
after backbone.eval(): changed = False

Reading the output

• The 1000-class stand-in has 25,557,032 parameters. After the move, the only tensors with requires_grad=True are fc.weight (shape $5 \times 2048 = 10240$ values) and fc.bias (5 values), 10,245 in total, which is the 2048*5+5 the program prints. That is 0.044 percent of the 23,518,277 parameters in the modified model (the 1000-class head, 2,049,000 values, is gone).
• The assert compares the list of trainable names to exactly ["fc.weight", "fc.bias"], so it fails if anything else is trainable or the head is missing. The "wrong order" line shows it can fail: replacing the head first and then freezing everything leaves 0 trainable tensors, and the model would silently never learn.
• The optimiser receives only the trainable parameters (2 tensors). Passing all parameters also works in PyTorch: SGD and Adam skip any parameter whose .grad is None, as is the case for a frozen parameter that has never had a gradient, and they allocate no momentum buffers (the running average of past gradients that momentum=0.9 keeps per parameter) for it. Filtering is still the better habit because it states the intent and is safe against one trap: a parameter that was trained earlier and then frozen still holds its old .grad, and an optimiser given that parameter would keep updating it with that stale gradient. Filtering, or clearing the gradients with optimizer.zero_grad(set_to_none=True), avoids this.

The part that is easy to miss: BatchNorm

ResNet-50 has BatchNorm layers. A BatchNorm layer normalises each channel using the mean and variance of the current batch while training, and using stored running statistics (a running mean and running variance, averages accumulated over past batches) at inference. Those stored numbers are buffers: tensors that live inside a module and are saved with the model, but that the optimiser never touches because they are not parameters. Freezing weights therefore does not stop them. In train() mode, every forward pass updates them by the rule

$\text{running\_mean} \leftarrow (1-m)\cdot\text{running\_mean} + m\cdot\text{batch\_mean}, \qquad m = 0.1 \text{ by default.}$

The program checks this directly: bn1.running_mean after one training-mode forward pass equals 0.9 times its old value plus 0.1 times the per-channel mean of the batch that bn1 saw (the output of conv1), and the line printed True confirms it. No gradient is involved, so requires_grad=False cannot prevent it. Once the backbone is put in eval() mode (where BatchNorm uses its stored statistics and does not update them) and only the head is in train mode, the statistics stay fixed. For a frozen backbone, call eval() on it every time you switch to training, because model.train() flips everything back. If you want the frozen network to adapt its statistics to your data, that is a deliberate choice, not a side effect.

Practical notes

• With a fully frozen backbone you can run the images through it once under torch.no_grad() (a context that tells PyTorch not to record the operations needed for gradients, which saves memory), cache the 2048-dimensional features, and train the head on the cached features; that saves memory and most of the time.
• Pass the optimiser a larger learning rate for the new head than you would for fine-tuning the backbone; if you later unfreeze layer4, give it a smaller one in its own parameter group.
• Use the same preprocessing (resize, scaling, per-channel mean and standard deviation) that the pretrained weights were trained with.
• The code above relies on the head being called fc, which is true of ResNet models; other architectures name it differently (classifier, head), so look at print(model).

Clarify SLOs & workload
I’d confirm traffic shape (QPS (queries per second), burstiness), input sizes, and correctness/latency trade-offs (accuracy vs quantization). Aim: p50 20ms, p99 200ms for a heavy model.

High-level architecture

• Ingress → lightweight front proxy (Envoy) → request router (per-model) → inference tier (GPU/CPU/TPU pools) → response.
• Observability: latency/queue metrics, per-stage tracing.

Serving frameworks

• Consider Triton for multi-backend, batching, model versioning; TorchServe for PyTorch pipelines; BentoML for flexible pipelines. Triton’s dynamic batching and model repository help meet SLOs.

Batching / dynamic batching

• Use small fixed batch sizes for p50; enable dynamic batching with max latency budget (e.g., 5–10ms wait) to increase throughput without exceeding p99.
• Adaptive batching: adjust batch timeout based on current tail latency and queue length.

Autoscaling & warm pools

• Combine reactive autoscaling (CPU/GPU util, queue length) with predictive scaling from traffic forecasts.
• Maintain a warm pool of pre-loaded model instances (or containers) to avoid cold starts. Keep a small number of GPU-backed instances idle.

Cold-start mitigation

• Lazy-weighted routing to warm instances; lightweight fallback to smaller/quantized model for quick responses when heavy model not ready.

Model partitioning & compilation

• Use model compilation (ONNX Runtime with TensorRT/XLA) and quantization (FP16/INT8) to reduce latency.
• Consider model sharding for extremely large models (sequence parallelism or pipeline parallelism) but weigh added network latency.

Hardware

• Use GPUs for heavy models; choose A100/H100 or inference-optimized instances with Tensor Cores. For low-cost/less heavy models, CPU with AVX512 and ONNX may suffice.

Caching

• Cache recent/identical inputs and partial outputs (embedding cache) at front proxy. Use TTL and cache hit-miss metrics.

Throughput vs tail-latency trade-offs

• Larger batches maximize throughput but increase tail latency. Use hybrid strategy: benchmark batch-size vs p99, enforce max wait time, and prefer smaller batches under bursty/latency-sensitive traffic.

Monitoring & iterative tuning

• Continuously monitor p50/p99, per-batch latency, queue times. Run canary experiments for quantized/compiled models and measure accuracy-latency trade-offs.

Worked capacity example (from a folded near-duplicate): with single-request GPU compute time of 25ms at batch size 1 and a CPU fallback of 80ms/request, one GPU replica saturates at 1/0.025s = 40 requests/sec, and one CPU replica at 1/0.080s = 12.5 requests/sec. Targeting 1000 requests/sec (queries per second) while keeping utilization at roughly 60% (so bursts do not blow through the p99 target) needs about 1000/(400.6) ≈ 42 GPU replicas, versus roughly 1000/(12.50.6) ≈ 133 CPU replicas for the same throughput - a concrete illustration of why the GPU path is chosen for this SLO (service-level objective) even though a single GPU instance costs more than a single CPU instance: replica count, not raw compute, is what needs to be provisioned and paid for at this scale.

Direct answer
An LSTM cell adds three gates and a separate cell state on top of a plain recurrent unit, letting the network learn WHEN to remember, overwrite, or expose information, which is what lets gradients (and information) survive across many more timesteps than a vanilla RNN can manage.

Structured elaboration
Given input $x_t$, previous hidden state $h_{t-1}$, and previous cell state $c_{t-1}$:
$f_t = \sigma(W_f[h_{t-1},x_t]+b_f)\quad\text{(forget gate: how much of }c_{t-1}\text{ to keep)}$
$i_t = \sigma(W_i[h_{t-1},x_t]+b_i)\quad\text{(input gate: how much new content to write)}$
$\tilde c_t = \tanh(W_c[h_{t-1},x_t]+b_c)\quad\text{(candidate cell content)}$
$c_t = f_t \odot c_{t-1} + i_t \odot \tilde c_t\quad\text{(cell state update)}$
$o_t = \sigma(W_o[h_{t-1},x_t]+b_o)\quad\text{(output gate: how much of }c_t\text{ to expose)}$
$h_t = o_t \odot \tanh(c_t)$
Sigmoid gates produce values in $(0,1)$, acting as soft, learnable on/off switches; tanh keeps candidate content and the exposed hidden state in $(-1,1)$, centered around zero, which helps gradient-based optimization.

Why the gating structure preserves long-range dependencies: the cell-state update is ADDITIVE ($f_t\odot c_{t-1} + i_t\odot\tilde c_t$), not a repeated multiplicative transform of the kind that causes vanishing gradients in a vanilla RNN's hidden-state recurrence. When the forget gate is close to 1, the gradient with respect to $c_{t-1}$ passes through nearly unattenuated (multiplied by approximately 1, not by a small, saturating activation derivative), so error signal can flow back across many timesteps largely intact, as long as the network has learned to keep the relevant forget gates open along the way.

Worked example
Suppose at some timestep $f_t=0.95$ (mostly remember), $i_t=0.1$ (write a little new content), $c_{t-1}=2.0$, $\tilde c_t = 0.5$: $c_t = 0.95(2.0)+0.1(0.5)=1.9+0.05=1.95$, close to the previous cell value, as expected when the forget gate is near 1. Contrast with $f_t=0.1$ (mostly forget), same other values: $c_t = 0.1(2.0)+0.1(0.5)=0.2+0.05=0.25$, showing the cell state genuinely discarding most of its prior content when the network has learned that forgetting is appropriate at this step.

Trade-offs & pitfalls
A common oversimplification is describing the cell state as "immune" to vanishing gradients; it is much MORE resistant than a vanilla RNN's hidden state, but the forget gate itself is still a learned, saturating sigmoid, so if training pushes forget gates toward the extremes (stuck near 0) the additive benefit is undermined. A practical mitigation often used is initializing the forget-gate bias to a small positive value (e.g. +1), biasing the network toward remembering by default early in training, which tends to make long-range dependencies easier to learn from the start rather than relying on training to discover that remembering is useful.

Why attention is O(n^2)

For a sequence of n tokens, each head has queries Q and keys K, each n by d_k (d_k is the size of one head). The score matrix is Q times K transposed, shape n by n: every query is compared with every key, and each comparison is a dot product costing d_k multiply-adds. So the time is n times n times d_k, i.e. O(n^2 d). The softmax normalizes each row over all n keys, so a row cannot be finished until all n scores exist, and the weighted sum with V (n by d_k) costs another n^2 d_k. If the n by n matrix is stored, memory is O(n^2) per head, and backpropagation needs it again. Doubling the length quadruples both.

The main ways to cut the cost, with their trade-offs

The rows run from exact to increasingly approximate; the order to try them in is under "How to choose". Orthogonal but often confused: multi-query attention (all heads share one set of keys and values) and grouped-query attention (groups of heads share one) shrink the key-value cache that decoding reads, not the n^2 score computation.

The online softmax on four numbers

Softmax needs the sum over all keys, which seems to force all scores to exist at once. The trick is to keep a running maximum m, a running denominator l and a running numerator, and to rescale the old totals by exp(old m - new m) whenever a later tile raises the maximum; subtracting the maximum is the usual guard against overflow, and the rescaling keeps every earlier tile consistent with it, which is why the result is exact. Here is one query with four given scores and values, processed two keys at a time.

import numpy as np

# One query, four keys, one number per value. Scores are given directly.
scores = np.array([1.0, 3.0, 2.0, 5.0])
values = np.array([10.0, 20.0, 30.0, 40.0])

p = np.exp(scores - scores.max())
print("full softmax-weighted value:", round(float((p / p.sum()) @ values), 4))

# Online version: two tiles of two keys; keep a running max m, denominator l and numerator acc.
m, l, acc = -np.inf, 0.0, 0.0
for tile in (slice(0, 2), slice(2, 4)):
    s, v = scores[tile], values[tile]
    m_new = max(m, s.max())
    scale = np.exp(m - m_new)              # shrinks what was accumulated under the old max
    e = np.exp(s - m_new)
    l = l * scale + e.sum()
    acc = acc * scale + e @ v
    m = m_new
    print(f"after keys {tile.start}-{tile.stop - 1}: max m = {m:.1f}, scale = {scale:.4f}, denominator l = {l:.4f}, numerator = {acc:.4f}")
print("tiled result acc / l:", round(float(acc / l), 4))

Output:

full softmax-weighted value: 36.8806
after keys 0-1: max m = 3.0, scale = 0.0000, denominator l = 1.1353, numerator = 21.3534
after keys 2-3: max m = 5.0, scale = 0.1353, denominator l = 1.2034, numerator = 44.3835
tiled result acc / l: 36.8806

Reading it: after the first tile the maximum is 3 and there was nothing to rescale (the scale printed for it is exp(-inf) = 0, which just wipes the empty start). The second tile contains a score of 5, which raises the maximum from 3 to 5, so the old denominator and numerator are multiplied by exp(3 - 5) = 0.1353 before the new terms are added; every term is again measured against the new maximum. That rescaling is why the final acc / l is exactly the full softmax-weighted value, while only two scores were held at a time.

Code: one exact and two approximate methods

On a synthetic 64-token input with d_k = 16, the program (1) implements tiled attention with an online softmax and checks it equals full attention, (2) implements a window mask, and (3) implements linear attention with the feature map elu(x) + 1, checking that the reordered computation equals the quadratic-order one. It reports relative error (the length of the difference between an approximation's output and full attention's output, divided by the length of full attention's output: 0 means identical, 1 means the error is as large as the output itself, and above 1 means the approximation is further from the right answer than an all-zero output would be) on two inputs: a random one with no locality, and one built as a random walk, so neighbouring tokens are similar and nearby keys get large scores. "Attention mass inside window" is the average share of each row's softmax weight that falls inside the window. A last block repeats the comparison on 20 other seeds and prints the median error, so no conclusion rests on one draw.

import torch
import torch.nn.functional as F

torch.set_num_threads(1)

torch.manual_seed(0)
n, d, blk = 64, 16, 16
q, k, v = (torch.randn(n, d) for _ in range(3))

full = F.softmax(q @ k.T / d ** 0.5, dim=-1) @ v

# 1. Exact: tiled attention with an online softmax (the FlashAttention idea), never builds n x n.
def tiled(q, k, v, blk):
    out = torch.zeros_like(q)
    m = torch.full((n, 1), float("-inf"))     # running max of scores
    l = torch.zeros(n, 1)                      # running softmax denominator
    for s in range(0, n, blk):
        sc = q @ k[s:s + blk].T / d ** 0.5     # only an n x blk tile at a time
        m_new = torch.maximum(m, sc.amax(dim=-1, keepdim=True))
        p = torch.exp(sc - m_new)
        scale = torch.exp(m - m_new)
        l = l * scale + p.sum(dim=-1, keepdim=True)
        out = out * scale + p @ v[s:s + blk]
        m = m_new
    return out / l
print("tiled == full (exact):", torch.allclose(tiled(q, k, v, blk), full, atol=1e-5))

# 2. Approximate: sliding window of w keys on each side.
def window(q, k, v, w):
    idx = torch.arange(n)
    allowed = (idx[:, None] - idx[None, :]).abs() <= w
    sc = (q @ k.T / d ** 0.5).masked_fill(~allowed, float("-inf"))
    return F.softmax(sc, dim=-1) @ v, int(allowed.sum())

# 3. Approximate: kernelized linear attention with feature map phi(x) = elu(x) + 1.
def linear(q, k, v):
    pq, pk = F.elu(q) + 1, F.elu(k) + 1
    kv = pk.T @ v                      # (d, d), cost O(n d^2)
    z = pq @ pk.sum(dim=0, keepdim=True).T
    return (pq @ kv) / z
pq, pk = F.elu(q) + 1, F.elu(k) + 1
quad = ((pq @ pk.T) @ v) / (pq @ pk.T).sum(dim=-1, keepdim=True)
print("linear (reordered) == linear (quadratic order):", torch.allclose(linear(q, k, v), quad, atol=1e-5))

def rel_err(a, ref):
    return ((a - ref).norm() / ref.norm()).item()

def report(name, q, k, v):
    ref = F.softmax(q @ k.T / d ** 0.5, dim=-1) @ v
    print(f"--- {name} input (full attention scores computed: {n * n})")
    print(f"tiled     : scores per tile {n * blk}, relative error {rel_err(tiled(q, k, v, blk), ref):.4f}")
    idx = torch.arange(n)
    p = F.softmax(q @ k.T / d ** 0.5, dim=-1)
    for w in (4, 16):
        out, cnt = window(q, k, v, w)
        mass = (p * ((idx[:, None] - idx[None, :]).abs() <= w)).sum(-1).mean().item()
        print(f"window w={w:<2}: scores computed {cnt}, attention mass inside window {mass:.2f}, relative error {rel_err(out, ref):.4f}")
    print(f"linear    : state size {d * d} numbers, relative error {rel_err(linear(q, k, v), ref):.4f}")

report("random", q, k, v)
x = torch.randn(n, d).cumsum(0)               # a random walk: neighbouring tokens are similar
x = x / x.norm(dim=-1, keepdim=True) * 6.0
report("local", x, x, v)

# Same comparison over 20 seeds (median relative error), so no conclusion rests on one draw.
def trial(seed):
    g = torch.Generator().manual_seed(seed)
    qq, kk, vv = (torch.randn(n, d, generator=g) for _ in range(3))
    xx = torch.randn(n, d, generator=g).cumsum(0)
    xx = xx / xx.norm(dim=-1, keepdim=True) * 6.0
    row = []
    for a, b in ((qq, kk), (xx, xx)):
        ref = F.softmax(a @ b.T / d ** 0.5, dim=-1) @ vv
        row += [rel_err(window(a, b, vv, w)[0], ref) for w in (4, 16)] + [rel_err(linear(a, b, vv), ref)]
    return row
rows = torch.tensor([trial(s) for s in range(20)])
med = rows.median(dim=0).values
print("--- median relative error over 20 seeds")
print(f"random: window w=4 {med[0]:.2f}, window w=16 {med[1]:.2f}, linear {med[2]:.2f}")
print(f"local : window w=4 {med[3]:.2f}, window w=16 {med[4]:.2f}, linear {med[5]:.2f}")
print(f"local window w=16 beats random window w=16 in {(rows[:, 4] < rows[:, 1]).sum().item()} of 20 seeds")

Output:

tiled == full (exact): True
linear (reordered) == linear (quadratic order): True
--- random input (full attention scores computed: 4096)
tiled     : scores per tile 1024, relative error 0.0000
window w=4 : scores computed 556, attention mass inside window 0.14, relative error 1.8768
window w=16: scores computed 1840, attention mass inside window 0.45, relative error 1.0044
linear    : state size 256 numbers, relative error 0.6690
--- local input (full attention scores computed: 4096)
tiled     : scores per tile 1024, relative error 0.0000
window w=4 : scores computed 556, attention mass inside window 0.54, relative error 0.7136
window w=16: scores computed 1840, attention mass inside window 0.93, relative error 0.1056
linear    : state size 256 numbers, relative error 0.7717
--- median relative error over 20 seeds
random: window w=4 2.18, window w=16 1.11, linear 0.74
local : window w=4 0.84, window w=16 0.21, linear 0.82
local window w=16 beats random window w=16 in 20 of 20 seeds

Reading it:

• The tiled method matches full attention (error 0.0000) while holding only n by 16 scores at a time (1,024 instead of 4,096).
• A window helps when the attention is local. On the random input a window is a poor stand-in: with w = 4 it captures only 0.14 of the weight and the error is 1.8768 on this draw; the median over 20 seeds is 2.18 for w = 4 and 1.11 for w = 16. On the local input w = 16 captures 0.93 of the weight and the error falls to 0.1056 on this draw (median 0.21 over 20 seeds, and lower than the random-input w = 16 error in all 20), while computing 1,840 of the 4,096 scores.
• Errors above 1 on the random input come from a simple effect: full attention averages many unrelated random value vectors, which shrinks its output, while a narrow window averages only a few, which gives a longer vector that points elsewhere. Real text is usually far more local than random numbers, so the local input is the nearer picture, and only your own data can say how local your task is.
• Linear attention here has error 0.6690 on the random input and 0.7717 on the local one (medians over 20 seeds: 0.74 and 0.82): softmax and the feature map produce different weight shapes. These are untrained random projections on synthetic inputs, so the numbers measure how far a cheaper layer's output is from softmax attention's, not task quality. In practice a model with linear attention is trained as it is, and the fair comparison is its accuracy on your task against the full-attention model. What the numbers do show is that each approximation has an assumption about the attention pattern, not a ranking of methods. Measure on your task.

How to choose

Commit to this order: first use the exact tiled kernel and fix the memory problem without any accuracy loss. If FLOPs or length still do not fit, use a window plus global tokens when the task is mostly local (documents, code), and retrieval when relevant information is sparse. Consider linear or low-rank layers only if you can validate the quality loss on your own evaluation. What flips it: tasks that need exact copy or lookup from far away push you back toward full attention with more hardware, or retrieval.

Common mistakes

• Calling FlashAttention an approximation; it is exact and removes memory, not FLOPs.
• Expecting a window of w to give a receptive field (how far back a token's output can depend on, directly or through layers) of w; stacking L layers lets information travel about L times w.
• Ignoring that generation uses a growing cache: sparse patterns must also bound the cache.

Running the code

Save the program as demo.py and run python demo.py. The output above was produced with Python 3.12, numpy 2.1.3 and torch 2.5.1 (CPU). The seed is fixed in the program.

Direct answer

I'd structure the three months as a small number of month-scale milestones, each with concrete evidence I'm actually on track, and I'd bias the early weeks toward habits, how I verify information, who actually knows what, how work really gets reviewed, over a rigid task list, since those habits compound and a task list rarely survives contact with how things actually work.

Structured elaboration

• Month one is about orientation habits, not output. I focus on the meta-skills that determine how fast the whole ramp goes: how to verify what I'm told here, who actually has the answers versus who's just available, and how work really gets reviewed and shipped. I also pick one small but real piece of work, not a throwaway exercise, small enough to be safe but real enough to teach me the actual constraints, and finish it.
• Month two expands scope with less hand-holding, and I deliberately pick a task that stretches a specific gap month one exposed, rather than repeating something month one already proved I could do.
• Month three takes something closer to end-to-end with minimal supervision, and functions as the real check on whether the earlier ramp actually took, not just whether I felt more comfortable.
• Track progress against visible evidence each month, not a feeling. A shipped piece of real work, a question I can now answer without help, a review I no longer need: these are checkable in a way "I feel more settled" isn't.
• Keep running notes on what I'm learning as I go, mainly for myself: writing it down forces me to notice what I actually understand versus what I only think I understand, and it happens to save me from re-deriving the same answer a second time later.
• Hold the longer arc in view. The point of a genuinely good first-ninety-days plan isn't just fitting into the new team, it's building toward what I'll be trusted with next, so I pick milestones that show growth, not just that I've reached the floor of the new role.

Worked example

Moving from a general security role into an application-security specialty I hadn't worked in directly before, I spent the first two weeks less on formal training material and more on habits: sitting in on a few real code reviews to see how security issues actually got raised and resolved here, and figuring out which two colleagues actually knew the history behind our trickiest existing systems. My first real piece of work was reviewing one moderate-risk change end to end, small enough that a mistake was recoverable, but real enough to teach me the team's actual review norms rather than the documented ones. By month two, I took on a task that specifically stretched a gap month one had exposed: I hadn't yet had to reason about a vulnerability class that came up more often here than in my old role, so I deliberately picked a task involving that. By month three, I led a review independently that would have needed a second pair of eyes back in month one, and used that as the actual evidence the ramp had worked, not just a feeling of familiarity. I kept a short running document of what I was learning throughout, which turned out useful a few months later when a similar issue came up and I could look back at my own notes instead of re-figuring it out from scratch.

Trade-offs and pitfalls

A plan that's all reading and passive orientation with no real work in the loop tends to feel productive without actually testing anything. The opposite mistake, front-loading too much scope before the meta-skills like who to ask and how review works are in place, tends to produce avoidable mistakes early that damage trust. And judging yourself only by how comfortable you feel, rather than by concrete evidence like a piece of finished work or a question you can now answer alone, is an easy way to think you're on track when you're not.

Direct answer

With limited labeled data to validate a new fraud model, a small canary rollout is almost always the safer starting point over a full production launch, since it lets you gather real production evidence before committing to broad exposure, as long as the canary's blast radius is genuinely small and monitored closely.

Structured elaboration

1. Canary approach. Ship to a small percentage of live traffic, closely monitor both model-level metrics (precision, recall against whatever confirmed labels arrive) and business-level guardrails (customer complaints, transaction friction), and expand gradually as confidence grows.
2. Full-rollout approach. Faster to reach full business impact if the model works, but any undetected problem (a mis-calibrated threshold, a data issue that only shows up at scale) affects the whole business at once, which is a much larger risk given how little validation data exists.
3. Metrics to track during the canary. Both the model's own precision and recall as confirmed labels trickle in, and the business cost signals (false-positive complaint volume, estimated fraud losses in the canary segment versus a comparable control segment) that would catch a real problem quickly.
4. Risk of each path. The canary's main risk is a slower path to full value and the extra engineering work of running two populations differently; the full rollout's main risk is a costly, hard-to-reverse mistake at scale if the model behaves unexpectedly outside of what the limited validation data could show.

Worked example

Starting the canary at a small percentage of transactions, holding a comparable control segment on the existing system, and setting explicit graduation criteria (a defined number of days with no anomalous complaint spikes and precision holding within an acceptable range against confirmed labels) turns "when do we go full rollout" from a gut call into a pre-agreed, evidence-based decision.

Trade-offs and pitfalls

The common mistake is running a canary without a real control group or explicit graduation criteria, which leaves the team unable to say confidently when it's safe to expand. The other risk is treating the canary phase as a formality and expanding on a fixed timeline regardless of what the early data shows, which defeats the purpose of testing with limited validation data in the first place.

Direct answer

Mentoring someone you rarely see combines deliberate async artifacts with narrow, well-prepared live time, but the shape of that changes further when the gap isn't just distance or time zone. Culture, hands-on skills that need physical access, and group settings each introduce their own specific friction that a generic "be more async" answer misses.

Baseline async toolkit

• Recorded walkthroughs instead of live explanations, so the reasoning survives the time-zone gap.
• Written runbooks and checklists instead of verbal context that only exists once.
• Threaded async status updates instead of live stand-ups.
• Infrequent, scheduled live time used for judgment calls and open questions, not status updates that could have been written down.

Culture, not just the clock

Mentoring across different cultural norms changes communication and feedback style, not only cadence. Direct, pointed critique that reads as normal in one context can read as harsh or face-threatening in another, and in some cultures a mentee may not push back or admit confusion even when they have it, because that would read as disrespectful. Adjustments: ask the mentee to restate feedback back in their own words to check it landed as intended, prefer written feedback they can process privately over being put on the spot verbally, and actively invite disagreement rather than assuming silence means agreement.

When the skill is physical or hands-on

If the mentee can't access the same lab, hardware, or physical setup the mentor has, a video call alone doesn't transfer the skill, no matter how much conversation happens. Workarounds: remote access into shared real hardware or a virtual lab where one exists, high-fidelity recordings of the technique from multiple angles, and having the mentee submit their own attempt as recorded evidence (video, logs, output) for asynchronous review as a substitute for watching over their shoulder. The honest answer names this as a real limitation rather than pretending remote conversation is equivalent.

Facilitating a remote group, not just a 1:1

Running a remote group critique is a different skill from managing 1:1 async cadence. It needs explicit turn-taking since silence reads very differently on a call than in a room, a written artifact everyone reviews beforehand so live time goes to discussion instead of a first read, and deliberately calling on quieter participants, since remote settings tend to amplify whoever is already most comfortable speaking up.

Worked example

Mentoring someone with only a narrow daily overlap window involved recorded walkthroughs for anything routine, and reserving the one live weekly slot purely for judgment calls that didn't compress well into writing. Early feedback delivered directly and pointedly in that format landed harder than intended, since it read as more severe without the in-person context to soften it. Shifting to written feedback they could sit with, followed by an open question in the next live slot, got a much more honest back-and-forth than direct verbal critique had.

Trade-offs and pitfalls

A common mistake is treating "remote" as one problem solved by one toolkit, more meetings or better docs, regardless of what's actually causing the friction. The stronger answer separates distance, time zone, culture, physical access, and group dynamics, and picks a fix matched to the actual friction rather than a generic one. Assuming a video call is a full substitute for hands-on access is a specific version of this mistake worth naming explicitly.