Microsoft AI Engineer - Mid-Level Interview Preparation Guide

Direct answer: Separate chaining stores colliding keys in a linked list (or small array/tree) per bucket; open addressing (linear/quadratic probing, double hashing) stores every key directly in the table itself, probing to the next slot on collision. Both give average-case O(1) get/put/delete under a good hash function and bounded load factor, but an attacker who can choose (or predict) keys that all hash to the same bucket can force every operation to O(n) - a real denial-of-service vector, not just a theoretical curiosity, mitigated by randomized hash seeding at process startup and hash functions resistant to seed-independent collision construction.

Structured elaboration

• Separate chaining: each bucket holds a small collection (commonly a linked list, or - in some modern implementations like Java 8+'s HashMap - a balanced tree once a bucket grows large) of all keys hashing there. Lookup cost is O(1 + chain length); with load factor kept bounded, expected chain length is O(1).
• Open addressing: on collision, probe subsequent slots (linear: next slot; quadratic: quadratically-increasing offsets; double hashing: a second hash function determines the probe sequence) until an empty slot is found. Avoids the pointer-chasing overhead of chaining (better cache locality, since probing stays within the contiguous backing array), but degrades faster as load factor approaches 1 (must keep load factor well below 1, commonly under 0.7, whereas chaining degrades more gracefully).
• The attack: if the hash function is known (or its output is predictable, e.g. a naive non-cryptographic hash with a fixed seed) and an attacker controls input keys (e.g. HTTP form-field names, JSON keys), they can construct a large set of keys that all collide into the same bucket(s), degrading every operation from O(1) to O(n) and creating an algorithmic denial-of-service - a single request with thousands of colliding keys can pin a server's CPU.

Worked example

This is not hypothetical: this exact vulnerability class ("hash flooding") was responsibly disclosed and patched across essentially every major web-application language runtime around 2011-2012, after researchers demonstrated that a small POST request with carefully-crafted colliding form-field names could consume many CPU-seconds parsing what should have been a millisecond-scale request. The fix adopted across languages was randomizing the hash seed per-process at startup (so the attacker can't predict the collision-inducing keys ahead of time without knowing the runtime's random seed) and, in some cases, switching to collision-resistant hash functions (like SipHash) specifically designed to make seed-independent collision construction computationally infeasible even if an attacker can observe hash outputs.

Trade-offs & pitfalls

• "Average-case O(1)" is a claim about TYPICAL inputs, not a security guarantee - any system that hashes attacker-controlled keys needs the worst-case-resistant mitigations (randomized seeding, or a cryptographically-motivated hash), not just a "good enough in practice" hash function.
• Randomized per-process seeding means hash order becomes non-deterministic across process restarts - code that accidentally depends on hash-iteration order (a common latent bug) will surface intermittently once seeding is randomized.
• Open addressing's degradation is sharper (clustering effects can compound near a full table) than chaining's, so open-addressing implementations typically resize more aggressively (lower load-factor threshold) to stay safely away from the cliff.

Approach: split into two delivery channels — fast, low-cost inference-time explanations for users and richer, higher-fidelity offline reports for auditors/data scientists.

Inference-time techniques (low latency, user-friendly)

• Sparse feature attributions: pre-computed or model-provided scores for top-K signals (e.g., “Recommended because you viewed X”) — very fast, minimal compute.
• Model-side attention/feature masks: expose attention weights or learned gating for short explanations when using Transformer-like or attention models.
• Example-based explanations: return 1–3 nearest-neighbor examples from user/item embeddings (cheap with ANN index).
• Counterfactual snippets via rule templates: simple “If you hadn’t clicked Y, we wouldn’t have recommended Z” generated from decision rules.
• Budgeting: limit to top-1–3 explanation items to cap latency and UI complexity.

Offline/reporting techniques (high fidelity)

• SHAP or Integrated Gradients computed on batched data for global and local attributions.
• Surrogate models: train interpretable trees or rule lists to approximate complex model behavior for audits.
• Counterfactual search at scale (more compute) to show minimal changes to alter a recommendation.
• Aggregate metrics: feature importance distributions, fairness and calibration reports, stability analyses (how often explanations change).
• Store full inference traces/logs with seeds for reproducibility.

Engineering tradeoffs

• Latency vs fidelity: real-time SHAP is too slow; use approximations or pre-compute. Trade simpler model outputs or ANN lookups for sub-100ms UX.
• Accuracy vs interpretability: simpler surrogate models/feature-reduction increase interpretability but may misrepresent complex model decisions — mark fidelity scores.
• Model complexity vs explainability: deep models give best accuracy but require heavy offline explainability tooling; incorporate hybrid architecture (complex ranker + interpretable re-ranker) to balance both.
• Cost vs coverage: batch offline compute increases cloud costs but enables regulator-grade reports; mitigate with sampling and incremental updates.
• Privacy/compliance: explanations must avoid leaking other users’ data — prefer aggregated features or synthetic nearest examples.

Operational best practices

• Define SLAs for explanation latency and fidelity; surface fidelity/confidence with each explanation.
• Instrument and log explanations and model inputs to enable audits and rollback.
• User test explanation UX to ensure clarity and reduce cognitive load.

Approach (step-by-step):

1. Identify state space and base cases your memoized recursion uses (all (i,j) values and which return immediately).
2. Extract recurrence: dp(i,j) = f(dp(a,b) for (a,b) in predecessors). List predecessors for each state.
3. Choose iteration order so every state's predecessors are already computed when you compute it. That requires a topological ordering of the implicit DAG of states.
4. Translate recursive calls into nested loops that follow that order, initialize dp with base values, and fill table iteratively.

Safe iteration order (how to determine):

• Build dependency directions. If dp[i][j] depends only on dp[i-1][j] and dp[i][j-1], both predecessors have smaller i or smaller j. A safe order is increasing i and, for each i, increasing j (row-major), or increasing j then i (column-major) — both work because predecessors lie "earlier".
• General rule: sort states so predecessor → successor. For grid-like indices, monotonic increasing loops matching dependency axes are safe.

Concrete example (dp[i][j] depends on dp[i-1][j] and dp[i][j-1]):

• Base: dp[0][], dp[][0] known.
• Loop row-major:

def tabulate(m, n):
    dp = [[0]*(n+1) for _ in range(m+1)]
    # initialize base cases
    for i in range(m+1):
        dp[i][0] = base_i0(i)
    for j in range(n+1):
        dp[0][j] = base_0j(j)
    for i in range(1, m+1):
        for j in range(1, n+1):
            dp[i][j] = combine(dp[i-1][j], dp[i][j-1])
    return dp[m][n]

Pros of conversion:

• Eliminates recursion overhead and stack depth; often faster and more memory-efficient (no memo recursion stack).
• Easier to analyze and to apply memory optimizations (rolling arrays).
• Predictable iteration for parallelization or vectorization.

Cons / caveats:

• Finding correct order can be nontrivial for complex, non-acyclic, or cyclic-like dependencies (need topological sort).
• Tabulation may require allocating the full DP table even when memoization touches a small subset; higher memory for sparse reachable states.
• More boilerplate to initialize many base cases; less intuitive if recurrence branches deeply.

When to prefer each:

• Use tabulation for performance, stack safety, and when many states are reachable. Keep memoization when reachable states are sparse, or order is hard to derive.

Recommendation (short): Adopt a hybrid approach — base on a mature open-source framework (e.g., PyTorch/TensorFlow) for core tooling and model portability, and invest in a lightweight internal mobile-optimized layer and tooling (runtime, ops, profiling, model compression pipelines). This balances velocity, talent reuse, and long-term control for mobile-first needs.

Why:

• Engineering velocity: Open-source core accelerates prototyping, leverages community advances, and reduces duplicate effort. An internal mobile runtime and CI/CD plugins optimize delivery without reimplementing whole stack.
• Ownership model: Internal layer gives product teams ownership of mobile-specific features and deployment policies; core remains community-driven with internal contributors.
• Mobile optimization: Dedicated runtime, quantization, pruning pipelines, hardware abstraction (NNAPI/Metal) and profiling ensure low-latency, low-power models.
• Recruitment/skills: Using mainstream OSS attracts talent; internal extensions provide growth opportunities for engineers to specialize in mobile ML infra.
• Licensing risk: Stick to permissive OSS (BSD/Apache) and legal review; isolate any proprietary modules in closed repos to reduce contamination risk.
• Maintainability: Rely on upstream for low-level improvements; maintain a thin compatibility/ops layer and automated upgrade tests.

12-month roadmap (quarterly):

• Q1 (M0–M3): Requirements, evaluate OSS candidates, define mobile KPIs, pilot two PoCs (edge small models) on OSS + internal runtime prototype. Hire 2 mobile ML infra engineers.
• Q2 (M4–M6): Build internal mobile runtime v0 (quantization, delegate to NNAPI/Metal), model compression pipeline, CI hooks; integrate with OSS training workflow; run 3 product pilots.
• Q3 (M7–M9): Harden runtime (A/B on latency/accuracy), add profiling/observability, enforce licensing scans, developer docs, onboarding program; migrate one flagship product to runtime.
• Q4 (M10–M12): Optimize for power, add auto-tuning/benchmarks, scale to additional products, handoff to platform team with SLAs, open internal RFCs for upstream contributions.

Success metrics (track monthly/quarterly):

• Engineering velocity: time-to-prototype reduced by 30% (baseline), PR-to-production cycle time.
• Mobile performance: median model inference latency down X ms, energy per inference decreased by Y%, model size reduced by Z% with <1% accuracy loss.
• Adoption: number of product teams using runtime (target ≥3 by M9, ≥8 by M12).
• Reliability/maintainability: automated upgrade test pass rate ≥95%, mean time to resolve infra bugs <48h.
• Talent/ops: reduce external contractor reliance by 50% and hire/retain targeted specialists.
• Legal: zero licensing incidents; all third-party components approved.

Trade-offs: This slows full open-source independence and requires internal maintenance costs, but minimizes duplication, retains speed, and ensures mobile performance and legal safety.

Direct answer

Given a fixed compute budget, scaling laws tell you there is a specific compute-optimal balance between model size and dataset size that minimizes loss, and the practical decision procedure is to use the empirically-fit power-law relationship (or a published reference curve like Chinchilla's) to solve for the parameter count and token count that jointly satisfy your compute budget, rather than maximizing model size and training on whatever data happens to be available.

Structured elaboration

The trade-off you're actually deciding. For a fixed compute budget $C$, approximated as $C \approx 6ND$ (roughly six FLOPs per parameter per training token, a standard approximation), increasing model size $N$ while holding $D$ fixed means fewer training tokens per parameter, and increasing $D$ while holding $N$ fixed means training longer on a smaller model. Neither extreme is optimal: a very large model trained on comparatively few tokens is undertrained relative to what its capacity could use, and a very small model trained on enormous data eventually saturates, since its limited capacity cannot extract further gains from additional data.

Estimating a compute-optimal configuration. Empirically-fit scaling-law curves (most influentially the Chinchilla scaling law) give you the specific ratio between $N$ and $D$ that minimizes loss for a given compute budget; using that ratio, you can solve directly for the $(N, D)$ pair, given $C$, that sits on the compute-optimal frontier, rather than guessing.

Practical steps to make the best use of compute. Beyond simply solving the $(N, D)$ equation, curating or augmenting data matters directly: if you don't have enough distinct high-quality tokens to reach the compute-optimal $D$ for your target $N$, you either need to reduce $N$ to match the data you actually have, invest in acquiring or generating more high-quality data, or accept training past the single-epoch compute-optimal point by repeating data for additional epochs (with diminishing, but not zero, returns).

Worked example

Suppose the compute budget is $C = 6\times10^{23}$ FLOPs. Using $C \approx 6ND$, a compute-optimal allocation close to this budget is roughly $N = 25\times10^{9}$ parameters (25B) and $D = 4\times10^{12}$ tokens (4 trillion): $6 \times 2.5\times10^{10} \times 4\times10^{12} = 6\times10^{23}$ FLOPs, which I verified numerically matches the budget exactly. If your team only has 500 billion distinct high-quality tokens available, roughly one-eighth of the 4 trillion the compute-optimal 25B-parameter model wants, you have three real options: reduce the model size to one whose compute-optimal token requirement matches the 500B tokens you actually have (solving $6N \times 5\times10^{11} = 6\times10^{23}$ gives $N \approx 2\times10^{11}$, an 8x LARGER model, which is now compute-optimal for that smaller dataset at the SAME total compute, since the fixed budget gets reallocated toward more parameters rather than more tokens); train the 25B model past one epoch by repeating the 500B tokens roughly 8 times, accepting diminishing returns from data repetition; or invest engineering effort in expanding the usable dataset (deduplication of a larger raw corpus, licensing more data, synthetic data generation) rather than compromising either the model size or accepting repeated-epoch training.

Trade-offs & pitfalls

The compute-optimal point from scaling laws minimizes TRAINING loss per unit of TRAINING compute, but it says nothing about inference cost, which recurs on every single production request for the life of the model, unlike training cost, which is paid once. A team planning to serve the model at very high volume for years often deliberately trains a SMALLER-than-compute-optimal model on MORE tokens than the pure training-optimal point calls for (intentionally "overtraining" relative to the scaling-law optimum), because the resulting savings in inference latency and cost over the model's serving lifetime outweigh the extra one-time training cost of processing more tokens on a smaller model. Treating the scaling-law compute-optimal point as the automatic final answer, without separately weighing this inference-cost dimension, is the most common practical mistake here.

Direct answer
An LSTM cell adds three gates and a separate cell state on top of a plain recurrent unit, letting the network learn WHEN to remember, overwrite, or expose information, which is what lets gradients (and information) survive across many more timesteps than a vanilla RNN can manage.

Structured elaboration
Given input $x_t$, previous hidden state $h_{t-1}$, and previous cell state $c_{t-1}$:
$f_t = \sigma(W_f[h_{t-1},x_t]+b_f)\quad\text{(forget gate: how much of }c_{t-1}\text{ to keep)}$
$i_t = \sigma(W_i[h_{t-1},x_t]+b_i)\quad\text{(input gate: how much new content to write)}$
$\tilde c_t = \tanh(W_c[h_{t-1},x_t]+b_c)\quad\text{(candidate cell content)}$
$c_t = f_t \odot c_{t-1} + i_t \odot \tilde c_t\quad\text{(cell state update)}$
$o_t = \sigma(W_o[h_{t-1},x_t]+b_o)\quad\text{(output gate: how much of }c_t\text{ to expose)}$
$h_t = o_t \odot \tanh(c_t)$
Sigmoid gates produce values in $(0,1)$, acting as soft, learnable on/off switches; tanh keeps candidate content and the exposed hidden state in $(-1,1)$, centered around zero, which helps gradient-based optimization.

Why the gating structure preserves long-range dependencies: the cell-state update is ADDITIVE ($f_t\odot c_{t-1} + i_t\odot\tilde c_t$), not a repeated multiplicative transform of the kind that causes vanishing gradients in a vanilla RNN's hidden-state recurrence. When the forget gate is close to 1, the gradient with respect to $c_{t-1}$ passes through nearly unattenuated (multiplied by approximately 1, not by a small, saturating activation derivative), so error signal can flow back across many timesteps largely intact, as long as the network has learned to keep the relevant forget gates open along the way.

Worked example
Suppose at some timestep $f_t=0.95$ (mostly remember), $i_t=0.1$ (write a little new content), $c_{t-1}=2.0$, $\tilde c_t = 0.5$: $c_t = 0.95(2.0)+0.1(0.5)=1.9+0.05=1.95$, close to the previous cell value, as expected when the forget gate is near 1. Contrast with $f_t=0.1$ (mostly forget), same other values: $c_t = 0.1(2.0)+0.1(0.5)=0.2+0.05=0.25$, showing the cell state genuinely discarding most of its prior content when the network has learned that forgetting is appropriate at this step.

Trade-offs & pitfalls
A common oversimplification is describing the cell state as "immune" to vanishing gradients; it is much MORE resistant than a vanilla RNN's hidden state, but the forget gate itself is still a learned, saturating sigmoid, so if training pushes forget gates toward the extremes (stuck near 0) the additive benefit is undermined. A practical mitigation often used is initializing the forget-gate bias to a small positive value (e.g. +1), biasing the network toward remembering by default early in training, which tends to make long-range dependencies easier to learn from the start rather than relying on training to discover that remembering is useful.

Direct answer
Strict dependency pinning (exact versions, lockfiles) maximizes build reproducibility and stability at the cost of manually managing security-patch updates; flexible version ranges get security patches automatically but risk an unreviewed, unexpected dependency update silently changing behavior; the right policy is strict pinning as the default, with an automated, TESTED update process (not manual, ad hoc bumps) so security patches still flow through without sacrificing reproducibility.

Structured elaboration

• Reproducibility: exact pinned versions (a lockfile capturing the FULL resolved dependency tree, not just top-level ranges) guarantee that a build today and the same build next month use IDENTICAL dependency code, which matters enormously for ML specifically, since a subtly different version of a numerical library can change model behavior in ways that are hard to detect without exact reproducibility as a baseline to compare against.
• Security patches: a flexible range (numpy>=1.20,<2.0) picks up patch releases automatically on the next install, including security fixes, without any explicit action; strict pinning means a security patch requires an explicit, deliberate version bump, which can lag if the process isn't automated and monitored.
• Build stability: flexible ranges risk a NEW patch release (even a supposedly 'safe' patch bump) introducing an unexpected behavior change or a new bug, breaking a build that was working fine yesterday for no code change of your own; strict pinning eliminates this specific risk entirely, at the cost of the manual-update burden above.
• A policy that gets both: pin exactly by default (lockfile-based), but run an AUTOMATED dependency-update bot (Dependabot, Renovate, or similar) that opens a PR for each available update, which then runs through the FULL test suite (including, for ML, a golden-set accuracy/performance regression check per the companion CI survivor) before a human merges it; this gets the security-patch velocity of automatic updates while keeping every single update explicitly reviewed and tested before it's live, rather than either silently picking up an update (flexible ranges) or manually, irregularly bumping versions (unautomated strict pinning).

Worked example
A project pins torch==2.1.0 exactly in its lockfile; a security patch (torch==2.1.1) is released; an automated bot opens a PR bumping the pin, which triggers the full CI suite including the golden-set model-evaluation gate; the PR shows the new version passes all tests with no accuracy regression, and a human reviewer merges it within a day of the patch's release, achieving both reproducibility (every commit has an exact, known dependency set) and reasonably fast security-patch adoption (not manual, ad hoc, and easy to neglect).

Trade-offs and pitfalls
An automated update bot that opens PRs but that NOBODY actually reviews and merges in a timely way provides none of the security benefit while still adding review-queue noise; the policy only works if merging dependency-update PRs is treated as routine, prioritized work, not something that piles up indefinitely, which is as much a team-process commitment as a tooling choice.

Separately: reproducibility across dev workstation, CI, and a multi-GPU cluster
Making an ML experiment reproducible across environments requires controlling every source of randomness AND every source of environment drift, not just pinning code:

• Seeding every source of randomness: set the seed for Python's own random, numpy, and the ML framework's own RNG, which are all SEPARATE generators; seeding only one leaves the others uncontrolled. On a multi-GPU/multi-worker setup, seed each worker deterministically (for example base_seed + worker_rank) rather than letting each worker seed itself from system entropy, which would make even a single-node-vs-cluster comparison non-reproducible.
• Deterministic-ops flags: enable the framework's deterministic-algorithms mode (for example torch.use_deterministic_algorithms(True), torch.backends.cudnn.deterministic = True), since some GPU kernels have a faster but non-deterministic default implementation that produces a different, though equally valid, numeric result run to run even with every seed fixed.
• Containerization: run dev, CI, and the cluster from the SAME container image, built from the same lockfile and Dockerfile described above, so a library-version or OS-level difference between where you develop and where the cluster actually trains isn't a hidden variable.
• A reproducibility checklist/test: run the SAME short training config (a handful of steps, small enough to be fast) twice, once on a dev workstation and once in CI/on the cluster, and assert the resulting loss values (or a checkpoint checksum) match within a defined, documented numeric tolerance; a divergence beyond that tolerance fails the check and should block trusting a full training run's results until the source of the drift (an unseeded RNG, a differing deterministic-ops flag, a genuine environment mismatch) is identified.

Worked example: a CI check runs a 50-step training config with a fixed seed on both a CPU-only dev machine and a GPU CI runner, comparing the resulting loss curve; a difference beyond the documented tolerance (accounting for expected CPU-versus-GPU floating-point variation) fails the check, catching an accidentally-unseeded data-shuffling step that was silently making every run non-reproducible even on the SAME hardware, before it wastes a multi-day cluster run whose results can't be trusted or reproduced.

Direct answer

I'd structure the three months as a small number of month-scale milestones, each with concrete evidence I'm actually on track, and I'd bias the early weeks toward habits, how I verify information, who actually knows what, how work really gets reviewed, over a rigid task list, since those habits compound and a task list rarely survives contact with how things actually work.

Structured elaboration

• Month one is about orientation habits, not output. I focus on the meta-skills that determine how fast the whole ramp goes: how to verify what I'm told here, who actually has the answers versus who's just available, and how work really gets reviewed and shipped. I also pick one small but real piece of work, not a throwaway exercise, small enough to be safe but real enough to teach me the actual constraints, and finish it.
• Month two expands scope with less hand-holding, and I deliberately pick a task that stretches a specific gap month one exposed, rather than repeating something month one already proved I could do.
• Month three takes something closer to end-to-end with minimal supervision, and functions as the real check on whether the earlier ramp actually took, not just whether I felt more comfortable.
• Track progress against visible evidence each month, not a feeling. A shipped piece of real work, a question I can now answer without help, a review I no longer need: these are checkable in a way "I feel more settled" isn't.
• Keep running notes on what I'm learning as I go, mainly for myself: writing it down forces me to notice what I actually understand versus what I only think I understand, and it happens to save me from re-deriving the same answer a second time later.
• Hold the longer arc in view. The point of a genuinely good first-ninety-days plan isn't just fitting into the new team, it's building toward what I'll be trusted with next, so I pick milestones that show growth, not just that I've reached the floor of the new role.

Worked example

Moving from a general security role into an application-security specialty I hadn't worked in directly before, I spent the first two weeks less on formal training material and more on habits: sitting in on a few real code reviews to see how security issues actually got raised and resolved here, and figuring out which two colleagues actually knew the history behind our trickiest existing systems. My first real piece of work was reviewing one moderate-risk change end to end, small enough that a mistake was recoverable, but real enough to teach me the team's actual review norms rather than the documented ones. By month two, I took on a task that specifically stretched a gap month one had exposed: I hadn't yet had to reason about a vulnerability class that came up more often here than in my old role, so I deliberately picked a task involving that. By month three, I led a review independently that would have needed a second pair of eyes back in month one, and used that as the actual evidence the ramp had worked, not just a feeling of familiarity. I kept a short running document of what I was learning throughout, which turned out useful a few months later when a similar issue came up and I could look back at my own notes instead of re-figuring it out from scratch.

Trade-offs and pitfalls

A plan that's all reading and passive orientation with no real work in the loop tends to feel productive without actually testing anything. The opposite mistake, front-loading too much scope before the meta-skills like who to ask and how review works are in place, tends to produce avoidable mistakes early that damage trust. And judging yourself only by how comfortable you feel, rather than by concrete evidence like a piece of finished work or a question you can now answer alone, is an easy way to think you're on track when you're not.

Requirements & constraints: define target metrics (e.g., deployment-latency budget, min recall), dataset size, and which object classes/conditions matter (small/occluded/day/night). Lock these before experiments.

1. Data splitting

• Create a single randomized split into Train / Val / Test (e.g., 60/20/20) stratified by class, camera/source, object size, and scene to preserve distributional factors.
• Hold the Test set strictly for final evaluation (never used for tuning). Save split indices and hash of files.

2. Tuning strategy (rigorous, fair)

• Use nested tuning: inner loop uses k-fold cross-validation or repeated validation on the Train+Val combined portion (e.g., 4-fold on Train for hyperparam search, evaluate on Val to pick best). Alternatively: use Train for fitting, Val for hyperparameter selection; repeat the whole process with multiple random seeds.
• Ensure identical hyperparameter search budget and search space for both algorithms (same number of trials, same parameter ranges). Use Bayesian/Random search with fixed trial count.

3. Evaluation protocol & seeds

• For final evaluation, train top-k models per algorithm (from tuning) with multiple random seeds (>=5) and evaluate on Test. Report mean and 95% CI.
• Fix all RNG seeds (data shuffling, augmentation, framework, cudnn deterministic flags) and record them. Log environment details (library versions, GPU/CPU, driver).

4. Implementation parity

• Use identical preprocessing, augmentation pipeline, anchor settings, NMS parameters, image resizing, and class mappings. If architecture imposes differences, document and minimize.

5. Metrics & compute reporting

• Primary detection metrics: mAP (COCO-style: AP@[.50:.95]), plus AP50 and AP75, per-class AP, AP small/medium/large, recall@k.
• Operational metrics: inference latency (median, p95) on target hardware, throughput (images/sec), peak GPU memory, FLOPs, model size.
• Report metric distributions (per-image AP) and bootstrap CIs.

6. Statistical testing

• Use paired, non-parametric tests that respect paired predictions:
  • Matched-pair bootstrap on AP: resample images, compute delta(AP) distribution, derive CI and p-value.
  • Alternatively, use Wilcoxon signed-rank on per-image AP differences (if distribution not normal).
• For categorical detection-level comparisons (detected vs missed per ground-truth), McNemar's test or permutation tests on matched pairs.
• Correct for multiple comparisons (Bonferroni or Benjamini-Hochberg) when testing many metrics or classes.
• Declare significance threshold (e.g., alpha=0.05) and report effect sizes.

7. Reproducibility & reporting

• Provide full experiment manifest: code repo+commit, container image or environment.yml, random seeds, split files, hyperparam logs, hardware, runtime per epoch, and trained checkpoints.
• Publish evaluation script that computes metrics deterministically.
• Run ablations: sensivity to seed, augmentation, and input size.

Result interpretation

• Prefer practical significance (latency/size tradeoffs) alongside statistical significance. Show confidence intervals and whether differences meet deployment thresholds.

This protocol ensures fair comparison, repeatability, and statistically defensible conclusions.

Direct answer
State the decision and the immediate next steps out loud before the meeting ends, then send a short written follow-up within the hour that restates the same thing, so there's both an in-the-room confirmation and a durable record that matches it.

Structured elaboration

1. Confirm verbally before people leave the room (or call). In the last minute or two, say "so to confirm, we've decided X, and the next steps are Y owned by Z by Thursday, does that match everyone's understanding?" This catches a misalignment while everyone who can correct it is still present.
2. Watch for silence versus agreement. Nobody objecting isn't the same as everyone actively agreeing; a direct question ("does that match?") is more reliable than just pausing and moving on if no one immediately speaks up.
3. Send the written follow-up promptly, ideally within the hour, restating the same decision and action items. The verbal confirmation and the written one should say the same thing; if they don't, that's usually a sign the verbal confirmation was rushed or unclear.
4. Keep the written version short and scannable, matching the same content as the verbal confirmation rather than adding new information the room didn't actually agree to.
5. Flag anything genuinely still unresolved, in both the verbal check and the written follow-up, rather than letting an unresolved point quietly look settled just because the meeting ended.

Worked example
Verbal, at the end of the meeting: "So to confirm: we're going with the phased rollout, Sam owns the migration plan by next Friday, and we're holding off on the customer announcement until that's done. Does that match what everyone heard?"

Written follow-up sent the same hour: "Recap from today: decided on the phased rollout. Sam: migration plan due next Friday. Customer announcement is on hold until the migration plan is ready. Shout if this doesn't match what you remember."

The two versions state the identical decision and owner, and the written version explicitly invites correction rather than assuming silence means agreement.

Trade-offs and pitfalls

• Skipping the verbal confirmation and only sending a written recap later means any misunderstanding surfaces after people have already left and possibly acted on their own interpretation.
• Skipping the written follow-up and only confirming verbally means anyone who wasn't in the room, or who forgets, has no record to check against.
• A written recap that silently adds detail beyond what was verbally confirmed can create a new source of disagreement; keep the two consistent, and if you realize something needs adding, flag it explicitly as new rather than folding it in unannounced.