An ML-enabled recommendation feature has a fundamentally different failure-mode landscape than a typical CRUD service: bugs can be silent and statistical (a model that's technically running but subtly wrong) rather than loud and immediate (a crash or an exception), which changes what each pyramid level needs to actually verify.
Unit tests: data transformation and feature-engineering code
Test individual feature-engineering functions the same way you'd unit-test any pure logic: given a known input, does a specific transformation (a normalization, a categorical encoding, a time-windowed aggregation) produce the exact expected output. This is the one level that behaves like a traditional software unit test, since feature code, written well, is ordinary deterministic logic with no model or statistics involved yet.
Integration and component tests: model scoring and feature stores
Test that the SERVING path is wired correctly: given a known feature vector, does the model-scoring service return a response in the expected shape and value range (not whether the model's prediction is "good," just whether the scoring pipeline correctly invokes the model and returns its output); and does the feature store correctly serve the same feature values online that were computed offline for the same entity (catching training-serving skew, a notoriously common and hard-to-detect ML integration bug, at the integration level rather than discovering it only once it silently degrades live recommendation quality).
Model-validation tests: a level with no equivalent in a typical software pyramid
This is the category that has no direct analog in a conventional application: data-quality checks (are incoming features within expected ranges and free of an unexpected spike in nulls), drift detection (has the input feature distribution shifted meaningfully from what the model was trained on), and performance-threshold checks (does the model's offline evaluation metric, such as precision at k, remain above an agreed floor before this version is allowed to ship). These exist because a model can be "working" in the sense of running without errors while being statistically wrong in a way none of the earlier levels can see, since none of them evaluate prediction QUALITY, only prediction PLUMBING.
End-to-end tests: offline evaluation and online A/B validation
Offline evaluation runs the full candidate model against a held-out historical dataset and confirms its aggregate performance meets the bar before it's eligible to ship at all; online A/B validation then runs the candidate against a real slice of live traffic, comparing engagement or business metrics against the current production model, since offline metrics do not always predict real-world impact. Both are necessary and neither replaces the other: offline evaluation is cheap and fast but can miss real-world effects offline data doesn't capture; online validation is the only test that measures real user impact, but is slow (requires real traffic and enough time to reach significance) and carries real business risk during the test itself.
Acceptance criteria and monitoring alarms
Acceptance criteria should combine an offline metric floor (the model must not underperform a defined precision/recall or ranking-quality threshold on the held-out set) with an online guardrail (the model must not regress a defined business metric, such as engagement rate, beyond an agreed tolerance during its A/B test) before full rollout. Post-launch monitoring alarms should track live prediction-distribution drift (are the model's outputs statistically similar to what was seen during evaluation) and live feature-input drift (has the real-world input distribution moved away from what the model was trained and validated against), since both are leading indicators of a silent quality regression well before it would show up in a lagging business metric.
Trade-offs and pitfalls
The pitfall specific to ML systems is treating a green integration-test suite as sufficient confidence, the way it often is for conventional software, when a scoring pipeline can be perfectly wired and still serve statistically degraded predictions; the model-validation layer exists precisely to catch what "the code runs correctly" cannot.
