Direct answer
A CI/CD checklist for a model artifact with limited reproducibility should gate on: the training run being reproducible from a pinned configuration, a basic performance check against a known baseline, dependency and security scanning, and a deployment smoke test, with clear ownership assigned to each stage so reproducibility isn't left as an implicit, unowned expectation.
Structured elaboration
- Reproducibility check: re-run training from the exact pinned configuration (data snapshot reference, hyperparameters, random seeds, library versions) and verify the resulting model's key metrics match within an agreed tolerance; if they don't, that's itself the signal that something in the training pipeline isn't actually pinned the way it's assumed to be (an unpinned dependency, an unseeded random process, a data source that silently changed). Owner: the ML engineer or team responsible for the training pipeline.
- Basic performance check: evaluate the newly trained artifact against a fixed, held-out validation set and compare against the currently deployed model's baseline metrics, blocking promotion if performance regresses beyond an agreed threshold. Owner: same team, as part of the training pipeline's own CI.
- Security/dependency scanning: scan the model-serving container and its dependencies the same way you would any other deployable artifact, since a model-serving stack has the same supply-chain risk surface as any other service. Owner: platform/security team's existing scanning pipeline, applied to this artifact like any other.
- Deployment smoke test: after deploying to staging, run a small set of known inputs through the live serving endpoint and check the outputs match expectations (or match the training-time evaluation output for those same inputs), catching serving-specific bugs (a preprocessing mismatch between training and serving, a version skew) that training-time checks alone wouldn't catch. Owner: whoever owns the serving infrastructure, coordinating with the ML team on what "expected" means for the check.
Worked example
A recommendation model's pipeline pins its training data snapshot, random seeds, and library versions in a config file checked into the same repository as the training code. CI re-runs training from that exact config and checks the resulting model's offline evaluation metric is within 0.5% of the previous run's value (catching accidental non-reproducibility); a basic performance gate then compares against the currently-deployed model's baseline on a held-out set, blocking promotion if it regresses; a container scan checks the serving image's dependencies; and a deployment smoke test sends 20 known user profiles through the staged serving endpoint and confirms the recommendations match what the training-time evaluation produced for those same profiles, catching a preprocessing mismatch that once caused a silent serving-time bug.
Trade-offs & pitfalls
The most common gap is skipping the reproducibility check specifically, because it feels like extra work when the model "already trained fine once"; without it, a config that looks pinned but silently isn't (an unseeded library call, a data source that changed underneath the pinned reference) can produce a materially different model on the next run with nobody noticing until a much later, harder-to-diagnose failure.
