Direct answer
Producing reproducible performance benchmarks in CI across different hardware means controlling for environmental variance as tightly as the CI environment allows, running enough repetitions with proper warm-up to separate a real signal from measurement noise, and setting pass/fail thresholds as a statistical band around expected variance rather than a single hard number that any natural fluctuation would trip.
Structured elaboration
- Controlling environmental variance: use dedicated (not shared, noisy-neighbor-prone) test machines where possible; disable non-essential background services and processes that could introduce contention; account for thermal effects (a benchmark run on a machine that's been under sustained load may throttle differently than one starting cold), which for engine/game workloads specifically can materially skew frame-time results if not controlled for.
- Handling inherent non-determinism: a game engine's frame timing has real, expected run-to-run variance even on identical hardware running identical code; the benchmark needs to treat some variance as normal rather than trying to eliminate it entirely, which isn't achievable.
- Representative workloads: choose benchmark scenarios that reflect real gameplay conditions (a representative level or scene, not just a synthetic micro-benchmark), so a regression caught here reflects something a real player would actually notice.
- Warm-up runs: discard results from the first several runs (JIT warm-up, cache warming, asset streaming settling) before recording the numbers that count, since early runs are systematically different from steady-state and including them would bias the comparison.
- Statistical sampling and thresholds: run enough repetitions to characterize the natural variance (not just a single run per configuration), and set a pass/fail threshold as a statistical band (e.g. flag only if the new result falls outside a confidence interval derived from historical variance) rather than a hard fixed number, since a hard number will either flag normal noise as a regression or fail to catch a real one depending on how tightly it's set relative to actual variance.
Worked example
A game studio's CI benchmark runs each build against a fixed representative gameplay scene on dedicated benchmark hardware (background services disabled, thermal preconditioning via a fixed warm-up period before measurement), discarding the first 3 of 10 runs as warm-up and computing the mean and standard deviation of the remaining 7. A build is flagged as a performance regression only if its mean frame time exceeds the historical baseline's mean by more than 2 standard deviations of that baseline's own observed variance, rather than a fixed millisecond threshold that would either be too loose for a stable metric or too tight for a naturally noisier one.
Trade-offs & pitfalls
The most common mistake is a fixed absolute threshold picked without reference to the metric's actual observed variance, which either produces frequent false-positive regressions on a naturally noisy metric or misses a real regression on an unusually stable one; deriving the threshold from the metric's own historical variance, and explicitly discarding warm-up runs, are what actually make this kind of benchmark trustworthy enough to gate on.
