Performance Trade-offs & Optimization Strategy Questions

Approach

Order techniques roughly by reversibility and how directly they attack the actual binding constraint, memory or power, starting with the cheapest, least accuracy-risky option and validating accuracy loss at each step before moving to the next, more aggressive one. This is a computer vision (CV) model, meaning software that interprets images, being deployed on an edge device, hardware near the data source, often battery-powered and resource-constrained, rather than in the cloud.

The order I'd reach for

1. Quantization first, reducing the numeric precision used to store the model's weights, for example from 32-bit to 8-bit numbers, since it typically gives a large size and speed win, often several times smaller, for comparatively small accuracy loss, and is cheap and reversible to test.
2. Pruning next if more headroom is needed, removing less-important weights from the model, which carries more accuracy risk but is tunable, prune a little or a lot.
3. Only then, architecture changes or knowledge distillation, training a smaller model to mimic a larger one, since these need more engineering time and a longer validation cycle.

What tells you you've gone too far

Define the accuracy floor before touching the model, not after, for example "we can tolerate up to a 2 percentage point drop from the current 91% baseline," set against what the deployment actually needs. Treat each technique as a checkpoint: measure accuracy on a validation set matching real deployment conditions after each step, and stop as soon as you cross the floor, rather than stacking every technique and checking once at the end, which makes it hard to know which change caused the damage.

Worked example (illustrative numbers)

Baseline model: 91% accuracy, 24 megabytes, which does not fit an 8 megabyte device memory budget. Quantization to 8-bit cuts size roughly four times, to about 6 megabytes, with accuracy dropping to about 90.3%, a small loss, already meeting the memory budget. If more headroom is still needed, pruning 30% of the smallest-magnitude weights might cut size further to about 4.2 megabytes but accuracy to 89.0%, right at the 2-point floor. I would stop there rather than pruning further, since the next increment is where the deployment's tolerance would actually be crossed.

Power is a separate constraint from memory

Quantization and pruning both tend to reduce compute and so power draw, but if power is the truly binding constraint, for example a battery-powered sensor, I would also consider reducing input resolution or how often the model runs, since that trades latency or coverage rather than raw model size, a different lever than the accuracy-based ones above.