Direct answer
Batch inference requests onto a GPU by accumulating events until either a batch-size threshold or a maximum wait time is reached, whichever comes first, so you get GPU-efficient batch sizes without letting any single event wait indefinitely during a quiet period.
Structured elaboration
GPU inference is generally far more efficient per-item at larger batch sizes (better hardware utilization), which pulls toward waiting to accumulate a big batch. But a pure size-based trigger means, during low-traffic periods, an event could wait a very long time for the batch to fill, which is unacceptable if there's a latency SLA (service-level agreement) on the prediction. The standard resolution is a dual trigger: flush whichever of "batch size N reached" or "time window T elapsed since the oldest buffered item" happens first, bounding worst-case added latency to T regardless of traffic volume.
Worked example
For a scoring pipeline needing predictions within roughly 200 milliseconds end to end, you might set batch size to 64 and the time window to 50 milliseconds: during high traffic, batches fill to 64 quickly and inference runs at full GPU efficiency; during a lull, even a half-full batch flushes after 50 milliseconds rather than waiting indefinitely, keeping worst-case added latency bounded and predictable regardless of how bursty the traffic is.
Trade-offs and pitfalls
A batch size set too small forfeits GPU efficiency gains (defeating the purpose of batching at all); a window set too long violates the latency SLA during quiet periods even if the size threshold is fine. The added latency this introduces is a genuine, board-line design cost that has to be weighed against the throughput and cost efficiency GPU batching buys you; a design that ignores the latency side of that trade will look fine in a load test and fail an actual production SLA the first time traffic is uneven.
