Glossary · Infrastructure & serving
Dynamic Batching
A runtime policy that forms inference batches from queued requests according to compatible shapes, maximum size, priority, and allowed queue delay.
Why it matters
Grouping requests can improve hardware utilization, but waiting for a batch can make latency worse when traffic is sparse or requests differ sharply.
In practice
Set queue-delay and batch limits from measured latency objectives, separate incompatible request shapes, and compare throughput with tail latency at realistic arrival rates.
Common confusion
Dynamic batching assembles batches from queued work. Continuous batching changes membership while autoregressive generation is already running.
Related terms
Sources
Browse the learning paths to see this term in context — every lesson is free to read.