Batching

Batching: Grouping several requests together so a model processes them in one pass, improving hardware use and overall throughput.

Browse the full glossary →