LocalAI's parakeet-cpp backend now coalesces concurrent audio transcription requests into a single batched call, raising GPU utilization under load.
Enables dynamic batching for concurrent audio transcription requests to increase GPU throughput.
Operators running LocalAI for speech-to-text under concurrent load can get up to 1.84x higher transcription throughput on GPU by enabling dynamic batching, reducing the need for more GPUs to handle the same request volume. The feature requires raising the `batch_max_size` parameter and depends on a specific version of the parakeet.cpp library. Continued monitoring is needed to see how batching behaves with different model sizes, GPU architectures, and under extreme concurrency to ensure stability and predictable latency.