Concurrent TTS Performance Observation (VoxCPM2 + vLLM-Omni)

[mishradibyajyoti]

Hi Team,

I tested the /api/tts_direct_stream endpoint under concurrent load and observed the following behavior.

Environment:

Server: AWS EC2 (L40S GPU)
Serving Stack: vLLM-Omni v0.20.0
Endpoint: /api/tts_direct_stream
Client: Python (requests + ThreadPoolExecutor) for concurrent load testing

Model Serving Command:

vllm-omni serve $model_path
--omni
--served-model-name tts-service
--host 0.0.0.0
--port 8000
--enforce-eager
--dtype bfloat16

Test Results:

1 concurrent request
Latency: ~0.59s
TTFB: ~0.11s

10 concurrent requests
Average latency: ~3.08s
Maximum latency: ~4.46s
TTFB increases from ~0.49s for the first few requests to ~3.53s for the last requests.

It appears that requests are being processed in batches rather than all 10 starting simultaneously. The first few requests begin processing immediately, while the remaining requests wait before inference starts, resulting in higher TTFB and overall latency.

Could you please confirm:

-Is there a concurrency limit (or maximum number of parallel inference slots) configured in the current vLLM-Omni/VoxCPM2 setup?
-Is this expected behavior for the current deployment, or is there any server-side configuration that can improve concurrent throughput?
-What's the minimum GPU for getting concurrency.
-Are there recommended settings (e.g., scheduler, batching, max_num_seqs, or deployment configuration) to handle higher concurrent TTS requests?

Thanks.

[a710128]

Hi @mishradibyajyoti, thanks for the detailed benchmark — this is a really useful report! 🙏

Short answer: what you're seeing is expected continuous-batching behavior plus a concurrency cap in the default VoxCPM2 recipe, not a hidden serialization bug. Here are answers to each of your questions.

1. Is there a concurrency limit? → Yes, it's max_num_seqs

The number of requests processed in parallel is controlled by max_num_seqs (the max number of sequences in a batch). The official VoxCPM2 deploy recipe sets this conservatively:

max_num_seqs: 8
max_num_batched_tokens: 4096

So with 10 concurrent requests, the first 8 enter the current batch and the remaining ones wait for the next scheduling iteration — which is exactly why your TTFB climbs in waves (~0.49s → ~3.53s). This is normal vLLM continuous-batching, not per-request serialization.

One thing to double-check: /api/tts_direct_stream isn't a standard vLLM-Omni endpoint. The official serving paths are /v1/audio/speech and /v1/audio/speech/stream (WebSocket). If you're going through a custom wrapper, make sure the wrapper itself isn't adding its own queueing on top.

2. How to improve concurrent throughput

Two independent levers:

• More VRAM → more concurrency. GPU memory determines KV-cache size, which determines how many sequences can run in parallel. Raise --gpu-memory-utilization (default 0.92) and increase --max-num-seqs to admit more requests at once.
• More compute → better RTF & latency under load. A stronger GPU improves per-step throughput, so RTF and response time hold up better as concurrency grows.

Also — drop --enforce-eager for production. It disables CUDA graphs and is meant as a dev/debug knob; it lowers steady-state decode performance. Removing it should improve your latency right away.

3. Minimum GPU for concurrency

A single RTX 4090 (24GB) is enough to run VoxCPM2 with concurrency — 24GB is the recommended minimum in the official recipe.

4. Recommended settings

Note that results will vary significantly by GPU — VRAM sets the concurrency ceiling and compute sets the RTF. As a starting point:

vllm-omni serve $model_path \
  --omni \
  --served-model-name tts-service \
  --host 0.0.0.0 --port 8000 \
  --dtype bfloat16 \
  --max-num-seqs 16 \
  --max-num-batched-tokens 8192 \
  --gpu-memory-utilization 0.9

Tune --max-num-seqs and --gpu-memory-utilization upward for your L40S (48GB has plenty of headroom over the 24GB baseline) until you hit your latency/throughput target, and remove --enforce-eager.

Hope this helps! Feel free to share follow-up numbers after retuning. 🚀

[mishradibyajyoti]

Hi @a710128 ,

vllm-omni serve $model_path --omni --served-model-name voxcpm2 --host 0.0.0.0 --port 8000 --dtype bfloat16 --max-num-seqs 16 --max-num-batched-tokens 8192 --gpu-memory-utilization 0.9

Sharing results from our VoxCPM2 (vllm-omni) concurrency test on the current L40 instance, ramping from 1 to 10 simultaneous streaming requests:

• 1 concurrent: TTFA ~0.17s, RTF 0.18x (well within real-time headroom)
• 2 concurrent: TTFA ~0.30s, RTF ~0.31x
• 3 concurrent: TTFA ~0.40-0.44s, RTF ~0.42x
• 5 concurrent: TTFA ~0.61-0.69s, RTF ~0.5-0.7x
• 8 concurrent: TTFA ~1.24-1.36s, RTF crosses 1.0x (server no longer faster than real-time)
• 10 concurrent: TTFA ~0.64-1.42s, RTF 1.05-1.36x (fully saturated, worse than real-time)

So up to about 3 concurrent streams we're close to our TTFT target of 0.1-0.2s, but latency degrades quickly past that, and by 8-10 concurrent the GPU is compute-saturated (RTF > 1.0x means it can no longer keep up with real-time generation at all).

Two questions for the team:

1. Is there anything on the current L40 setup (model config, batching settings, etc. ) that could meaningfully cut TTFT before we look at hardware? Want to rule out any easy wins first.

2. If we need to reliably support ~20 parallel streams at a 0.1-0.2s TTFT target, our current data suggests the L40 doesn't have the compute headroom for that. Can we get input on which GPU/server would be a better fit so we can size this properly instead of guessing?

Thanks!

[a710128]

I haven't tested L40/L40S myself. For reference, here are RTX 4090 numbers for VoxCPM2 from nanovllm-voxcpm — short prompt, no LoRA, gpu_memory_utilization=0.8, RTF measured as (wall_time - TTFB) / audio_duration:

concurrency	TTFB p50	RTF_per_req_mean
1	0.067s	0.103
8	0.079s	0.131
16	0.086s	0.176
32	0.114s	0.284
64	0.189s	0.605

Closed-loop, 60 users, no LoRA: TTFB p50 0.514s, RTF p50 0.674, 3.0 rps, 180/180 ok.

Full tables including LoRA configs are in the README.

[mishradibyajyoti]

test_concurrency_v3.py

Hi,

I’m testing VoxCPM2 using vLLM-Omni on an L40 instance and would like your input on the runtime/configuration and potential performance optimizations.

Server command:

vllm-omni serve $model_path \
  --omni \
  --served-model-name voxcpm2 \
  --host 0.0.0.0 \
  --port 8000 \
  --dtype bfloat16 \
  --max-num-seqs 16 \
  --max-num-batched-tokens 8192 \
  --gpu-memory-utilization 0.9

I’m using the attached test_concurrency_v3.py script for the concurrency testing, with the following command:

python test_concurrency_v3.py --ref-audio audio_ref.wav

On the current L40 instance, I observed the following results while ramping from 1 to 10 simultaneous streaming requests:

• 1 concurrent: TTFA ~0.17s, RTF ~0.18x
• 2 concurrent: TTFA ~0.30s, RTF ~0.31x
• 3 concurrent: TTFA ~0.40–0.44s, RTF ~0.42x
• 5 concurrent: TTFA ~0.61–0.69s, RTF ~0.5–0.7x
• 8 concurrent: TTFA ~1.24–1.36s, RTF >1.0x
• 10 concurrent: TTFA ~0.64–1.42s, RTF ~1.05–1.36x

Based on these results, the L40 appears to become compute-saturated around 8–10 concurrent streams.

Could you please help us understand the following?

1. What runtime/configuration are you using for VoxCPM2/vLLM-Omni when you achieve better concurrency/latency? In particular, it would be useful to know the exact vLLM/vLLM-Omni version, PyTorch/CUDA version, and any relevant runtime or backend configuration.

2. Can you review my server command and test script and identify any configuration or implementation changes that could meaningfully improve TTFA/TTFT and concurrent streaming performance on the L40? I’d like to make sure we are not leaving any easy performance optimizations on the table before considering different hardware.

I’m sharing the complete test script and the vLLM-Omni server configuration so that the setup can be reviewed end-to-end.

Thanks!