Guidance Needed: Orpheus-TTS + TensorRT-LLM Optimization for Concurrency

[shekharmeena2896]

Hi, I have successfully continued the pre training and then finetuned the model on high quality voices.

the next major challenge, using the model for production and serve the multiple concurrent users. my use case is to serve it for telephony customer support agent, I tried various method could not set up a inference pipeline which serves the purpose.

I tried vllm, tensorrt, tried to quantise but everything seems to fail

[SachinTelecmi]

@shekharmeena2896 can you tell me how you constructed the dataset for pre training ?? and also what you have to do if you want to change the codec and replace the SNAC to some other codec???

[shekharmeena2896]

Hi, if you want to construct the dataset for pre training there is dataset preparation script in GitHub of this repo, u can check that out and also u can't change the codec as the llama backbone was already continued pretraining using snac codec, if you want to change u have build the different model with huge corpora of data like 100,000 hours

[egvikosh]

Also facing same issue, but only in case second request come during execution of first request. If they are all done sequentially all good.
I think there is some issue in wrapper code of OrpheusModel. I see that there is for instance Baseten have completely different implementation and model works just fine, but its tight coupled with Baseten cloud.
Does someone have solution or patch to OrpheusModel class?

[lvt1693]

I tried with tensorrt-llm and it was really simple:

1. Install tensorrt-llm (I suggest using docker image with triton server), or you can just instal lin your env if you want some headache
2. Follow the guide for serving llama3 model for tensorrt-llm: quantize/convert checkpoint, build, and serve.
3. Create a separated process for sending LLM request and snac decoder to audio. streaming is better.

I got 4 ccu on rtx3090 so baseten's claim on their H100 is quite possible for us too.

[shekharmeena2896]

@lvt1693 so you just build the tensorRt engine on quantised model of orpheus. then how do u use it like can u explain me step by step

[lvt1693]

Assume that you got orpheus LLM up and ready.
When you send the correct llm request, it will return text output as audio token.
And we must use snac to decode to audio.
Take a look at 2 repo orpheus fastapi and orpheus streaming, you can take the code from there and implementing the LLM request + audio decode for orpheus.
Just throw them into chatpgt or cursor and it will be done in 1 or 2 minutes.

[stefanionescu]

@shekharmeena2896 if you quantize to int4-awq on TRTLLM and run Orpheus with low temp (0.3-0.45 was the sweet spot in my tests, higher produced too many artifacts) you can get to 16 concurrent, real-time streams on A100. If you run the 4bit quant on H100 you should be able to exceed Baseten's concurrency numbers.

[piercarlo62]

Hello @lvt1693,
I replicated the inference on the Triton Server with the TensorRT-LLM backend. I quantised the model with the FP8 algorithm and tried four concurrent inferences with the following parameters:

• text: "Hello, this is a streaming text-to-speech test to evaluate concurrent buffer performance."
• voice: "Tara"
• max_tokens: 990
• temperature: 0.5
• top_k: 50
• top_p: 0.9
• repetition_penalty: 1.1
• end_id: 128258

With decoupling audio decoding on client side activated, the stream is executed.
My GPU is an RTX 5090, which is more powerful than yours.
I am executing audio decoding on the client side (GPU) with a buffer of four frames (28 tokens), and I am warming the SNAC model before the inference.

However, the first audio chunk I get is in the range of 210–330 ms, which is much worse than yours.

Could you share your settings, such as the quantisation algorithm and other useful information, to help me understand why your results are better?

Thankyou very much

Piercarlo

[lvt1693]

hi @piercarlo62 I'm pretty low tech so I just simply do these:

paths

export HF_TOKEN='api-key'
export HF_MODEL=./orpheus-3b-ft
export QOUT=./models/orpheus3b/quant_int4awq_int8kv
export EOUT=./engines/orpheus3b/int4awq_sm86

1) quantize (INT4-AWQ + INT8 KV)

python quantize.py
--model_dir $HF_MODEL --dtype float16 --qformat int4_awq
--awq_block_size 128 --kv_cache_dtype int8 --calib_size 256
--output_dir $QOUT

2) build engine

trtllm-build
--checkpoint_dir $QOUT
--output_dir $EOUT
--gpt_attention_plugin float16 --gemm_plugin float16
--remove_input_padding enable --context_fmha enable
--kv_cache_type paged
--max_input_len 1024 --max_seq_len 4096 --max_batch_size 24

trtllm-serve "$EOUT" --host 0.0.0.0 --port 7861
--kv_cache_free_gpu_memory_fraction 0.10 --max_batch_size 1024 --tokenizer ./orpheus-3b-ft

====
I use https://github.com/Lex-au/Orpheus-FastAPI
with modified speechpipe.py
And run the server_stream-work.py

You can check my files. hope you could make a clean repo so i can use too 🥹

server_stream-work.py
speechpipe.py