Silence at the start of generations

[Jimmys-Code]

Generating the word "hi"

We're currently experiencing an issue with our voice agent where there's a noticeable delay before the TTS output begins. Despite achieving sub-200ms for the first audio token, the generated audio sometimes includes up to 700ms of silence at the start.

We suspect this might be related to the training data not being properly trimmed. Has anyone else encountered this issue, and are there any known solutions or workarounds to reduce the initial silence?

Thank you for your assistance!

[jjmlovesgit]

Idea for a Workaround --- I use FFMPEG’s silenceremove filter to preprocess the audio and add or remove silence. Strive to do it all in memory and do not write out the file etc. for speed.... Ask ChatGPT about code below etc..... ffmpeg is stuff it knows well...

import subprocess

cmd = [
"ffmpeg", "-i", "input.mp3",
"-af", "silenceremove=start_periods=1:start_threshold=-50dB:start_mode=once",
"-f", "wav", "-"
]

process = subprocess.Popen(cmd, stdout=subprocess.PIPE)
subprocess.run(["ffplay", "-"], stdin=process.stdout)

[keepingitneil]

We remove silence at the beginning but obviously this still incurs the latency added by generating those tokens if you are targeting real-time applications.

We made an RMS-based streaming one as follows:

import logging
import numpy as np

MAGIC_NUMBER_THRESHOLD = 0.004


# TODO: fades, append new inference, etc.
class SilenceRemover:
    """orpheus tends to generate silence at the beginning. Here we remove it to reduce head-of-line latency"""

    def __init__(self):
        self._started = False
        self._silence_time = 0
        self._window = b""

    def push_bytes(self, audio_bytes: bytes):
        if self._started:
            return audio_bytes

        self._silence_time += len(audio_bytes) / 2 / 24000.0
        frame = np.frombuffer(audio_bytes, dtype=np.int16)
        float_arr = frame.astype(np.float32) / 32768.0
        rms = np.sqrt(np.mean(np.square(float_arr)))
        if rms < MAGIC_NUMBER_THRESHOLD:
            return b""

        logging.info(f"Removed {self._silence_time} from beginning")
        self._started = True
        return audio_bytes

For removing the actual silence from the model, we've had some success with LoRa fine tunes with data that just doesn't have silence at the beginning.

[stefanionescu]

Ended up creating synthetic speech with Orpheus and finetuned other models with the target voices. No more silence and also found faster models.

Tried for days to remove the leading silence with LoRA but couldn't do it.

Either that or as @keepingitneil said, trim.