How to get cluster centroids from NeuralDiarizer

[RakeshReddyKondeti]

Hi @nithinraok ,

I’m using NeuralDiarizer and want to access the cluster centroids for each speaker (after clustering). I’d like to use these for speaker verification or matching speakers across sessions.

Is there a clean way to extract or save these centroids from the diarizer pipeline?

Thanks!

[tango4j]

Hi.

Get access to the raw speaker labels obtained from the clustering at here :

NeMo/nemo/collections/asr/parts/utils/speaker_utils.py

Line 514 in 559cc97

cluster_labels = cluster_labels.cpu().numpy()

Then use those labels to find the centroid of these embeddings:

NeMo/nemo/collections/asr/parts/utils/speaker_utils.py

Line 495 in 559cc97

embeddings_in_scales=uniq_embs_and_timestamps['embeddings'],

The main clustering algo is in
https://github.com/NVIDIA-NeMo/NeMo/blob/main/nemo/collections/asr/parts/utils/longform_clustering.py#L27

Note that NeMo now offers an end-to-end streaming diarization model called Sortformer (streaming version).
https://huggingface.co/nvidia/diar_streaming_sortformer_4spk-v2

However, end-to-end version makes you hard to get the centroids so clustering diarization would serve for your purpose.

[RakeshReddyKondeti]

Hi @tango4j,

Thanks for your response!

Just to clarify, I’m using the NeuralDiarizer pipeline for inference purposes, and I want to directly access the cluster centroids (average embeddings per speaker) after clustering, for use in speaker verification or assigning the real name to the speaker.

From my understanding, the NeuralDiarizer pipeline already computes and stores these centroids internally. For example, after running diarization, I can access them via:

from nemo.collections.asr.models import NeuralDiarizer
from pyannote.audio import Annotation

model =  NeuralDiarizer.from_pretrained(
            model_name= str(Path(self.config.cache_dir)/self.config.nemo.model_name),
            vad_model_name= str(Path(self.config.cache_dir)/self.config.nemo.vad_model_name) if self.config.nemo.vad_model_name else None,
            map_location= self.config.device,
            verbose= self.config.verbose,
        )

diarization: Annotation = self.model(
            audio_filepath, num_speakers = self.config.num_speakers, 
            # max_speakers= self.config.num_speakers, 
            num_workers=0, verbose= True)

emb_sess_avg_dict = model.clustering_embedding.emb_sess_test_dict
for scale_idx, session_dict in emb_sess_avg_dict.items():
    for uniq_id, emb_info in session_dict.items():
        avg_embs = emb_info['avg_embs']  # shape: [embedding_dim, max_speakers]

A few questions:

• Is this the recommended way, or is there a more direct/cleaner method in the API to extract or save these averaged embeddings?
• The shape of avg_embs is [embedding_dim, max_speakers], not [embedding_dim, num_speakers]. Should I slice to [avg_embs[:, :num_speakers] to get the averaged embeddings for num_speakers ?
• What is the best practice for setting max_speakers? I noticed that setting it to 1 causes errors, but values ≥2 work fine. Is there a technical reason for this?
• Since this provides averaged embeddings per scale, which scale's embeddings should be used for best results in speaker verification or matching?

Just wanted to confirm I’m not missing anything, since it looks like the pipeline already provides what I need. Thanks!

[tango4j]

1. You could use emb_info['avg_embs'] for your purpose. However, this is designed for diarization purpose so if you want to actually compare and verify speaker IDs, these embeddings do not have enough fidelity for that purpose since these are generated from relatively short audio segments.

2. There will be "num_speakers" of average embeddings per scale and per speakers. This "num_speakers" number is either counted by clustering or given by the user depending on the configurations.

3. max_speaker=1 is prohibited because there is no point of running diarization or Multispeaker ASR on it. If there is only one speaker, It is recommended to use VAD or regular ASR instead.

4. The longer the better. The longer segments you use for extracting speaker embedding, the more accurate speaker IDs you extract from that embedding. Here is very intuitive diagram I drew in a paper (https://arxiv.org/pdf/2011.10527)

However, empirically we concluded that there is no accuracy gain roughly after 10 second mark. Having said that, under 10 seconds of lengths, it is always the longer the better. This means embeddings based on 0.5 second or 1.5 second length segments are not very reliable. Thus, there is always tradeoff between temporal resolution and fidelity of speaker embeddings.