Big gap in WER between online and offline CTC decoding

[chiendb97]

I tried offline decoding using hlg_decode.cu and online decoding using online_decode.cu. And here is the result:

• For model librispeech conformer ctc: offline decoding: 3.49% WER, online decoding: 19.08% WER
• For our model: offline decoding: ~3% WER, online decoding: ~18% WER
  (WER online decoding is much larger than offline decoding (both use the same am output), online decoding uses chunk size 16)

Could you please tell me the difference between offline decoding and online decoding? In addition, could you tell us the result of 2 kinds of decoding.
Thanks!

[danpovey]

There are examples in Sherpa of real-time/streaming/online decoding, I think that might be a better starting point?
Normally you need to use a model that has been trained with streaming in mind.

[chiendb97]

There are examples in Sherpa of real-time/streaming/online decoding

Can you please specify which example it is? I did look into sherpa repo but did not find any examples about CTC-based streaming.

Normally you need to use a model that has been trained with streaming in mind.

I used the same AM output for both offline and streaming decoding. I don't think the gap can be that big.

[pkufool]

Can you please specify which example it is? I did look into sherpa repo but did not find any examples about CTC-based streaming.

Sorry, there is no ctc hlg streaming decoding in Sherpa, only one example in k2/torch/bin (I think it is the online_deocde.cu you used).

I used the same AM output for both offline and streaming decoding. I don't think the gap can be that big.

We normally test the streaming decoding method with a streaming model, may be you can try the online_decode.cu with a streaming model. A offline model is not suitable for a streaming decoding method.

[danpovey]

But @pkufool I think that binary just evaluates the nnet for the entire file and simulates streaming, so surely it should in principle give the same results as the offline decoding if it was given a non-streaming model? (Even though this would not be useful in practice).

[chiendb97]

@pkufool @danpovey How I tested was that I read the audio file and evaluated nnet output for the entire audio. Then I used that output to simulate streaming as in online_decode.cu and used the final text result to compute the WER. I did the test twice, using the conformer ctc model from icefall and my conformer ctc model (using wenet). However, the results obtained were not as good as offline decoding in both cases.
I tried to print out the lattice (lattice.fsa.values) of the online decoder and noticed that the first few lattices are quite the same as that of the offline decoder. But then it started to differ.

[danpovey]

hm, how did it differ?
@pkufool do you think there is possibly a bug that is affecting him?
@chiendb97 what version of k2 are you using? see if a newer version helps.

[chiendb97]

what version of k2 are you using? see if a newer version helps.

I am using the latest version of k2.

[pkufool]

@pkufool do you think there is possibly a bug that is affecting him?

Yes, I think there could be some bugs. I will look into the code.

[svandiekendialpad]

I am currently experiencing the same issue. Offline decoding is fine but any form of streaming using OnlineDenseIntersecter increases WER by an unreasonable amount with almost all new errors coming from deletions.

[pkufool]

I am currently experiencing the same issue. Offline decoding is fine but any form of streaming using OnlineDenseIntersecter increases WER by an unreasonable amount with almost all new errors coming from deletions.

OK, I am debuging it.

[svandiekendialpad]

Any updates @pkufool?

[pkufool]

Any updates @pkufool?

Sorry, I did not fix it at that day and forgot it, will return to it.

[pkufool]

@svandiekendialpad @chiendb97 Does the differences only happens when using --use_ctc_decoding=false (i.e decoding with an ngram) ?

[binhtranmcs]

Hi @pkufool, I just ran tests again using librispeech conformer ctc, here is the result:

• Using --use_ctc_decoding=true, I got WER=7.3%.
• Using offline ctc decoding in ctc_decode.cu, I got WER=2.6%.

So I think there is still a significant difference between online and offline implementations regardless of using n-gram (though the gap is smaller).