How to identify positive samples and negative samples?

[MAxx8371]

In the paper, it mentions that the score is defined as follow. Is that calculated by summing the logprob of each token of the ground true y conditioned on (e,x)? If that were right, the score would not be between 0 and 1 as shown in the pic. In that case, what is the threshold used for identifying pos&neg samples? Appreciate it!

[OhadRubin]

In the paper, it mentions that the score is defined as follow. Is that calculated by summing the logprob of each token of the ground true y conditioned on (e,x)?

Yes, you are correct.

If that were right, the score would not be between 0 and 1 as shown in the pic. I

Yes. I found it to be more intuitive to explain with probability instead of logprobs.... It means the same thing.

In that case, what is the threshold used for identifying pos&neg samples?

We just take the top-5 as positive and the bottom-5 as negative.
Since all the candidates are from a BM25, the bottom-5 are good negatives