bootstrap sampling, (single or paired possible)

[JJumSSu]

Hello, I implemented the feature of a significance test based on bootstrapping sampling from this issue (#70).

I referenced the code from '''https://github.com/mjpost/sacrebleu/pull/11/files''' and '''https://github.com/neubig/util-scripts/blob/master/paired-bootstrap.py'''.

Single or paired systems' confidence intervals and significance tests can now be conducted.

(paired systems)

(single system)

system1 represents the prediction file of the first input, system2 represents the second input delimited by a tab. (e.g. paste system1 system2 | sacrebleu ... )

Also when a single system is provided(standard version) with bootstrap-trials more than 1, the original sacreBLEU score and the confidence intervals are provided as an output.

[mjpost]

Once #88 is merged, I will take a look at this. I'm adding collaborators so there will no longer be a bottleneck on me.

[JJumSSu]

No problem! Take your time :)

[martinpopel]

I haven't reviewed all the code properly. Just a few comments.
Also, would it be possible to move most of the code into a separate file to keep the main sacrebleu.py of a reasonable size?
(If the the answer is "not easily", we can leave it for future PRs, I think.)

[martinpopel]

document also paired_significance_test and significance_value

[martinpopel]

It would be nice if the bootstrap statistics (e.g. p-value) are available through the API, i.e. returned in the BLEU object. However, this is just a suggestion - I haven't thought about it much and it could be done in a separate PR.

[martinpopel]

@JJumSSu: Thank you very much for this PR. Can you please rebase on the current master (or merge in) and resolve the conflicts?

[JJumSSu]

@martinpopel : Thank you for your thoughtful comment!
May I take a look at it next week?
I'm currently in the middle of doing something else.

[ozancaglayan]

Working my way through several papers, wikipedia articles and codes written for this purpose. I think this way of computing CI using the t-statistic (i.e. 1.96 * (stdev / sqrt(n_bootstrap_samples))) may not be the correct way of doing this. This is what is explained in Philipp Koehn's (@phikoehn) paper (section 4): Unfortunately, this method to compute confidence intervals does not work for the BLEU metric, since the BLEU metric is not the mean of single sentence scores
Moreover, neither the Moses' script nor the @neubig 's code above (also his compare-mt code) applies this technique but instead sorts the scores and picks 25th and 975th elements if n_samples == 1000.

The t-statistic method above always yields a CI offset of ~0.02 for BLEU scores between [0, 100].

So I need some guidance here to implement this bootstrap resampling and CI functionality correctly. In the meantime I'll replicate Moses' script's behavior to be on the safe side.

(I don't know if mentioning people outside the contributors actually works but let's see.)