feature support for the significance test by bootstrapping?

[JJumSSu]

Hi, team. Thanks for the cool library.

I'm looking for the feature of computing BLEU score with confidence interval like in that of Moses. I noticed that similar issue has been raised in issue

I just want to check if any new updates have been made regarding this feature in sacreBLEU.

Thank you.

[mjpost]

A common request (#11 #40)! I’m unlikely to do this myself but would happily accept a PR.

[JJumSSu]

I found this neat code(link) from Graham Neubig's repository and am currently using it which supports detokenized BLEU, namely SacreBLEU. I'll work on a PR if I can afford the time later. Thank you!

[martinpopel]

@JJumSSu: note that https://github.com/neubig/util-scripts/blob/master/paired-bootstrap.py is a naive implementation, unacceptably slow for real use: with num_samples=10000, it calls the tokenization and extraction of matching n-grams 10000 times. However, any sane implementation of BLEU bootstrap should precompute these statistics and then re-sample just the statistics (of matching 1-4 grams, and sentence lengths). This is done in https://github.com/mjpost/sacrebleu/pull/11/files, but it implements only the non-paired version.

The paired property is the main obstacle when adding paired bootstrap to SacreBLEU - we would need to change the interface, so that you can provide multiple systems' translations at once. SacreBLEU currently expects the system translation on the stdin and does not provide a way to name the system. We would need to change this e.g. to accepting tab-separated systems on the input and optionally providing system names, e.g. paste moses.txt nematus.txt | sacrebleu --bootstrap_samples=10000 --system_names=Moses,Nematus reference.txt.

[mjpost]

Tab-delimited STDIN would be the way to do this (although I would like to have dashes in param names, e.g., --bootstrap-samples).

System names could default to something like "System 1", "System 2", etc. Providing names is a nice option, though.

[mjpost]

It shouldn't be hard to do this quickly, since compute_bleu (which computes BLEU from sufficient statistics) is already factored out from corpus_bleu (which does the extract from streams and tokenization).

[JJumSSu]

I'm sorry I checked late. I will work on this PR ASAP.
Thanks for the novel comments @martinpopel @mjpost

[JJumSSu]

Closing since the PR was made (#78).