Difference score for one sentence with sentence_blue() vs Command Line --sentence-level

[ymoslem]

SacreBLEU version: 1.4.12
Python: 3.7

Issue: Using Python sentence_blue() and the Command Line with the --sentence-level flag (or -sl) for only one sentence gives different scores.
Question: I need sentence_blue() from Python to give the same output as the Command Line with the --sentence-level flag (or -sl), what should I do?

From the Command Line:

echo "Producția de zahăr primă va fi exprimată în ceea ce privește zahărul alb;" | sacrebleu -sl <(echo "producţia de zahăr brut se exprimă în zahăr alb;")

Output:

BLEU+case.mixed+numrefs.1+smooth.exp+tok.13a+version.1.4.12 = 8.5 35.7/15.4/4.2/2.3 (BP = 1.000 ratio = 1.400 hyp_len = 14 ref_len = 10)

From Python

Using sentence_blue()

ref = "producţia de zahăr brut se exprimă în zahăr alb;"
sys = "Producția de zahăr primă va fi exprimată în ceea ce privește zahărul alb;"

bleu = sacrebleu.sentence_bleu(sys, ref)
print(bleu.score)

Output:

0.0

Using corpus_blue()

ref = [["producţia de zahăr brut se exprimă în zahăr alb;"]]
sys = ["Producția de zahăr primă va fi exprimată în ceea ce privește zahărul alb;"]

bleu = sacrebleu.corpus_bleu(sys, ref)
print(bleu.score)

Output:

8.493098745313148

[martinpopel]

This is related to #64, but slightly more complicated.

First, note the disclaimer: computing BLEU on the sentence level is not its intended use, BLEU is a corpus-level metric.
If you need a sentence-level metric, try e.g. chrF (sacrebleu -m chrf).

Second, you should use sacrebleu.sentence_bleu(sys, [ref]) instead of sacrebleu.sentence_bleu(sys, ref) because it takes a list of reference translations for a given sentence. (I fixed the documentation in #99.)

Third, corpuse_bleu uses smooth_method='exp' by default, but sentence_bleu uses smooth_method='floor'.
If you use the same smoothing method, you will get the same result (there is also a difference in the use_effective_order default value, but it does not change the result in this case).

>>> import sacrebleu
>>> ref = "producţia de zahăr brut se exprimă în zahăr alb;"
>>> sys = "Producția de zahăr primă va fi exprimată în ceea ce privește zahărul alb;"
>>> sacrebleu.sentence_bleu(sys, [ref],  smooth_method='exp').score
8.493098745313148
>>> sacrebleu.sentence_bleu(sys, [ref],  smooth_method='floor').score 
0.0
>>> sacrebleu.corpus_bleu([sys], [[ref]],  smooth_method='exp').score                                                                                                                                         
8.493098745313148
>>> sacrebleu.corpus_bleu([sys], [[ref]],  smooth_method='floor').score                                                                                                                                         
0.0

Fourth, the floor smoothing method needs a smoothing value, but the default is 0, i.e. no smoothing. I am not sure if this is sensible. I guess the problem is, there is no obvious/standard smoothing value for floor (unlike for add-k, where it is 1).

Fifth, the command-line --sentence-level option does not use the code in sacrebleu/compat.py, so the default smoothing methods is not floor, but exp. This is probably a bug and --sentence-level should have the same default for smoothing method as sacrebleu.sentence_bleu.
@ozancaglayan, @mjpost: What should be the default smoothing method for sentence-level BLEU? I vote for exp or add-k unless we change also the default smoothing value for floor to a non-zero value. We should also add the smoothing value to the signature (esp. if a non-default value is used).

$ echo "Producția de zahăr primă va fi exprimată în ceea ce privește zahărul alb;" | sacrebleu -sl -s floor -sv 0 -b -w 4 <(echo "producţia de zahăr brut se exprimă în zahăr alb;")
0.0000

Sixth, note that sacreBLEU uses a case-sensitive BLEU by default. So your two sentences share just 5 unigrams and 2 bigrams (de zahăr and alb ;), but no trigrams and no 4-grams. This is the reason why the BLEU is 0 if no smoothing is used.

>>> sacrebleu.sentence_bleu(sys, [ref],  smooth_method='floor').counts                                                                                                                                        
[5, 2, 0, 0]

[ymoslem]

Many thanks, Martin, for the detailed answer!

Now, I get the same output if I add ref in a list and use smooth_method='exp'

import sacrebleu

ref = "producţia de zahăr brut se exprimă în zahăr alb;"
sys = "Producția de zahăr primă va fi exprimată în ceea ce privește zahărul alb;"

bleu = sacrebleu.sentence_bleu(sys, [ref], smooth_method='exp')
print(bleu.score)

Thanks for your confirmation that sacreBLEU uses a case-sensitive BLEU by default because this is what we need. I just notice that actually "producţia" and "Producția" use different accents under the "t"; so this is something for me to check.

Many thanks again!
Yasmin

[martinpopel]

"producţia" and "Producția" use different accents under the "t"

Yes. The former is t-cedilla (U+0163), which can be used in the Gagauz language. However, your sentences are in Romanian, where the latter - t-comma (U+021B) should be used, but this letter was not part Unicode versions before 1999, which is why T-cedilla is often used instead even in Romanian.

I was considering to add (optional) Unicode normalization (NFC) and punctuation normalization to SacreBLEU, but it would not help with this particular case anyway.

[ozancaglayan]

@mjpost @martinpopel the issue regarding the defaults for --sentence-level is because the CLI defaults to corpus BLEU, hence smoothing method defaults to exp. If one gives --sentence-level without --smoothing-method, the method stays as exp. I agree that this should be fixed in a reasonable way. Adding the value to the signature should also be done, but that's the easier part of the problem :)

• NLTK has smoothing=False by default and does not seem to impose a particular smoothing
• Currently, floor without --smooth-value is equivalent to none actually. I think we can start by setting its default to 0.1 which is the default in both the paper and NLTK.
• This code seems to use add-1 if smooth=True: https://github.com/tensorflow/nmt/blob/master/nmt/scripts/bleu.py
• The human-ranking correlations are OK with exp (method 3) in the paper above. So maybe the easiest is to also use exp for sentence-level BLEU? This will mean that there will be no behaviour change for CLI users but the API users will see different numbers from now on.

[ozancaglayan]

ping @mjpost @martinpopel

[martinpopel]

I like c183417 but we should also report the smoothing value in the BLEU signature.

[ozancaglayan]

The last one changes the smoothing method of sentence_bleu to exp as well. This is a choice, we can also keep it at floor but since we changed the value from 0 to 0.1, the numbers would never be backward-comparable.

We can also add a warning to this function that'll be printed once on stderr if the function is used:

+    warnings.warn(
+        'In versions >= 1.4.15, the smoothing method for sentence_bleu() is '
+        'synchronized with the sacreBLEU command-line utility, and set to `exp`. '
+        'We also changed the default smoothing value for "floor" from 0 to 0.1. '
+        'Therefore, to obtain the exact same sentence-level BLEU scores as before, '
+        'you need to set smooth_method and smooth_value to \'floor\' and 0.0, respectively.',
+        RuntimeWarning,
+    )
+

What are your thoughts?

[ozancaglayan]

Smooth value in signature:

sacrebleu $ sacrebleu -sl -w 4 -i sys ref --smooth-value 10
BLEU+case.mixed+numrefs.1+smooth.exp+tok.13a+version.1.4.14 = 8.4931 35.7/15.4/4.2/2.3 (BP = 1.000 ratio = 1.400 hyp_len = 14 ref_len = 10)
sacrebleu $ sacrebleu -sl -w 4 -i sys ref --smooth-value 1
BLEU+case.mixed+numrefs.1+smooth.exp+tok.13a+version.1.4.14 = 8.4931 35.7/15.4/4.2/2.3 (BP = 1.000 ratio = 1.400 hyp_len = 14 ref_len = 10)
sacrebleu $ sacrebleu -sl -w 4 -i sys ref --smooth-method floor
BLEU+case.mixed+numrefs.1+smooth.floor[0.10]+tok.13a+version.1.4.14 = 4.5169 35.7/15.4/0.8/0.9 (BP = 1.000 ratio = 1.400 hyp_len = 14 ref_len = 10)
sacrebleu $ sacrebleu -sl -w 4 -i sys ref --smooth-method floor --smooth-value 0
BLEU+case.mixed+numrefs.1+smooth.floor[0.00]+tok.13a+version.1.4.14 = 0.0000 35.7/15.4/0.0/0.0 (BP = 1.000 ratio = 1.400 hyp_len = 14 ref_len = 10)
sacrebleu $ sacrebleu -sl -w 4 -i sys ref --smooth-method add-k
BLEU+case.mixed+numrefs.1+smooth.add-k[1.00]+tok.13a+version.1.4.14 = 14.8826 35.7/21.4/7.7/8.3 (BP = 1.000 ratio = 1.400 hyp_len = 14 ref_len = 10)
sacrebleu $ sacrebleu -sl -w 4 -i sys ref --smooth-method add-k --smooth-value 0
BLEU+case.mixed+numrefs.1+smooth.add-k[0.00]+tok.13a+version.1.4.14 = 0.0000 35.7/15.4/0.0/0.0 (BP = 1.000 ratio = 1.400 hyp_len = 14 ref_len = 10)
sacrebleu $ sacrebleu -sl -w 4 -i sys ref --smooth-method none
BLEU+case.mixed+numrefs.1+smooth.none+tok.13a+version.1.4.14 = 0.0000 35.7/15.4/0.0/0.0 (BP = 1.000 ratio = 1.400 hyp_len = 14 ref_len = 10)
sacrebleu $ sacrebleu -sl -w 4 -i sys ref --smooth-method none --smooth-value 992303920
BLEU+case.mixed+numrefs.1+smooth.none+tok.13a+version.1.4.14 = 0.0000 35.7/15.4/0.0/0.0 (BP = 1.000 ratio = 1.400 hyp_len = 14 ref_len = 10)

[martinpopel]

I like the signatures proposal. In addition, we could warn if --smooth-value is provided with smoothing methods none and exp (which do not require any smoothing value).

[mjpost]

This looks good to me, except why is the BLEU score precision so fine-grained? There should be only one number after the decimal point.

Maybe this is the wrong time, but what do you think about returning the signature as a JSON object? Something like the following (but a single line, so it could be parsed with jq):

{
  "signature": "BLEU+case.mixed+numrefs.1+smooth.exp+tok.13a+version.1.4.14 = 8.4931 35.7/15.4/4.2/2.3 (BP = 1.000 ratio = 1.400 hyp_len = 14 ref_len = 10)",
  "metric": "BLEU",
  "score": "8.4",
  "case": "mixed",
  "numrefs": 1,
  "smooth": "add-k[1.00]",
  "tok": "13a",
  "version": "1.4.14",
}

The "signature" line is for backwards-compatibility.

[ozancaglayan]

@mjpost that's because I was running it with -w 4, don't worry =)

Let's summarize:

• Do we all agree on setting the floor's default to 0.1?
• Do we all agree to change the smoothing method from floor to exp for compat.sentence_bleu() function?