Language in CanonicalizeLanguageTag is unclear

[dilijev]

https://tc39.github.io/ecma402/#sec-canonicalizelanguagetag

It takes the steps specified in RFC 5646 section 4.5, or successor, to bring the language tag into canonical form, and to regularize the case of the subtags, but does not take the steps to bring a language tag into "extlang form" and to reorder variant subtags.

The last bit is a bit unclear as to whether reordering variant subtags is required by the spec.

I think this should be rewritten as one of the following options to make it clear whether this is required or not.

It takes the steps specified in RFC 5646 section 4.5, or successor, to bring the language tag into canonical form, and to regularize the case of the subtags, and to reorder variant subtags, but does not take the steps to bring a language tag into "extlang form."

It takes the steps specified in RFC 5646 section 4.5, or successor, to bring the language tag into canonical form, and to regularize the case of the subtags, but does not take the steps to bring a language tag into "extlang form" nor to reorder variant subtags.

FWIW, it seems that implementations based on the ICU as well as ChakraCore (using Windows Globalization) do reorder variant subtags. (EDIT: I got my terminology mixed up, this point and the following example are not especially important anyway as the issue is about a possible lack of clarity in the language.)

Intl.getCanonicalLocales('de-de-u-kn-true-co-phonebk');
// equals 'de-DE-u-co-phonebk-kn-true'

@bterlson FYI

[caridy]

I'm supportive of this. Can you send a pull request so we can get other folks to look at it?

[anba]

FWIW, it seems that implementations based on the ICU as well as ChakraCore (using Windows Globalization) do reorder variant subtags.

Intl.getCanonicalLocales('de-de-u-kn-true-co-phonebk');
// equals 'de-DE-u-co-phonebk-kn-true'

This example does not include variant subtags, but instead is an example of Unicode extension subtag canonicalization (https://tools.ietf.org/html/rfc6067#section-2.1.1), which is currently described as optional in 6.2.3:

The specifications for extensions to BCP 47 language tags, such as RFC 6067, may include canonicalization rules for the extension subtag sequences they define that go beyond the canonicalization rules of RFC 5646 section 4.5. Implementations are allowed, but not required, to apply these additional rules.

So I'm not sure what's proposed here.

[dilijev]

@anba Oops, got my terminology mixed up. Thanks for the clarification. Edited above.

However, I still think it is unclear from the language whether or not "to reorder variant subtags" is required to be done / not required / required NOT to be done.

Basically I want to know whether each of the following is required (in parens, what I believe is intended from the language in the spec):

1. Bring the language tag into canonical form (required to be done)
2. Regularize the case of the subtags (required to be done)
3. Bring tag into "extlang form" (required NOT to be done)
4. Reorder variant subtags (??)

If 4. is required NOT to be done, I think the text should be reworded as my second suggestion above:

It takes the steps specified in RFC 5646 section 4.5, or successor, to bring the language tag into canonical form, and to regularize the case of the subtags, but does not take the steps to bring a language tag into "extlang form" nor to reorder variant subtags.

Possibly nor should just be or. My grammar is escaping me this morning.

However, I think it is worthwhile to be even more explicit:

A conforming implementation shall take the steps specified in RFC 5646 section 4.5, or successor, to bring the language tag into canonical form, and to regularize the case of the subtags. Furthermore, a conforming implementation shall not take the steps to bring a language tag into "extlang form", nor shall it reorder variant subtags.

[anba]

Just getting back to this.

4. Reorder variant subtags (??)

This step should not be done. "[...], but does not take the steps to bring a language tag into "extlang form" and to reorder variant subtags." in CanonicalizeLanguageTag should be interpreted as perform neither the "extlang form" transformation steps nor reorder variant subtags.

We don't want to require reordering variant subtags, because then we'd need to include more parts of the IANA language subtag registry in implementations (e.g. to differentiate which variant subtags are tied to specific prefixes and which variant subtags are general purpose). And from https://tools.ietf.org/html/rfc5646#section-4.1:

If a variant lists a second variant in one of its 'Prefix' fields, the first variant SHOULD appear directly after the second variant in any language tag where both occur. General purpose variants (those with no 'Prefix' fields at all) SHOULD appear after any other variant subtags. Order any remaining variants by placing the most significant subtag first. If none of the subtags is more significant or no relationship can be determined, alphabetize the subtags.

Phrases like "Order any remaining variants by placing the most significant subtag first." seem hard to get right.

[dilijev]

Okay, I'll make a PR with clarifying wording a la (copied from above):

A conforming implementation shall take the steps specified in RFC 5646 section 4.5, or successor, to bring the language tag into canonical form, and to regularize the case of the subtags. Furthermore, a conforming implementation shall not take the steps to bring a language tag into "extlang form", nor shall it reorder variant subtags.