CultureInfo.TextInfo.ListSeparator broken in .Net 5

[olmobrutall]

As part of updating of updating the Csv class in Signum.Utilities to .Net 5, I've realize that the value for ListSeparator has changed for many cultures that use , instead of . for decimals separators.

foreach (var ci in new [] { "", "en", "en-GB", "en-US", "es", "es-ES", "de", "de-DE", "fr", "fr-FR" })
{
	Console.WriteLine($"Culture {ci}\tListSeparator {CultureInfo.GetCultureInfo(ci).TextInfo.ListSeparator}");
}

netcoreapp3.1

Culture         ListSeparator ,
Culture en      ListSeparator ,
Culture en-GB   ListSeparator ,
Culture en-US   ListSeparator ,
Culture es      ListSeparator ;
Culture es-ES   ListSeparator ;
Culture de      ListSeparator ;
Culture de-DE   ListSeparator ;
Culture fr      ListSeparator ;
Culture fr-FR   ListSeparator ;

net5 rc2

Culture         ListSeparator ,
Culture en      ListSeparator ,
Culture en-GB   ListSeparator ,
Culture en-US   ListSeparator ,
Culture es      ListSeparator .
Culture es-ES   ListSeparator .
Culture de      ListSeparator .
Culture de-DE   ListSeparator .
Culture fr      ListSeparator  
Culture fr-FR   ListSeparator  

Both tests in the same machine.

Previously I was using ListSeparator as a way to detect the separator that will be used in a Csv file. Is this the intended purpose?

Tagging subscribers to this area: @tarekgh, @safern, @krwq
See info in area-owners.md if you want to be subscribed.

[tarekgh]

@olmobrutall In .NET 5.0 we have switched to depend on the ICU instead of NLS Win32 APIs. Usually ICU behavior is more correct as it picks its globalization data from CLDR (Unicode Standard). Please have a look at the doc https://docs.microsoft.com/en-us/dotnet/standard/globalization-localization/globalization-icu for more details about the change in .NET 5.0. We offer a config switch too to allow apps revert back to old behavior if needed too. the doc has details how you can use the config switch.

Another point, Globalization data is not considered constant and can change. It is not right to take any dependency on the Globalization data or assume it will never change. If you see any data doesn't make sense or is not correct, the issue can be raised to the CLDR and can be tracked to be discussed and fixed if it is real issue.

Let me know if you have any question.

[ANahr]

Sorry @tarekgh, but to me this surely looks like a bug.
If you look at the relevant code at

runtime/src/libraries/Native/Unix/System.Globalization.Native/pal_localeStringData.c

Line 246 in 0e402bc

case LocaleString_ListSeparator:

you can see that it isn't even implemented (falls through to ThousandsSeparator). So either this is a bug in .Net Core or the info is generally not available in ICU, which would then still be a bug (And would not be an issue for CLDR if it is not part of it). As native german I can tell that using . as a list separator does not make any sense.

[ANahr]

After checking our code this would be a .Net 5 blocker for us for the majority of our software products unless we would remove all uses of ListSeparator. A ListSeparator has no connection of any kind (and should not have) to the ThousandsSeparator.

[olmobrutall]

Thanks @ANahr, nice the you found the code.

I've checked .Net Core 3.1 behaviour to see if ListSeparator can be inferred from DecimalsSeparator somehow.

ListSeparator ,   NumberDecimalSeparator . (180)
    , as, as-IN, bn, bn-BD, bn-IN, bo, bo-CN, chr, chr-Cher, chr-Cher-US, en, en-001, en-029, en-AG, en-AI, en-AS, en-AU, en-BB, en-BI, en-BM, en-BS, en-BW, en-BZ, en-CA, en-CC, en-CK, en-CM, en-CX, en-CY, en-DM,
    en-ER, en-FJ, en-FK, en-FM, en-GB, en-GD, en-GG, en-GH, en-GI, en-GM, en-GU, en-GY, en-HK, en-IE, en-IL, en-IM, en-IN, en-IO, en-JE, en-JM, en-KE, en-KI, en-KN, en-KY, en-LC, en-LR, en-LS, en-MG, en-MH, en-MO,
    en-MP, en-MS, en-MT, en-MU, en-MW, en-MY, en-NA, en-NF, en-NG, en-NR, en-NU, en-NZ, en-PG, en-PH, en-PK, en-PN, en-PR, en-PW, en-RW, en-SB, en-SC, en-SD, en-SG, en-SH, en-SL, en-SS, en-SX, en-SZ, en-TC, en-TK,
    en-TO, en-TT, en-TV, en-TZ, en-UG, en-UM, en-US, en-VC, en-VG, en-VI, en-VU, en-WS, en-ZM, en-ZW, es-MX, es-US, gu, gu-IN, he, he-IL, hi, hi-IN, hy, hy-AM, iu, iu-Cans, iu-Cans-CA, iu-Latn, iu-Latn-CA, ja,
    ja-JP, km, km-KH, kn, kn-IN, ko, ko-KR, kok, kok-IN, ks-Deva, ks-Deva-IN, la, la-001, mi, mi-NZ, mn-Mong, mn-Mong-CN, mn-Mong-MN, mni, mni-IN, moh, moh-CA, mr, mr-IN, ne, ne-NP, or, or-IN, pa, pa-Guru, pa-IN, pap,
    pap-029, quc, quc-Latn, quc-Latn-GT, quz-PE, sa, sa-IN, sd-Deva, sd-Deva-IN, syr, syr-SY, ta, ta-IN, te, te-IN, th, th-TH, ug, ug-CN, zh, zh-CN, zh-Hans, zh-Hant, zh-HK, zh-MO, zh-SG, zh-TW


ListSeparator ;   NumberDecimalSeparator . (264)
    aa, aa-DJ, aa-ER, aa-ET, ak, ak-GH, am, am-ET, ar, ar-001, ar-AE, ar-BH, ar-DJ, ar-DZ, ar-EG, ar-ER, ar-IL, ar-IQ, ar-JO, ar-KM, ar-KW, ar-LB, ar-LY, ar-MA, ar-OM, ar-PS, ar-QA, ar-SA, ar-SD, ar-SO, ar-SS, ar-SY,
    ar-TD, ar-TN, ar-YE, asa, asa-TZ, bem, bem-ZM, bez, bez-TZ, bin, bin-NG, bm, bm-Latn, bm-Latn-ML, bo-IN, brx, brx-IN, byn, byn-ER, ce, ce-RU, cgg, cgg-UG, cy, cy-GB, dav, dav-KE, de-CH, de-LI, dje, dje-NE, dz,
    dz-BT, ebu, ebu-KE, ee, ee-GH, ee-TG, es-419, es-BR, es-BZ, es-CU, es-DO, es-GT, es-HN, es-NI, es-PA, es-PE, es-PR, es-SV, ff-Latn-NG, fil, fil-PH, ga, ga-IE, gd, gd-GB, gsw, gsw-CH, gsw-LI, guz, guz-KE, gv,
    gv-IM, ha, ha-Latn, ha-Latn-GH, ha-Latn-NE, ha-Latn-NG, haw, haw-US, ibb, ibb-NG, ig, ig-NG, ii, ii-CN, it-CH, jmc, jmc-TZ, kam, kam-KE, kde, kde-TZ, khq, khq-ML, ki, ki-KE, kln, kln-KE, ko-KP, kr, kr-Latn,
    kr-Latn-NG, ks, ks-Arab, ks-Arab-IN, ksb, ksb-TZ, ku-Arab-IR, kw, kw-GB, lag, lag-TZ, lg, lg-UG, lkt, lkt-US, lrc, lrc-IQ, lrc-IR, luo, luo-KE, luy, luy-KE, mas, mas-KE, mas-TZ, mer, mer-KE, mfe, mfe-MU, mg,
    mg-MG, mgo, mgo-CM, ml, ml-IN, mn, mn-Cyrl, mn-MN, ms, ms-MY, ms-SG, mt, mt-MT, my, my-MM, mzn, mzn-IR, naq, naq-NA, nd, nd-ZW, ne-IN, nso, nso-ZA, nus, nus-SS, nyn, nyn-UG, om, om-ET, om-KE, pa-Arab, pa-Arab-PK, rm,
    rm-CH, rof, rof-TZ, rwk, rwk-TZ, saq, saq-KE, sbp, sbp-TZ, sd, sd-Arab, sd-Arab-PK, ses, ses-ML, si, si-LK, sn, sn-Latn, sn-Latn-ZW, so, so-DJ, so-ET, so-KE, so-SO, ssy, ssy-ER, sw, sw-KE, sw-TZ, sw-UG, ta-LK,
    ta-MY, ta-SG, teo, teo-KE, teo-UG, ti, ti-ER, ti-ET, tig, tig-ER, tn, tn-BW, tn-ZA, to, to-TO, twq, twq-NE, ur, ur-IN, ur-PK, vai, vai-Latn, vai-Latn-LR, vai-Vaii, vai-Vaii-LR, vo, vo-001, vun, vun-TZ, wal,
    wal-ET, xh, xh-ZA, xog, xog-UG, yi, yi-001, yo, yo-BJ, yo-NG, zh-Hans-HK, zh-Hans-MO, zu, zu-ZA


ListSeparator ;   NumberDecimalSeparator , (375)
    af, af-NA, af-ZA, agq, agq-CM, ar-MR, ast, ast-ES, az, az-Cyrl, az-Cyrl-AZ, az-Latn, az-Latn-AZ, ba, ba-RU, bas, bas-CM, be, be-BY, bg, bg-BG, br, br-FR, bs, bs-Cyrl, bs-Cyrl-BA, bs-Latn, bs-Latn-BA, ca,
    ca-AD, ca-ES, ca-ES-valencia, ca-FR, ca-IT, co, co-FR, cs, cs-CZ, cu, cu-RU, da, da-DK, da-GL, de, de-AT, de-BE, de-DE, de-IT, de-LU, dsb, dsb-DE, dua, dua-CM, dyo, dyo-SN, el, el-CY, el-GR, en-ID, eo, eo-001, es,
    es-AR, es-BO, es-CL, es-CO, es-CR, es-EC, es-ES, es-GQ, es-PH, es-PY, es-UY, es-VE, et, et-EE, eu, eu-ES, ewo, ewo-CM, ff, ff-Latn, ff-Latn-BF, ff-Latn-CM, ff-Latn-GH, ff-Latn-GM, ff-Latn-GN,
    ff-Latn-GW, ff-Latn-LR, ff-Latn-MR, ff-Latn-NE, ff-Latn-SL, ff-Latn-SN, fi, fi-FI, fo, fo-DK, fo-FO, fr, fr-029, fr-BE, fr-BF, fr-BI, fr-BJ, fr-BL, fr-CA, fr-CD, fr-CF, fr-CG, fr-CH, fr-CI, fr-CM,
    fr-DJ, fr-DZ, fr-FR, fr-GA, fr-GF, fr-GN, fr-GP, fr-GQ, fr-HT, fr-KM, fr-LU, fr-MA, fr-MC, fr-MF, fr-MG, fr-ML, fr-MQ, fr-MR, fr-MU, fr-NC, fr-NE, fr-PF, fr-PM, fr-RE, fr-RW, fr-SC, fr-SN, fr-SY, fr-TD, fr-TG,
    fr-TN, fr-VU, fr-WF, fr-YT, fur, fur-IT, fy, fy-NL, gl, gl-ES, gsw-FR, hr, hr-BA, hr-HR, hsb, hsb-DE, hu, hu-HU, ia, ia-001, id, id-ID, is, is-IS, it, it-IT, it-SM, it-VA, jgo, jgo-CM, jv, jv-Java, jv-Java-ID,
    jv-Latn, jv-Latn-ID, ka, ka-GE, kab, kab-DZ, kea, kea-CV, kk, kk-KZ, kkj, kkj-CM, kl, kl-GL, ksf, ksf-CM, ksh, ksh-DE, ky, ky-KG, lb, lb-LU, ln, ln-AO, ln-CD, ln-CF, ln-CG, lo, lo-LA, lt, lt-LT, lu, lu-CD, lv, lv-LV, mgh,
    mgh-MZ, mk, mk-MK, ms-BN, mua, mua-CM, nb, nb-NO, nb-SJ, nds, nds-DE, nds-NL, nl, nl-AW, nl-BE, nl-BQ, nl-CW, nl-NL, nl-SR, nl-SX, nmg, nmg-CM, nn, nn-NO, nnh, nnh-CM, no, nr, nr-ZA, oc, oc-FR, os, os-GE, os-RU, pl,
    pl-PL, prg, prg-001, prs, prs-AF, ps, ps-AF, pt, pt-AO, pt-BR, pt-CH, pt-CV, pt-GQ, pt-GW, pt-LU, pt-MO, pt-MZ, pt-PT, pt-ST, pt-TL, rn, rn-BI, ro, ro-MD, ro-RO, ru, ru-BY, ru-KG, ru-KZ, ru-MD, ru-RU, ru-UA, rw,
    rw-RW, sah, sah-RU, se, se-FI, se-NO, se-SE, seh, seh-MZ, sg, sg-CF, shi, shi-Latn, shi-Latn-MA, shi-Tfng, shi-Tfng-MA, sk, sk-SK, sl, sl-SI, sma, sma-NO, sma-SE, smj, smj-NO, smj-SE, smn, smn-FI, sms, sms-FI,
    sq, sq-AL, sq-MK, sq-XK, sr, sr-Cyrl, sr-Cyrl-BA, sr-Cyrl-ME, sr-Cyrl-RS, sr-Cyrl-XK, sr-Latn, sr-Latn-BA, sr-Latn-ME, sr-Latn-RS, sr-Latn-XK, ss, ss-SZ, ss-ZA, st, st-LS, st-ZA, sv, sv-AX, sv-FI,
    sv-SE, sw-CD, tg, tg-Cyrl, tg-Cyrl-TJ, tk, tk-TM, tr, tr-CY, tr-TR, ts, ts-ZA, tt, tt-RU, tzm, tzm-Arab, tzm-Arab-MA, tzm-Latn, tzm-Latn-DZ, tzm-Latn-MA, tzm-Tfng, tzm-Tfng-MA, uk, uk-UA, uz, uz-Arab,
    uz-Arab-AF, uz-Cyrl, uz-Cyrl-UZ, uz-Latn, uz-Latn-UZ, ve, ve-ZA, wae, wae-CH, wo, wo-SN, yav, yav-CM, zgh, zgh-Tfng, zgh-Tfng-MA


ListSeparator ,   NumberDecimalSeparator , (20)
    arn, arn-CL, en-150, en-AT, en-BE, en-CH, en-DE, en-DK, en-FI, en-NL, en-SE, en-SI, en-ZA, gn, gn-PY, quz, quz-BO, quz-EC, vi, vi-VN


ListSeparator ?   NumberDecimalSeparator . (4)
    dv, dv-MV, nqo, nqo-GN


ListSeparator ?   NumberDecimalSeparator / (2)
    fa, fa-IR


ListSeparator ?   NumberDecimalSeparator . (3)
    ku, ku-Arab, ku-Arab-IQ

Doesn't look like it can easily be inferred, there are even a few cultures where they use the same ListSeparator and NumberDecimalSeparator.