Sort order of languages in Kiwix serve / library.kiwix.org is wrong

[Jaifroid]

See screenshot below. The language "Español" is not sorted under "E", but (I suppose) by its English name "Spanish", i.e., it appears in the list along with other languages beginning with "S". This is unintuitive. It would be better for the list to be sorted in UTF-8 order of localized language names, with English only being used as a fallback if we don't have the localized version (I'm not sure why we wouldn't, however).

[Jaifroid]

For info, this is how it is done in pwa.kiwix.org, where the sort order is by local language name, and the English designation is provided in parenthesis for those who may be looking for a particular language but don't know the local name or script. (The mapping isn't perfect, because it is provided by an array that I had to populate manually in places where info was missing.)

[kelson42]

Seems clearly buggy even if I wonder a bit why this has not been detected earlier. Should be sorted alphabetically, based on what is displayed (not based on a technical ISO language code). Not in favour either to print anything additional. To me this should be done in the backend.

[veloman-yunkan]

@Jaifroid This is a complex topic (unless there is an established common way to do it).

It would be better for the list to be sorted in UTF-8 order of localized language names

Then languages with non-latin scripts will come after languages with latin scripts.

For info, this is how it is done in pwa.kiwix.org, where the sort order is by local language name

I guess that in pwa.kiwix.org you map different scripts to the Latin alphabet through some loose phonetic correspondence and treat the order of letters in the Latin alphabet as a good reference for sorting. However the Latin alphabet lacks dedicated letters for a lot of sounds and has to use digraphs (like sh, dz, ts, etc). If the language self name starts with such a sound (e.g. dz) it is counterintuitive to look it up in languages starting with the first letter of the digraph (d). Then the order of sounds in different alphabets is different. For example, in Russian the phonetic analog of V (the cyrillic letter В) - is the third letter of the alphabet.

I definitely agree that the current sort order is silly for languages with Latin-based scripts. However the general problem can hardly be solved in a way that doesn't raise similar concerns by speakers of other languages.

[Jaifroid]

@veloman-yunkan Thanks for the explanation, which makes a lot of sense. Looking at it again, I think my solution was simply to order it alphabetically by the iso language codes (en, el, es) that Wikipedia uses, but making sure that same-language groups [like zh: '中文 (Chinese)', lzh: '文言 (Classical Chinese)'] appear together. This seems to give a better result than ordering by English names for languages but displaying localized names. I agree it's not perfect.

[kelson42]

@Jaifroid This is a complex topic (unless there is an established common way to do it).

there is a common way: use libicu https://unicode-org.github.io/icu/userguide/collation/

[veloman-yunkan]

@kelson42 I still think that the issue at hand is different - we are not talking about different locale-dependent collation methods. The challenge is to sort a list of languages that is composed of strings (language self-names) in different languages. The essence of the conflict can be illustrated by a fictional language Zxcvb - in the alphabet of that language Z is the first letter, thus for the list of languages to be intuitive for the speakers of Zxcvb that language must appear in the beginning (but that is absolutely confusing for other users of the Latin-based alphabets).

[kelson42]

@veloman-yunkan I understand but there is no concrete/real evidence that sorting the language strings (each in its own language) using the collation of UI language is going to give a wrong result, or does it? If it does, what viable alternative do you propose?

To me, there clearly no perfect solution AFAIK, but we need to be pragmatic.

[Jaifroid]

Short of allowing users to set their locale and then using a locale-specific sort algorithm (which would probably be nonsensical unless we knew names and spellings of languages in each of the many Wikipedia languages), then we have to have an approximation. The ISO-639 language codes as used by Wikipedia have a natural relationship to the ZIM archives we make. These are mostly the two-letter codes, but they occasionally use three-letter codes for more esoteric languages or variants thereof. I know we don't only do Wikipedia, but it is the most multi-lingual target we have. Unless someone has a better idea, or libicu provides an internationally accepted, global way of sorting international language names that is not specific to any one locale (and still allows giving language names in their native scripts)...

[kelson42]

@Jaifroid Wikipedia don't use ISO code for languages AFAIK in a visible manner, except for URLs... and most of the users don't know how URLs work. Here you really have a tech. trope IMO.

[kelson42]

An other solution would be to use a sophisticated solution which makes the question of the sorting less relevant. The language selector of Wikipedia is reusable AFAIK.

[Jaifroid]

An other solution would be to use a sophisticated solution which makes the question of the sorting less relevant. The language selector of Wikipedia is reusable AFAIK.

That sounds interesting!