Map three different language lookups correctly

[tfmorris]

Feature Request

Problem / Opportunity

There are (at least) three different mappings which need to be made, independently, from the user's browser's language settings, taking into account their language preferences in priority order:

1. Book text language - Browser language lookup is incorrectly using ISO 639-1 instead of BCP 47 #12003 covers this.
2. UI localization language for Babel - this should use Babel's negotiate_locale to find the best match
3. Preferred Wikipedia link - I suspect Wikipedia has an algorithm / lookup to do this, but it would require investigation

As an example, consider zh-HK and zh-TW:

BCP 47 code: zh-HK
Book language codes: cmn + chi (chi is a macro language code for Chinese, while cmn refers to Mandarin Chinese)
Wikipedia code: zh (Chinese Wikipedia)

BCP 47 code: zh-TW
Book language codes: yue + chi
Wikipedia code: zh-yue? (Cantonese Wikipedia is much smaller than Chinese, so zh might be a better recommendation)

Unfortunately, the current implementation conflates these three mappings. It also only considers the first language in the Accept-Language header, when the user may have a localization that's supported and preferred to English as their second or third choice. eg. pt-BR, pt, es-419, en

Proposal

The full browser Accept-Language info should be retained for use in these mappings and for future uses like identifying locales which share a common language across multiple nations, such as Portugese Brazilian (pt-BR) or Latin American Spanish (es-419).

Babel's Locale.negotiate_locale should be used to find best matches so that more than the first locale/language is considered.

Breakdown

Implementation Details (for maintainers)

Related files

Refer to this map of common Endpoints:
*

Requirements Checklist

Checklist of requirements that need to be satisfied in order for this issue to be closed:

• [ ]

Stakeholders

---

Instructions for Contributors

• Before creating a new branch or pushing up changes to a PR, please first run these commands to ensure your repository is up to date, as the pre-commit bot may add commits to your PRs upstream.

[mekarpeles]

We're struggling a bit on the i18n front and would really benefit from a community lead to take on i18n PM'ing. As a first step, only capacity I have is to spin up an agent to classify all these and propose some sort of project management.

[bhardwajparth51]

Hey @mekarpeles! I'd love to help out here. I went through the issue and the core problem makes sense to me — the three mappings (book language, UI locale, Wikipedia) are currently tangled together and only the first Accept-Language value is being used.
I took a quick look in the codebase and found the entry point is infogami/utils/i18n.py (the parse_lang_header function). I'm genuinely interested in taking this on. Would it be okay if I start by posting a clean technical breakdown of the three separate functions + the exact places to update?
Happy to go at whatever pace works for the team!

[mekarpeles]

@bhardwajparth51 this seems like a significant effort and I don't have the bandwidth to review the solution. I can try to contribute project management towards i18n, to clear up what is being asked holistically within our i18n landscape but I cannot in good faith recommend you take on this issue because I'm not sure yet how I'd support you.

[eshantharjun9-hub]

I am interested in this issue can i do it

[eshantharjun9-hub]

#12110 closes this