uppercase/lowercase functions are not portable?

[stevengj]

Since @ScottPJones was mentioning upper/lowercase functions recently, I took a quick look at them and I noticed that we are calling towupper and towlower, which are C99 functions that accept wchar_t arguments.

Unfortunately, this means that they are broken on Windows (where wchar_t is 16 bits) for any character outside the BMP. Even on other platforms with a 32-bit wchar_t, they are going to return different results on different systems, and many systems will have out-of-date Unicode tables. They are also locale-dependent; I'm not sure if this is desirable for us.

utf8proc has up-to-date upper/lower/titlecase mapping data already in its "database" (generated from http://www.unicode.org/Public/UNIDATA/UnicodeData.txt), so maybe we should just add a utf8proc_toupper function (etc.) to utf8proc to make this accessible. Then we could call that (probably plus a check for the common case of ASCII codepoints).

[ScottPJones]

Yep... although I wouldn't solve this with utf8proc... I'd intended to tackle this shortly (it's something else we need for our project, and I'd prefer to use Julia as much as possible).

[JeffBezanson]

although I wouldn't solve this with utf8proc

Why not? If utf8proc does this wrong, we should just fix it instead of re-implementing.

[ScottPJones]

Because, I believe it can better be implemented in pure Julia (sorry, now I'm a convert!)

[jiahao]

Essentially a duplicate of #7848

The Unicode standard defines upper-lower case mappings on a per character basis, but a correct transformation must necessarily take into account the locale. (See the infamous Turkish i - İ, ı - I pairings as the most egregious examples.)

[ScottPJones]

I don't think this really is a duplicate of #7848, and should be reopened.
If you look at ICU (probably the most widely used library for this sort of stuff), you can do either locale dependent or locale independent case mappings...
I think that in Base, Julia should give consistent results on all platforms, using locale independent mappings.
I think fancy locale dependent mappings, as discussed in #7848, belong in a package (I'll have to look, they may already be handled by the ICU.jl package).

[jiahao]

Did you read the discussion in #7848? The conclusion was exactly what you said, that we should have a locale independent choice in Base and have the locale specific choices not in base.

[ScottPJones]

I missed the last couple of comments (I'm in a meeting right now!) Sorry, my fault totally... The only difference is that I think any locale dependent mappings should be done as extending the Base methods via a Package...

[ScottPJones]

Also, I think @stevengj's point was something else, and this may need to still be open... it was about characters outside the BMP not being handled (even in a local independent fashion), on Windows,
simply due to the way the functions are implemented...

[jiahao]

If it really makes you happier, I'll close #7848 in favor of this issue.

[ScottPJones]

Actually, I don't think #7848 needs to be closed either 😀
IMO, this one is about non-BMP case mapping being broken on Windows (good catch, @stevengj),
and #7848 is about the need for having both locale independent (in Base) and locale dependent (hopefully in a package outside of Base).

[stevengj]

Pure Julia code for this has its pluses and minuses. On the minus side, it is more work: this information is already in utf8proc, and adding a couple of functions to expose it to Julia is literally something like 6 lines of C code. Also on the minus side, having it in utf8proc makes it available to non-Julia users. On the plus side, writing code in Julia is more fun, and potentially allows for more optimizations (e.g. inlining).

For my own part, I tend to default to the path of least work.