Feature request: Full Case Folding

[biodranik]

Any plans to support full case folding, according to ftp://ftp.unicode.org/Public/UNIDATA/CaseFolding.txt ? It is very useful for code to match search queries entered by users, e.g. convert ß to ss etc.

[jiahao]

Not at the moment, but I'm sure a PR would be welcome.

[ScottPJones]

I believe that is handled in https://github.com/nolta/ICU.jl (@nolta kindly merged some PRs of mine there recently, to bring it up to 0.4).

[biodranik]

My question was about utf8proc as a separate C/C++ unicode processing library, to use it in projects different from Julia :)

[ScottPJones]

OK! I've also contributed to utf8proc, to fix some problems I found.
Any particular reason to use utf8proc instead of ICU? ICU now directly supports UTF-8, it's not just UTF-16 any more.

[stevengj]

Because ICU is huge and the API is enormous and annoying?

As @jiahao said, a PR for this would be welcome.

[biodranik]

Last time I checked ICU several years ago, the reason was that it was too complex and huge to use it in cross-platform iOS/Android/Desktop development. utf8proc is a perfect example of simplicity.
Are there also any speed benchmarks/comparisons? I'm not sure that ICU is fast enough for huge amount of text processing on mobile devices (our case).

[jiahao]

@deathbaba not currently (#12).

@ScottPJones there was no need to stray off topic by dragging in ICU and your contributions.

[ScottPJones]

OK, I was thinking more about Julia's use case of utf8proc vs. ICU, ICU is tiny compared to the enormous libraries that are used in core Julia for arrays.

@jiahao I don't think ICU is off topic, it does what @deathbaba asked for, and I'm rather familiar with both, which was all I meant by mentioned having worked on them. @deathbaba hadn't stated at all what his use case was, when I brought ICU up.

@deathbaba That is a good point, ICU does contain everything including the kitchen sink.
What things in particular do you need out of utf8proc? (I'd been thinking of rewriting a lot of it, to improve performance and cut size of tables, but then decided I'd rather do that in Julia instead of C, but maybe your use case is a good reason to go ahead and think about improving utf8proc).

[jiahao]

@deathbaba apologies, but it has become necessary to lock this issue.

[StefanKarpinski]

This is a completely reasonable feature request. ICU may work for @deathbaba or not, but telling people to use ICU instead of improving utf8proc is counterproductive.

[jiahao]

@deathbaba's reply (posted with his permission):

---

@jiahao thanks for the benchmarks reference, they looks pretty interesting!

@ScottPJones our case is to make a perfect search in MAPS.ME :) In general, we need to preprocess a lot of raw strings added by community of OpenStreetMap, and match these strings effectively on mobile device, for any language and any input. This includes stripping out all diacritics, full case folding, and even some special conversions which are not covered in Unicode standard but are important for users trying to find something.
I've already mentioned ß=>ss conversion, there are also non-standard Ł=>L, й=>и, famous turkish İ and ı conversions, all very important if you don't have a Ł key on your keyboard, for example, and trying to enter it as L (and find some Polish street for example).
Now we have our own highly-optimized implementation for NFKD and Case Folding. The issue came from recently released clang compiler update from Apple, which simply hangs while compiling our normalization code with -O2 or -O3 optimizations (original bug was caught here: https://code.google.com/p/android/issues/detail?id=173992 ). That's We're looking for alternative solutions now.

P.S. Good news are that we're going to open MAPS.ME source code very soon, so you'll see what we already have there.