i18n: further work, expr support by RenjiSann · Pull Request #8292 · uutils/coreutils

RenjiSann · 2025-06-30T20:40:06Z

Replaces #7986

Internationalization of `expr`

Handles non-UTF8 inputs without panicking
Support for locale-aware string comparison (which solves expr: 50n \> -51 is producing different result than GNU expr #5378)
Support for locale encoding aware for length, substr and index, which now support
- However, match can't be ported, as the regex engine we use (onig) doesn't handle non UTF-8 input (someday we will Replace Oniguruma #1145)
- This makes the GNU test expr-multibyte.pl pass at 75%, match tests are still missing. See expr is failing with multibyte chars #3132

Additions to `uucore::i18n`

Refactored the feature dependency tree to optimize compile times
Added a decimal module that allows one to access decimal and grouping separators etc...

github-actions · 2025-06-30T21:40:58Z

GNU testsuite comparison:

Congrats! The gnu test tests/fmt/goal-option is no longer failing!

github-actions · 2025-07-01T00:12:00Z

GNU testsuite comparison:

Skip an intermittent issue tests/misc/tee (fails in this run but passes in the 'main' branch)

src/uucore/src/lib/features/i18n/collator.rs

github-actions · 2025-07-01T10:59:11Z

GNU testsuite comparison:

Skip an intermittent issue tests/misc/stdbuf (fails in this run but passes in the 'main' branch)
Skipping an intermittent issue tests/misc/tee (passes in this run but fails in the 'main' branch)
Skipping an intermittent issue tests/timeout/timeout (passes in this run but fails in the 'main' branch)

sylvestre · 2025-07-01T12:01:03Z

src/uucore/src/lib/features/i18n/mod.rs

+        let mut split = locale_var_str.split(&['.', '@']);
+
+        if let Some(simple) = split.next() {
+            let bcp47 = simple.replace("_", "-");


maybe add a comment explaining what means bcp47 :)

github-actions · 2025-07-01T14:22:32Z

GNU testsuite comparison:

Skipping an intermittent issue tests/timeout/timeout (passes in this run but fails in the 'main' branch)

github-actions · 2025-07-02T03:16:25Z

GNU testsuite comparison:

Skip an intermittent issue tests/misc/stdbuf (fails in this run but passes in the 'main' branch)
Skipping an intermittent issue tests/misc/tee (passes in this run but fails in the 'main' branch)

RenjiSann · 2025-07-04T00:54:37Z

src/uu/expr/src/expr.rs

+    #[error("Unsupported non-utf8 argument passed to match: {}", _0.quote())]
+    UnsupportedNonUtf8Match(String),


@sylvestre This is a uutils only error at the moment. I am not sure whether adding an error like this or calling expect() at the problematic code path would be better.
Also, because I consider this a bit like an internal error, I didn't bother adding a translation. I will do it if you deem it necessary.

Hm, so this is a temporary error until we add support for non-utf8 arguments? In this case I would use expect().

I hope it is only temporary, but this is a pretty big limitation of oniguruma, not sure if we can find a workaround for this

Ok, so it's probably a permanent error and thus I would keep it as it is. I think it would be beneficial to include a translation, as it's a bit inconsistent to have all but one translated.

I've added a translation 👍

RenjiSann · 2025-07-04T01:00:51Z

src/uu/expr/src/locale_aware.rs

+    let mut opts = CollatorOptions::default();
+    opts.alternate_handling = Some(AlternateHandling::Shifted); // This is black magic
+    let _ = try_init_collator(opts);


Here, I don't exactly understand how the CollatorOptions affect the collating. I've just toyed with it until #5378 passed. That's why I said it's black magic.

github-actions · 2025-07-04T02:16:05Z

GNU testsuite comparison:

Skip an intermittent issue tests/misc/stdbuf (fails in this run but passes in the 'main' branch)
Skip an intermittent issue tests/misc/tee (fails in this run but passes in the 'main' branch)
Skipping an intermittent issue tests/tail/inotify-dir-recreate (passes in this run but fails in the 'main' branch)

cakebaker · 2025-07-06T12:04:50Z

There is a conflict :|

src/uucore/Cargo.toml

src/uu/expr/src/locale_aware.rs

github-actions · 2025-07-07T23:21:42Z

GNU testsuite comparison:

Skipping an intermittent issue tests/misc/tee (passes in this run but fails in the 'main' branch)
Skipping an intermittent issue tests/timeout/timeout (passes in this run but fails in the 'main' branch)

src/uucore/src/lib/features/i18n/mod.rs

github-actions · 2025-07-08T10:46:48Z

GNU testsuite comparison:

Skipping an intermittent issue tests/misc/stdbuf (passes in this run but fails in the 'main' branch)
Skipping an intermittent issue tests/timeout/timeout (passes in this run but fails in the 'main' branch)
Congrats! The gnu test tests/head/head-elide-tail is no longer failing!

src/uu/expr/src/locale_aware.rs

github-actions · 2025-07-08T14:00:43Z

GNU testsuite comparison:

Skipping an intermittent issue tests/misc/tee (passes in this run but fails in the 'main' branch)

- Add locale comparison test (issue uutils#5378) - Implement GNU expr-multibyte test

github-actions · 2025-07-09T14:21:15Z

GNU testsuite comparison:

Skip an intermittent issue tests/misc/stdbuf (fails in this run but passes in the 'main' branch)
Skip an intermittent issue tests/timeout/timeout (fails in this run but passes in the 'main' branch)

cakebaker · 2025-07-09T14:24:31Z

Good work!

RenjiSann mentioned this pull request Jun 30, 2025

i18n: Collation draft #7986

Closed

RenjiSann force-pushed the collating branch 2 times, most recently from 26a0eee to 644ea84 Compare June 30, 2025 20:46

RenjiSann force-pushed the collating branch from 13bf589 to 92b4264 Compare June 30, 2025 23:17

RenjiSann force-pushed the collating branch from 92b4264 to 7b07a09 Compare July 1, 2025 09:43

sylvestre reviewed Jul 1, 2025

View reviewed changes

src/uucore/src/lib/features/i18n/collator.rs Show resolved Hide resolved

RenjiSann force-pushed the collating branch from 7b07a09 to 4f82d48 Compare July 1, 2025 10:06

sylvestre reviewed Jul 1, 2025

View reviewed changes

RenjiSann force-pushed the collating branch from 4f82d48 to 7436bb2 Compare July 1, 2025 13:28

RenjiSann force-pushed the collating branch from 7436bb2 to ec5c28e Compare July 2, 2025 02:23

RenjiSann force-pushed the collating branch 3 times, most recently from 8ad5eb5 to 4b5b0d1 Compare July 4, 2025 00:48

RenjiSann requested a review from sylvestre July 4, 2025 00:48

RenjiSann commented Jul 4, 2025

View reviewed changes

RenjiSann force-pushed the collating branch 2 times, most recently from da3c388 to 9660754 Compare July 4, 2025 01:22

RenjiSann marked this pull request as ready for review July 4, 2025 03:13

cakebaker reviewed Jul 6, 2025

View reviewed changes

src/uucore/Cargo.toml Outdated Show resolved Hide resolved

cakebaker reviewed Jul 7, 2025

View reviewed changes

src/uu/expr/src/locale_aware.rs Show resolved Hide resolved

RenjiSann force-pushed the collating branch 2 times, most recently from bfb5cc6 to 7987ade Compare July 7, 2025 09:20

RenjiSann force-pushed the collating branch 2 times, most recently from 2b7d214 to c0538f0 Compare July 7, 2025 22:37

cakebaker reviewed Jul 8, 2025

View reviewed changes

src/uucore/src/lib/features/i18n/mod.rs Outdated Show resolved Hide resolved

cakebaker reviewed Jul 8, 2025

View reviewed changes

src/uucore/src/lib/features/i18n/mod.rs Outdated Show resolved Hide resolved

cakebaker reviewed Jul 8, 2025

View reviewed changes

src/uucore/src/lib/features/i18n/mod.rs Outdated Show resolved Hide resolved

RenjiSann force-pushed the collating branch from c0538f0 to c5e884d Compare July 8, 2025 09:19

cakebaker reviewed Jul 8, 2025

View reviewed changes

src/uucore/src/lib/features/i18n/mod.rs Outdated Show resolved Hide resolved

RenjiSann force-pushed the collating branch from c5e884d to d9357fa Compare July 8, 2025 09:39

RenjiSann added 2 commits July 8, 2025 12:02

i18n: small refactor, add decimal stuff

bb8744f

i18n: add collating

4b78537

RenjiSann force-pushed the collating branch from d9357fa to 4034791 Compare July 8, 2025 10:02

cakebaker reviewed Jul 8, 2025

View reviewed changes

src/uu/expr/src/locale_aware.rs Outdated Show resolved Hide resolved

cakebaker reviewed Jul 8, 2025

View reviewed changes

src/uu/expr/src/locale_aware.rs Outdated Show resolved Hide resolved

RenjiSann force-pushed the collating branch from 4034791 to a6a4e33 Compare July 8, 2025 13:15

RenjiSann added 6 commits July 9, 2025 15:34

expr: Handle non UTF8 inputs

a669a84

expr: add locale-aware index

5ec652b

expr: add locale-aware comparison

7bed9b5

expr: add locale-aware length

8c565ec

expr: add locale-aware substr

82a06e7

test(expr): add locale-related tests, remove deprecated tests

4ab3431

- Add locale comparison test (issue uutils#5378) - Implement GNU expr-multibyte test

RenjiSann force-pushed the collating branch from a6a4e33 to 4ab3431 Compare July 9, 2025 13:34

cakebaker merged commit 745ae4b into uutils:main Jul 9, 2025
116 of 117 checks passed

cakebaker mentioned this pull request Jul 10, 2025

expr: 50n \> -51 is producing different result than GNU expr #5378

Closed

RenjiSann mentioned this pull request Jul 15, 2025

expr failed to match multibyte character #8335

Closed

BrewTestBot mentioned this pull request Sep 6, 2025

uutils-coreutils 0.2.0 Homebrew/homebrew-core#236403

Merged

		#[error("Unsupported non-utf8 argument passed to match: {}", _0.quote())]
		UnsupportedNonUtf8Match(String),

Uh oh!

Conversation

RenjiSann commented Jun 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Internationalization of expr

Additions to uucore::i18n

Uh oh!

github-actions bot commented Jun 30, 2025

Uh oh!

github-actions bot commented Jul 1, 2025

Uh oh!

Uh oh!

github-actions bot commented Jul 1, 2025

Uh oh!

sylvestre Jul 1, 2025

Choose a reason for hiding this comment

Uh oh!

RenjiSann Jul 1, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jul 1, 2025

Uh oh!

github-actions bot commented Jul 2, 2025

Uh oh!

RenjiSann Jul 4, 2025

Choose a reason for hiding this comment

Uh oh!

cakebaker Jul 8, 2025

Choose a reason for hiding this comment

Uh oh!

RenjiSann Jul 8, 2025

Choose a reason for hiding this comment

Uh oh!

cakebaker Jul 9, 2025

Choose a reason for hiding this comment

Uh oh!

RenjiSann Jul 9, 2025

Choose a reason for hiding this comment

Uh oh!

RenjiSann Jul 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jul 4, 2025

Uh oh!

cakebaker commented Jul 6, 2025

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jul 7, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jul 8, 2025

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jul 8, 2025

Uh oh!

github-actions bot commented Jul 9, 2025

Uh oh!

Uh oh!

cakebaker commented Jul 9, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

RenjiSann commented Jun 30, 2025 •

edited

Loading

Internationalization of `expr`

Additions to `uucore::i18n`

RenjiSann Jul 4, 2025 •

edited

Loading