feat: implement `shred --random-source` by f8ith · Pull Request #7005 · uutils/coreutils

f8ith · 2024-12-27T03:26:00Z

new uucore feature: rand_read
removed rand_read_adapter from shuf
added random_source to shred

sylvestre · 2024-12-28T11:49:09Z

If you are fixing issue #5711 you are doing more than just a refactor.
If so,

Split the commit into 1) move into uucore 2) shed uses this new features
could you please update the commit message
Add new tests to https://github.com/uutils/coreutils/blob/main/tests/by-util/test_shred.rs
thanks

src/uucore/src/lib/features/rand_read.rs

sylvestre · 2025-01-01T13:57:19Z

Looks great but seems that it is failing the GNU test:
bash -v util/run-gnu-test.sh tests/uniq/uniq tests/shred/shred-passes to test it:


2024-12-30T08:00:14.3400715Z FAIL: tests/shred/shred-passes
2024-12-30T08:00:14.3400788Z ==============================
2024-12-30T08:00:14.3400792Z 
2024-12-30T08:00:14.3400879Z --- exp	2024-12-30 07:48:09.684864709 +0000
2024-12-30T08:00:14.3400953Z +++ out	2024-12-30 07:48:09.687864717 +0000
2024-12-30T08:00:14.3401022Z @@ -1,6 +1,6 @@
2024-12-30T08:00:14.3401108Z -shred: f: pass 1/3 (random)...
2024-12-30T08:00:14.3401184Z -shred: f: pass 2/3 (random)...
2024-12-30T08:00:14.3401266Z -shred: f: pass 3/3 (random)...
2024-12-30T08:00:14.3401343Z +shred: f: pass  1/3 (random)...
2024-12-30T08:00:14.3401435Z +shred: f: pass  2/3 (random)...
2024-12-30T08:00:14.3401579Z +shred: f: pass  3/3 (random)...
2024-12-30T08:00:14.3401702Z  shred: f: removing
2024-12-30T08:00:14.3401794Z  shred: f: renamed to 0
2024-12-30T08:00:14.3401865Z  shred: f: removed
2024-12-30T08:00:14.3401932Z 1+0 records in
2024-12-30T08:00:14.3402161Z 1+0 records out
2024-12-30T08:00:14.3402308Z 102400 bytes (102 kB, 100 KiB) copied, 3.64241e-04 s, 102 MB/s
2024-12-30T08:00:14.3402396Z --- exp	2024-12-30 07:48:09.698864745 +0000
2024-12-30T08:00:14.3402472Z +++ out	2024-12-30 07:48:09.789864976 +0000
2024-12-30T08:00:14.3402572Z @@ -1,23 +1,9 @@
2024-12-30T08:00:14.3402669Z -shred: f: pass 1/20 (random)...
2024-12-30T08:00:14.3402804Z -shred: f: pass 2/20 (ffffff)...
2024-12-30T08:00:14.3402969Z -shred: f: pass 3/20 (924924)...
2024-12-30T08:00:14.3403099Z -shred: f: pass 4/20 (888888)...
2024-12-30T08:00:14.3403182Z -shred: f: pass 5/20 (db6db6)...
2024-12-30T08:00:14.3403261Z -shred: f: pass 6/20 (777777)...
2024-12-30T08:00:14.3403333Z -shred: f: pass 7/20 (492492)...
2024-12-30T08:00:14.3403413Z -shred: f: pass 8/20 (bbbbbb)...
2024-12-30T08:00:14.3403485Z -shred: f: pass 9/20 (555555)...
2024-12-30T08:00:14.3403563Z -shred: f: pass 10/20 (aaaaaa)...
2024-12-30T08:00:14.3403644Z -shred: f: pass 11/20 (random)...
2024-12-30T08:00:14.3403859Z -shred: f: pass 12/20 (6db6db)...
2024-12-30T08:00:14.3403938Z -shred: f: pass 13/20 (249249)...
2024-12-30T08:00:14.3404011Z -shred: f: pass 14/20 (999999)...
2024-12-30T08:00:14.3404152Z -shred: f: pass 15/20 (111111)...
2024-12-30T08:00:14.3404280Z -shred: f: pass 16/20 (000000)...
2024-12-30T08:00:14.3404415Z -shred: f: pass 17/20 (b6db6d)...
2024-12-30T08:00:14.3404514Z -shred: f: pass 18/20 (eeeeee)...
2024-12-30T08:00:14.3404590Z -shred: f: pass 19/20 (333333)...
2024-12-30T08:00:14.3404673Z -shred: f: pass 20/20 (random)...
2024-12-30T08:00:14.3404743Z -shred: f: removing
2024-12-30T08:00:14.3404819Z -shred: f: renamed to 0
2024-12-30T08:00:14.3404891Z -shred: f: removed
2024-12-30T08:00:14.3405080Z +thread 'main' panicked at src/uucore/src/lib/features/rand_read.rs:61:13:
2024-12-30T08:00:14.3405350Z +reading random bytes from Read implementation failed; error: ReadError: failed to fill whole buffer
2024-12-30T08:00:14.3405465Z +stack backtrace:
2024-12-30T08:00:14.3405599Z +   0: rust_begin_unwind
2024-12-30T08:00:14.3405756Z +   1: core::panicking::panic_fmt
2024-12-30T08:00:14.3405949Z +   2: <uucore::features::rand_read::ReadRng<R> as rand_core::RngCore>::fill_bytes
2024-12-30T08:00:14.3406025Z +   3: uu_shred::uumain
2024-12-30T08:00:14.3406093Z +   4: shred::main
2024-12-30T08:00:14.3406302Z +note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
2024-12-30T08:00:14.3406426Z FAIL tests/shred/shred-passes.sh (exit status: 1)
2024-12-30T08:00:14.3406430Z

sylvestre · 2025-01-01T13:57:44Z

tests/by-util/test_shred.rs

+    at.touch(random_file);
+    at.write(random_file, "random contents");
+
+    ucmd.arg("--random-source=test_random_file")


please add a test to verify the behavior if the file is missing

github-actions · 2025-01-05T14:00:49Z

GNU testsuite comparison:

Skipping an intermittent issue tests/tail/inotify-dir-recreate (passes in this run but fails in the 'main' branch)

sylvestre · 2025-01-20T09:30:01Z

fails with:


──── TRY 3 STDERR:       coreutils::tests test_shred::test_random_source
thread 'test_shred::test_random_source' panicked at tests/by-util/test_shred.rs:199:10:
Command was expected to succeed. code: 1
stdout = 
 stderr = shred: test_random_file: No such file or directory

stack backtrace:
   0: rust_begin_unwind
             at /rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/std/src/panicking.rs:652:5
   1: core::panicking::panic_fmt
             at /rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/core/src/panicking.rs:72:14
   2: tests::common::util::CmdResult::success
             at ./tests/common/util.rs:417:9
   3: tests::common::util::UCommand::succeeds
             at ./tests/common/util.rs:1782:9
   4: tests::test_shred::test_random_source
             at ./tests/by-util/test_shred.rs:195:5
   5: tests::test_shred::test_random_source::{{closure}}
             at ./tests/by-util/test_shred.rs:185:24
   6: core::ops::function::FnOnce::call_once
             at /rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/core/src/ops/function.rs:250:5
   7: core::ops::function::FnOnce::call_once
             at /rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

github-actions · 2025-01-20T09:46:46Z

GNU testsuite comparison:

GNU test failed: tests/misc/stdbuf. tests/misc/stdbuf is passing on 'main'. Maybe you have to rebase?

sylvestre · 2025-01-20T14:50:25Z

note that i started #6243
if you want to salvage some work

f8ith · 2025-01-24T02:54:26Z

Sorry, I didn't see that you already had an open pull request. Thing is because rand's rng generator is different from whichever generator gnuutils uses -rand needs more entropy to output one random byte - when the input file is very small our implementation fails when gnuutils succeeds. Not sure how to resolve this, perhaps there is something we could configure in the rand crate.

BenWiederhake · 2025-04-14T00:38:47Z

@f8ith, are you still working on this?

if no: Can I take this over, is that okay?
if yes: It seems that GNU shred just uses the --random-source as-is. You can see this easily by using a text file (e.g. CONTRIBUTING.md) as random-source. So we don't really need a "read-rand-adapter", we just need to copy the bytes from one file to another. This makes sense in context, I think: If the random-source is some trusted generator, like /dev/urandom or a fifo or pipe or whatever, then there's no additional processing needed, so copying all the bytes is the best thing to do (and, in fact, any preprocessing would lower entropy).

f8ith · 2025-04-16T15:06:40Z

@f8ith, are you still working on this?

Feel free to take over this PR.

BenWiederhake · 2025-05-30T11:24:12Z

See #7948.

sylvestre force-pushed the shred-random branch from 73d493c to adcd222 Compare December 28, 2024 11:47

sylvestre reviewed Dec 28, 2024

View reviewed changes

src/uucore/src/lib/features/rand_read.rs Show resolved Hide resolved

f8ith force-pushed the shred-random branch from adcd222 to a74b7ea Compare December 30, 2024 01:40

f8ith changed the title ~~refactor: move random_source to uucore~~ refactor: move random_source to uucore, feat: --random-source shred Dec 30, 2024

f8ith changed the title ~~refactor: move random_source to uucore, feat: --random-source shred~~ feat: implement shred --random-source Dec 30, 2024

sylvestre reviewed Jan 1, 2025

View reviewed changes

f8ith added 3 commits January 20, 2025 10:20

refactor: move shuf/rand_read_adapter to uucore

4327a5c

feat: implement --random-source for shred

4fe7dc9

wip: add more checks

fb915a4

sylvestre force-pushed the shred-random branch from a4ad654 to fb915a4 Compare January 20, 2025 09:20

BenWiederhake closed this May 30, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: implement `shred --random-source`#7005

feat: implement `shred --random-source`#7005
f8ith wants to merge 3 commits intouutils:mainfrom
f8ith:shred-random

f8ith commented Dec 27, 2024 •

edited

Loading

Uh oh!

sylvestre commented Dec 28, 2024

Uh oh!

Uh oh!

sylvestre commented Jan 1, 2025

Uh oh!

sylvestre Jan 1, 2025

Uh oh!

github-actions bot commented Jan 5, 2025

Uh oh!

sylvestre commented Jan 20, 2025

Uh oh!

github-actions bot commented Jan 20, 2025

Uh oh!

sylvestre commented Jan 20, 2025

Uh oh!

f8ith commented Jan 24, 2025 •

edited

Loading

Uh oh!

BenWiederhake commented Apr 14, 2025

Uh oh!

f8ith commented Apr 16, 2025

Uh oh!

BenWiederhake commented May 30, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

f8ith commented Dec 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sylvestre commented Dec 28, 2024

Uh oh!

Uh oh!

sylvestre commented Jan 1, 2025

Uh oh!

sylvestre Jan 1, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jan 5, 2025

Uh oh!

sylvestre commented Jan 20, 2025

Uh oh!

github-actions bot commented Jan 20, 2025

Uh oh!

sylvestre commented Jan 20, 2025

Uh oh!

f8ith commented Jan 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

BenWiederhake commented Apr 14, 2025

Uh oh!

f8ith commented Apr 16, 2025

Uh oh!

BenWiederhake commented May 30, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

f8ith commented Dec 27, 2024 •

edited

Loading

f8ith commented Jan 24, 2025 •

edited

Loading