Add `loom` testing

[kpreid]

We are doing somewhat tricky things with locks, atomics, and various sorts of wakeups. There could be stealthy bugs that only show up rarely. We should use loom to test for such bugs. (This will require writing explicitly multi-threaded test cases that we don’t currently have, too.)

[kpreid]

Observations from a first experiment: loom has restrictions that are significant to nosy.

• It does not support sync::Weak, so we cannot use loom to fully test anything we do with Arc (e.g. dropping Gates).
• Its Mutex and RwLock do not support ?Sized contents. This might be easy to fix? It is also something we don’t strictly need to support ourselves, but “for testing” isn’t a great reason to change that. We could change the bounds under cfg(loom) but that’s a pain to do.

[kpreid]

Its Mutex and RwLock do not support ?Sized contents

This was fixed for Mutex only in loom 0.7.2. We do also use RwLock but not in a way that requires ?Sized.

[kpreid]

I managed to get a loom test running, but it reported a deadlock (I don’t know whether it is real or not) and then panicked internally and failed to write useful failure info:

Details

thread 'source_flatten' panicked at /Users/kpreid/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/loom-0.7.2/src/rt/execution.rs:216:13:
deadlock; threads = [(Id(0), Blocked(Location(None))), (Id(1), Terminated), (Id(2), Terminated), (Id(3), Terminated), (Id(4), Blocked(Location(None)))]
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

thread 'source_flatten' panicked at /Users/kpreid/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/loom-0.7.2/src/rt/thread.rs:276:39:
called `Option::unwrap()` on a `None` value
stack backtrace:
...
  22:        0x102e572f4 - core::option::Option<T>::unwrap::h76a895f6c0c8509c
                               at /Users/kpreid/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/option.rs:978:21
  23:        0x102e572f4 - loom::rt::thread::Set::active_mut::h560be83f312e8c22
                               at /Users/kpreid/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/loom-0.7.2/src/rt/thread.rs:276:27
  24:        0x102e71338 - loom::rt::object::Ref<T>::set_action::h239d28ae5fc62968
                               at /Users/kpreid/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/loom-0.7.2/src/rt/object.rs:387:9
  25:        0x102e6fd94 - loom::rt::object::Ref<T>::branch_action::{{closure}}::h5cbce326fa5f0ef1
                               at /Users/kpreid/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/loom-0.7.2/src/rt/object.rs:354:13
  26:        0x102e3e0e0 - loom::rt::branch::{{closure}}::h7127e01913d4731e
                               at /Users/kpreid/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/loom-0.7.2/src/rt/mod.rs:122:19
  27:        0x102e371d4 - loom::rt::scheduler::Scheduler::with_execution::{{closure}}::h2bec9f88e7e2219f
                               at /Users/kpreid/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/loom-0.7.2/src/rt/scheduler.rs:43:34
  28:        0x102e3a6f8 - loom::rt::scheduler::Scheduler::with_state::{{closure}}::h13fadf7935bd8ce7
                               at /Users/kpreid/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/loom-0.7.2/src/rt/scheduler.rs:131:28
  29:        0x102e44bf8 - scoped_tls::ScopedKey<T>::with::hcae036f9d695f9f6
                               at /Users/kpreid/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/scoped-tls-1.0.1/src/lib.rs:171:13
  30:        0x102e3a224 - loom::rt::scheduler::Scheduler::with_state::hf598e8446ba29113
                               at /Users/kpreid/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/loom-0.7.2/src/rt/scheduler.rs:131:9
  31:        0x102e36f30 - loom::rt::scheduler::Scheduler::with_execution::hd5ea8a48cdb8cd0a
                               at /Users/kpreid/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/loom-0.7.2/src/rt/scheduler.rs:43:9
  32:        0x102e419c8 - loom::rt::execution::h8c8e6a776f79f390
                               at /Users/kpreid/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/loom-0.7.2/src/rt/mod.rs:174:5
  33:        0x102e3d648 - loom::rt::branch::hb69613cdadc3bac1
                               at /Users/kpreid/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/loom-0.7.2/src/rt/mod.rs:121:25
  34:        0x102e6f710 - loom::rt::object::Ref<T>::branch_action::h807f23a0a2511179
                               at /Users/kpreid/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/loom-0.7.2/src/rt/object.rs:351:9
  35:        0x102e6633c - loom::rt::arc::Arc::branch::h620e256d918149ef
                               at /Users/kpreid/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/loom-0.7.2/src/rt/arc.rs:160:9
  36:        0x102e65d2c - loom::rt::arc::Arc::ref_dec::h39bf6823ea770b3d
                               at /Users/kpreid/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/loom-0.7.2/src/rt/arc.rs:113:9
  37:        0x102e68434 - <loom::sync::arc::Arc<T> as core::ops::drop::Drop>::drop::h312b5197ff4971c2
                               at /Users/kpreid/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/loom-0.7.2/src/sync/arc.rs:247:12
  38:        0x102e5fc10 - core::ptr::drop_in_place<loom::sync::arc::Arc<loom::rt::notify::Notify>>::h78c15702543fe81b
                               at /Users/kpreid/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/ptr/mod.rs:523:1
  39:        0x102de4bc0 - loom::future::block_on::h92bafc0aaf8205c8
                               at /Users/kpreid/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/loom-0.7.2/src/future/mod.rs:42:1
  40:        0x102de6048 - loom::source_flatten::{{closure}}::{{closure}}::hd67fbd2db5beeaaa
                               at /Users/kpreid/Projects/rust/mine/nosy/tests/loom/main.rs:31:13
  41:        0x102de0e58 - loom::thread::spawn_internal::{{closure}}::h26d66d11d30f81a6
                               at /Users/kpreid/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/loom-0.7.2/src/thread.rs:168:47
  42:        0x102ddd520 - loom::rt::spawn::{{closure}}::hf833bfb162728b32
                               at /Users/kpreid/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/loom-0.7.2/src/rt/mod.rs:78:13
  43:        0x102dee2a0 - core::ops::function::FnOnce::call_once{{vtable.shim}}::hc4fb27af3374c194
                               at /Users/kpreid/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/ops/function.rs:250:5
  44:        0x102e64134 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::hf0fbb89bba640728
                               at /Users/kpreid/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1993:9
  45:        0x102e3d40c - loom::rt::scheduler::spawn_thread::{{closure}}::hdb93e3ddb3c24e73
                               at /Users/kpreid/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/loom-0.7.2/src/rt/scheduler.rs:142:17
...
thread 'source_flatten' panicked at library/core/src/panicking.rs:226:5:
panic in a destructor during cleanup
thread caused non-unwinding panic. aborting.

This seems to be long-standing issue tokio-rs/loom#291 which has a PR allegedly fixing it tokio-rs/loom#297 but no review activity for years.