Tests fail in unpacked crates.io crate tarball, due to missing testdata

[sourcefrog]

Debian seems to build and run tests from the package uploaded to crates.io, which is reasonable enough. At the moment I exclude the testdata but this is a good reason not to. Anyhow, crates.io provides a kind of supplementary archive of the history of the project so it would be good if it's complete.

That would allow dropping https://sources.debian.org/patches/rust-cargo-mutants/23.10.0-1/disable-tests-missing-testdata.patch/

(cc @jelmer who seems to have kindly packaged, including finding this workaround, and uploaded it.)

However, this is more tricky than I expected because most of the testdata is comprised of example Rust trees and

https://doc.rust-lang.org/cargo/reference/manifest.html#the-exclude-and-include-fields:

Regardless of whether exclude or include is specified, the following files are always excluded:

• Any sub-packages will be skipped (any subdirectory that contains a Cargo.toml file).

We could somehow hide the testdata from cargo-package (e.g. renaming the files) but that seems like it would make development on the tree less straightforward.

We could potentially add a feature flag or other mechanism to let the tests pass if testdata is missing, and test from CI that this work.

Overall I'm not sure there's any much better option than manually cutting out the tests.

[sourcefrog]

Closing for now; can reopen if anyone has any better ideas. Maybe the simplest would be for Debian to fetch the github release tarball instead.

[jelmer]

👋

I did indeed package mutants (and some of the dependencies that were not yet packaged) for Debian - I use mutants regularly, it's one of the things I really liked in the Google3 ecosystem.

It would indeed be good if we could run more of the tests, or even if running tests from the crates.io archive works out of the box (whether that means running all tests or just some).

That way Debdan don't have to carry a patch (that will break as things change upstream), and it will make things easier for other packagers or people simply running from crates.io as well.

[sourcefrog]

I really liked it there too.

Maybe a good approach is: every test that wants testdata should just immediately succeed if the whole directory is missing. (I wish rust tests had a way to report "skipped" but I don't think they do at present.) In general that kind of pattern makes me concerned that some change will cause the test to succeed early even when it shouldn't, but after all we have cargo-mutants to catch that.

[jelmer]

Fetching from GitHub is technically possible but nontrivial - Debian has a fair amount of automation to package the 2000+ crates we package, and it's all based around crates.io today. Definitely something we could consider though.