Dependencies resolution with `--minimal-versions`

[matklad]

View all comments

Implementation PR: #5200
Docs: https://doc.rust-lang.org/nightly/cargo/reference/unstable.html#minimal-versions
Issues: Z-minimal-versions Nightly: minimal-versions

Steps:

• Document best-practices (Expand "CI best practicies" section to the guide #5656)

Unresolved questions:

• do we want to "impose" this feature on the ecosystem? Currently, everything seems to work fine due to eager dependency resolution. Adding --minimal-versions has costs: one-time ecosystem transition cost, cost to run CI job for minimal versions, cost to actually update minimal versions. There's anecdotal evidence that wrong minimal versions actually are a problem: https://www.reddit.com/r/rust/comments/8ob598/rust_minimum_versions_semver_is_a_lie/e027mtz/.

• should we implement "--minimal-versions-for-me-but-not-my-dependencies" as well, to make the initial roll-out of this feature easier?

Stabilization TODO:

• change -Z minimal-versions to just --minimal-versions and add it alongside --frozen/locked,
• this will require community-wide effort to make the things work, so a special announcement should be prepared. Announcement should describe the links problem and solution.

[Eh2406]

I would like to cc some persons:

• @klausi you did the implementation and many of the prs to get CI using it, including hear on cargo.
• @illicitonion you wrote up a great blog post on kicking the tyres on this feature.
• @matklad for your, already linked, articulit descriptions of the downsides of "imposeing" this on the ecosystem.
• @alexcrichton as you have encouraged some care when it comes to rolling out this feature.
• @dtolnay as you have seen real world bugs caused by not having this.
• Who did I miss that holds strong views for/against this?

I'd like to see us come to a shared understanding of what we would recommend users do with this feature both at rollout and at steady state in the idealized future.

• A comically-pro-this-feature opinion thinks a crate is only valid if it can be built with minimal-versions, so we should check at publish time with no override, and we should yank all existing crates that don't meet this standard.
• A comically-against-this-feature opinion thinks that the meaning of Cargo.toml is what cargo dose with it, so if ">=0.0" as a version requirement gets cargo build to work than what is the harm, So we should not stabilize this feature.

I think we are all in the middle, and probably closer than we think, we just need to articulate it. To start it off I will propose a position as a strawman, so you can point out where I am wrong.

All crates should try minimal-versions. If it can be made to work it should be added as a separate test in CI. As then things that depend on your crate will be able to use minimal-versions easily. yanking is generally not called for, only to be used when it predates rust 1.0.0 and it is a real problem for the ecosystem. If CI resources are tight, then minimal-versions can be combined with one of the other runs, like the minimal rust version or the beta run.

[klausi]

I'm currently not that interested in how strongly we should impose this option on the ecosystem. First we should make the minimal-versions option work properly. Not even cargo itself can be compiled with minimal-versions as explored in #5275.

Repeating the open points here:

1. cargo publish always needs to send the links key to the registry. This needs to be stabilized as default behaviour in stable cargo.
2. Mass update the crates.io registry to add the links key where it is missing right now.
3. Stabilize the -Z minimal-versions flag into --minimal-versions on Cargo stable

[Eh2406]

Well one small peace is done. At this time cargo publish from stable pushes the links key.

[klausi]

Oh cool, that is good news!

The inconsistent crates.io registry is still a problem. @alexcrichton said in rust-lang/crates.io#7310 that getting minimal-versions to work smoothly was not a priority then, so he didn't want to mass update the creates.io index. Is that still the assumption?

@SimonSapin argued that people will always use old cargo publish versions and by that make the crates.io registry inconsistent again. In order to prevent that we would have to implement server side rules on crates.io to prevent entries without links attributes. Which means breaking those old cargo publish versions.

For experimenting I'm maintaining a crates.io registry fork, that I update very irregularly. It has the links attribute fixed for some popular crates such as curl-sys.

CARGO_REGISTRY_INDEX=https://github.com/klausi/crates.io-index cargo build -Z minimal-versions

[Eh2406]

Is that still the assumption?

That is one of the questions I think we need to address. :-) Hopefully, @alexcrichton and @matklad will have a chance soon to articulate what there current thoughts are.

What we decided to recommend needs to work and fairly smoothly. That means that if there are hard changes that need to be made to make it work then we need to be willing to do them. Correspondingly if we are not willing to make the big changes then we need to recommend something smaller.

Big changes may include some or all of:

• foundational and otherwize stable crates releasing new versions just to bump minimal dep.
• yanking versions too old to be worth making a new release for.
• manually updating the index to add the links attribute, and requiring it to be correct for all future uploads.
• other ideas?

Recommend something smaller may include some or all of:

• recommending a --minimal-versions-for-me-but-not-my-grand-dependencies
• endorsing a community maintained fork of the index that simulates the "Big changes"
• making it clear that this is an optional "nice to have" and that it is not worth doing unless your dependencies are already doing it.
• having some kind of log of things cargo discover do not work, that gets included into the index if they get tried again. (this would require a lot of design work.)
• other ideas?

[alexcrichton]

I think we'll definitely want to get this working with the main index, and I think we probably just want to keep going as-is, fixing up crates and publishing them as we discover mismatches. In that sense I think it might be good to do some more work to get some of the "base crates" working and then perhaps make a post on internals asking for testers so we can discover new crates and publish new versions

[illicitonion]

I got a non-trivial crate (https://github.com/pantsbuild/pants/tree/c8b42cb52eca9acbca98eaaf9599a47bc7b1f51e/src/rust/engine) compiling with -Z minimal-versions, and have started sending out a few PRs to the ecosystem.

My two big questions from this process are:

1. What version of the log crate should libraries depend on? Nothing before 0.3.4 builds with post-1.0 rust. protoc depends on 0.* (https://github.com/stepancheg/rust-protobuf/blob/264debff3cd1cb26048925e90bec998941c2a328/protoc/Cargo.toml#L17), and many crates depend on 0.3. Most things which depend on 0.3 work with 0.4 It would be great to have some published advice on this one, as it's a slightly weird dependency in the first place. log itself notes some compatibility guidelines (https://docs.rs/log/0.4.2/log/#version-compatibility), but it would be nice to expand that to a firm suggestion ("You should always depend on the most recent version where possible" or "You should always depend on the oldest version which works for your library", or something else)
2. If we decide that --minimal-versions is something people should strive to support, how should people handle dependencies which aren't actively maintained, and which need updating? If a transitive dependency is the only problem with an otherwise functioning dependency which hasn't been touched in 2-3 years, and which doesn't respond to PRs, what should people do? Fork the crate? Try to get the bad transitive dep yanked? Give up?

[alexcrichton]

@illicitonion nice!

Those are indeed good questions too :). You can somewhat force the process by having some crate have a higher version bound (aka requiring 0.3.10 of log synthetically) but that's not a great solution to either problem. I think for now the best advice we'd have is "try to send a PR to the crate and get a new version published" but that indeed reduces the usability of this feature :(

[Eh2406]

Actually I quite like the suggestion,

If you minimal-versions build is broken by log then add a dependency on 'log = "3.10"' to your Cargo.toml.
If you can't do to your crate already depending on 'log = "4"' then depend on the helper crate 'logs-that-works-with-minimal-versions = "1"' witch is just a Cargo.toml that has 'log = "3.10"'.

I don't think "try to send a PR to the crate and get a new version published" reduces the usability, I think it is begging the question "If a transitive dependency is the only problem with an otherwise functioning dependency, and which doesn't respond to PRs, what should people do?".