Support self-fetching parameters in flake.nix

[nrdxp]

#5284 happened, so instead I'll open a proper issue to address what I feel would be the "correct" solution since I am incapable of implementing it atm.

I see no particular reason why self should be especially handicapped from the other inputs, but should be able to be easily handled as a regular git type input, or whichever active VCS Nix can detect.

you wouldn't even have to change the syntax, but simply allow inputs.self.submodules = true, for example.

Nix would simply have to know that inputs.self is the current self.

[edolstra]

There is a very good reason why self is different: we have to fetch the self flake before we can read its flake.nix. So any attributes like submodules have to be known in advance.

[nrdxp]

There is no possible mechanism to read the flake inputs before self is fetched? Like maybe just pulling the lock file into the Nix store for reading before the rest of the flake?

[mstone]

@nrdxp — I’ve been thinking about this a bit because it’s going to bite everyone who tries to flakeify upstream projects that use submodules, like, in my case, qemu.

So far the best idea I’ve come up with is to write the flake to use an auxiliary input, in addition to self, that fetches the same underlying repo just with submodules enabled.

Unfortunately, that still leaves several quality-of-life problems that I wonder if we can fix:

a) that the self input is fetched but is unused (other than for obtaining flake.nix/flake.lock)

b) that the self-with-submodules input is “locked” when we’d rather that it be “derived from” or “tied” to self — e.g. so that we see dirty working copy edits / whatever the flake-ref resolves to when being built, and don’t have to remember to push commits that bump flake.lock just to test changes.

(P.S. — if you’re interested in this approach, I wrote more at https://gist.github.com/mstone/4218e6fa9ef98e5153f543fb8e7b4743 with several variations in this idea, though I’m not really sure that any of them are ready for prime time. :-/)

[Kha]

There is a very good reason why self is different: we have to fetch the self flake before we can read its flake.nix. So any attributes like submodules have to be known in advance.

Would this still be the case with #3121? Afaiu, in that case we would evaluate flake.nix in the local checkout/git cache and only then copy self to the store.

[edolstra]

#3121 only addresses the case of a local git tree.

The fundamental issue is that we do a fetchTree to get the top-level flake, which requires having all the attributes needed for fetching (like submodules). If the top-level flake.nix can then change those attributes, we may have to do another fetchTree. Also the semantics get pretty ugly, since you could do something like inputs.self.url = some-arbitrary-url.

[kero0]

Am I missing something? Or why can't the answer just be as simple as manipulate self with command line args? So have a --fetch-submodules arg and add more as they are requested.

[nrdxp]

Command line args would work in theory, but if I'm conceptualizing it correctly, it would also break purity since the end result would be vastily different depending on the args passed.

[mstone]

While I like them for other reasons, I disagree that CLI improvements are sufficient here: — instead, I think the point here is that flake-authors need a way to specify all build instructions, including what code to build, as part of the flake definition — that way, end-users get something that “just works” without more detailed communication from the flake authors (e.g. of a special URL to build, or extra CLI arguments, etc.) whenever possible.

[kero0]

I think that it's inevitable that flake authors need to communicate with flake users.

As for purity or consistency, I don't see many scenarios in which someone would have a requirement for a flake and the flake would still build without it. A build that is missing files will pretty much always fail to build. If your building a top level flake you should be checking the docs released by the author and use the proper args.

[nrdxp]

IIRC, more general flake arguments passed from the cli to the flake builder were considered early on, similar to how one can pass arguments to nix expressions with the old nix-build, but were never implemented because of the same purity concerns. So, unless I misunderstood, I dont' see cli args being accepted as a solution.

[edolstra]

In principle you should be able to use a flake URL like nix build git+file:///path?submodules=true, but it looks like we're not handling the submodules parameter in URLs yet...

Also this wouldn't be very user-friendly

[mstone]

@edolstra, @nrdxp, @kero0 -- given the discussion above, I'd like to try to summarize what I think we've collectively concluded so far.

Please let me know if you think I've missed anything critical or if you think I've misstated/mischaracterized any of the following facts:

1. most flakes work fine without automatically-fetched git submodules.
2. fetching git submodules by default caused a bad performance regression, fixed by reverting that behavior via Revert "Merge pull request #4922 from nrdxp/default-submodules" #5284.
3. fetching git submodules by default is unwanted by some flake authors in any event.
4. nevertheless, a small but important set of flake use cases (specifically including flakeifying upstream projects that already use submodules) would benefit from giving flake authors an ergonomic way to tell flake-users' flake interpreters that the flake's source code requires submodule fetching.

Additionally, I think we seem to be orbiting the following "nascent UX principles" (though with some conflicting opinions about the latter two):

5. "flake users should be able to use query-string parameters to control the data of the FlakeRef of the InstallableFlake they are manipulating -- what we have, in earlier parts of the conversation, referred to as the notional 'inputs.self fetchTree data'."

6. (advocated by me:) "Flake authors should be able to ergonomically provide data and instructions to flake users via flake.nix in order to make the use cases supported by the nix CLI "just work". Today, this is done primarily for CLI commands like nix search, nix build, nix run, nix flake show, etc. via 'well-known attributes' like outputs, packages.<system>, defaultPackage."

7. (or alternately, paraphrasing @kero0): "Flake authors need to communicate with flake users. If your building a top level flake you should be checking the docs released by the author and use the proper args."

Finally, I think we are juggling the following emergent design constraints / conclusions:

8. The facts above prohibit us from fetching submodules by default for every git-based flake.

9. Accepting (6) above should discourage us from requiring that flake-authors communicate submodule-fetching requirements to end-users "out-of-band", i.e., outside of nix-meaningful instructions in flake.nix/flake.lock.

10. Accepting (7) above in combination with improving flake-ref query argument support likely would give us an answer for how flake author/user communication should work around coordinating submodule-fetching but "wouldn't be very user-friendly", per @edolstra.

11. However, all the solutions that we know about based on accepting (6) seem to require some kind of "double-fetching" (or in build system terms, "second evaluation", or in architectural style terms, "code on demand") scheme in order to enable flake.nix fetching and evaluation to happen before the particular value representing the self argument [or possibly another argument, but with self-like treatment in flake.lock] is determined.

Have I got anything wrong here?

Thanks! -- Michael

[nrdxp]

for 11, couldn't we avoid a double fetch by generating lock entries for submodules?

[mstone]

@nrdxp -- oooh, interesting, I'll need to think about that. I wonder if that might also make it possible to handle ?submodules=true instructions for github:...-style flake-refs...

[L-as]

My suggestion for how to fix this:

Flakes could have a metadata attribute set in flake.nix, as such:

{
metadata.fetch-git-submodules = true;
}

The metadata attributes would also have to be in normal-form, and would not allow any path literals.

This means that Nix can fetch the flake, check the metadata, then check whether to also fetch the submodules before evaluating any Nix code.
The metadata could also be in a separate JSON file if that is better.

With this you don't have the double-fetching problem, but it's restricted to Git. This isn't "clean" IMO, but IMHO the problem is Git itself not always fetching submodules. Git should have from the beginning enforced fetching submodules, and we have to work around that now.

The only downside with always fetching submodules in Nix is that it breaks backward compatibility in some subtle ways, and doesn't work well with git-crypt.