nix flakes: add support for git submodules

[DanilaFe]

I recently learned about Nix Flakes, and I tried out making my blog a perfectly reproducible build. The trouble is, my blog uses a Git submodule to reference some code that I then present in my blog posts. This is fine and all, but I run into an issue - although I can fetch my blog source using Flakes' input attribute, it doesn't fetch submodules! This means that the code that I reference in my build process is simply not there.

This seems to be almost possible; from a brief glance around Nix's source code, I discovered that a flake input gets turned into a "regular" input, which can point to a git repo and have a submodules attribute that, I think, would do what I want. It feels as though it was intended that this would work, but it doesn't.

It doesn't work because, for a reason unknown to me, a flake input's attributes must all be strings (except for flake). Here's the code that rejects non-string arguments:

nix/src/libexpr/flake/flake.cc

Lines 123 to 127 in 724b7f4

	if (attr.value->type() == nString)
	attrs.emplace(attr.name, attr.value->string.s);
	else
	throw TypeError("flake input attribute '%s' is %s while a string is expected",
	attr.name, showType(*attr.value));

If non-string arguments were not rejected, a submodules field could be forwarded to the Git input scheme, which I suspect would make this work. Is there a reason that Flakes rejects all non-string "third party" attributes? It puts them into an attribute set anyway. Right now, I'm falling into a rather amusing loop:

If submodules is a string, it's rejected by the Git input scheme. But if it's a bool, it's rejected by the Flake code.

[knedlsepp]

I think this is a duplicate of #4357

[blaggacao]

Apparently closed by #4435

[knedlsepp]

I think there is still one piece missing: the flake itself is not being fetched using submodules.

[DanilaFe]

Yes, perhaps it's worth closing this issue, and opening the original one (since this is a duplicate)?

Also, I am not sure how to go about requiring a flake itself to pull its own submodules. If a flake only has inputs and outputs, and an external definition determines the method for downloading it (a flake can be a local folder, too, or a tarball, can't it?), then where's the compromise?

A flake is assumed to be a git repo, so it would make sense for a universal "fetch submodules" option, perhaps as an output attribute. This would be orthogonal to the type = "git"; submodules = "true" attributes inside an input, since it wouldn't work for non-flake inputs.

[blaggacao]

What would be a good & concise way to present the design decision abstract primitives still in question to @edolstra / the maintainers? (possibly on another issue for proper framing)

[L-as]

@knedlsepp

I think there is still one piece missing: the flake itself is not being fetched using submodules.

I made an ugly patch where submodules are always fetched: L-as@ce0a547

The idea is that if a flake has submodules, they should probably always be fetched (why would you add them otherwise?).
My C++ skills are subpar though, and my patch may just make it fetch submodules for all git flake inputs (haven't tested), but it works for my case.

[ruler501]

@L-as yeah that's just turning it on for everything. If you look at the constructor for one of the classes for the nix develop command it has a default url of . making that instead be .?submodules=1 or whatever the correct format for that url is should cause it to only affect the flake in the repo the command is running in.

[L-as]

It has to work with all nix commands though, and I can't see the use in being forced to manually write ?submodules=1 every time.

[ncfavier]

The idea is that if a flake has submodules, they should probably always be fetched (why would you add them otherwise?).

Consider a public NixOS config repo with a private submodule. One may want to be able to build the config even without access to the private parts.

[L-as]

You could make it so that failure to clone the submodules is a non-fatal error.

[bqv]

The idea is that if a flake has submodules, they should probably always be fetched (why would you add them otherwise?).

Consider a public NixOS config repo with a private submodule. One may want to be able to build the config even without access to the private parts.

I'm not sure that's desirable behaviour. It's nondeterministic, then