Submodules of flakes are not working

[faulesocke]

Describe the bug
Having git submodules in a git repository I cannot access the submodules in my flake.nix. This is especially annoying when I want to provide a flake.nix for software that is split across multiple git repositories that are all included as submodules in one main repo.

Steps To Reproduce

git init top
git init sub

cd sub/
echo "foobar" > foobar
git add foobar && git commit -m foobar
cd ..

cd top/
git submodule add ../sub sub
git commit -m submodules
git add flake.nix

And top/flake.nix is

{
  outputs = { self, nixpkgs }:
  with import nixpkgs { system = "x86_64-linux"; };
  {
    defaultPackage.x86_64-linux = stdenv.mkDerivation {
      name = "foobar";
      src = self;
      installPhase = ''
        cat sub/foobar > $out
      '';
    };
  };
}

Then run nix build.

Expected behavior
I expect result to contain the string foobar.

Actual behaviour

% nix build
warning: Git tree '/tmp/top' is dirty
error: builder for '/nix/store/wlxmp43q9q13477yjc2hvzqja8ng3sgf-foobar.drv' failed with exit code 1;
       last 10 log lines:
       > unpacking sources
       > unpacking source archive /nix/store/kzsvnqmnjrpyvma444jaz3zv3kgry8az-source
       > source root is source
       > patching sources
       > configuring
       > no configure script, doing nothing
       > building
       > no Makefile, doing nothing
       > installing
       > cat: sub/foobar: No such file or directory
       For full logs, run 'nix log /nix/store/wlxmp43q9q13477yjc2hvzqja8ng3sgf-foobar.drv'.

nix-env --version output
nix-env (Nix) 2.8.1
I also tried 2.8.0 and 2.9.1. The submodule is just missing and I see no way to access it.

Additional context
Also manually adding the submodule as an input like so

{
  inputs.sub = {
    url = "./sub";
    flake = false;
  };
 ...
}

and then changing the src = self to src = sub in the outputs, just results in a weird error message instead of the submodule just working as it should:

% nix build
warning: Git tree '/tmp/top' is dirty
error (ignored): error: end of string reached
error: getting status of '/nix/store/ms70asysgv7alri3972wpzpd4r6ivbiz-source/sub': No such file or directory
(use '--show-trace' to show detailed location information)

Especially the error (ignored): error: end of string reached line is worrying me and probably show that this is actually a bug and should work. Of course this could also just be my own stupidity and I'm doing something completely wrong.

[n8henrie]

Working for me if you include the submodules=1 flag.

$ nix build .?submodules=1
$ cat result
foobar

[FlorianFranzen]

Even with submodules=1 is still very weird to use submodules efficiently. I started using submodules as separate input to improve cache invalidation, but both ways I found have major caveats:

Either you just use the submodule path:

inputs = {
    submodule = {
      url = "./path/to/submodule";
      flake = false;
    };
}

However, this updates the lockfile on each invocation and using the flake in the repl is cumbersome.

Or you use it as a git file input. That solves the repeated locking but just ignores it the referenced submodule is dirty:

inputs = {
    submodule = {
      url = "git+file:./path/to/submodule";
      flake = false;
    };
  };

[n8henrie]

Huh, I'm not referencing the submodule (directly) as an input at all and it's been working well for my use case (maintaining my nix-darwin flake-based config containing a vim submodule that references my "pre-nix" vim config). Haven't messed around with seeing what happens if vim is dirty though.

[AshtonKem]

Ahhhh! This just got me too. nix build .?submodules=1 worked for me, is there any way to configure this permanently in flake.nix?

[n8henrie]

• nix flakes: add support for git submodules #4423
• feat: support managing submodules via inputs.self #7862

[AshtonKem]

Phenomenal, thank you!

[doronbehar]

Does anyone else here experiences that for a nix build that requires submodules, if your tree is dirty the submodules are not checked out in the builder's src = self?

NOTE: I am using nix build -L .?submodule=1\#$PKG

[doronbehar]

Answering my own question: If the tree is dirty, and at least 1 git submodule is not checked out, the tree has to be clean for Nix to pick up the missing submodule using the ?submodule=1 flag. Makes sense sort of, but indeed it's a confusing. hopefully things will improve with #7862.

Worth noting for people reading to the end in future: it's ?submodules=1, not submodule in the singular.