Flakes using paths do not infer `git+file:` despite documentation to that effect

[lilyball]

Describe the bug

nix help flake explains that path:<path> flakes are equivalent to git+file://<path> flakes with an inferred dir parameter if the path or any of its parents is a git repository. This does not appear to be true in general. Paths without the path: prefix given to nix commands as arguments do infer the git+file type, but arguments with an explicit path: don't, nor do paths specified in the nix registry or given to builtins.getFlake.

So there's two issues here:

1. The documentation says path:<path> and git+file://<path> are roughly equivalent, except they're not as the path: flake copies the entire referenced source directory whereas git+file: copies the git worktree. This means the former will include untracked files that the latter skips (including the .git directory itself), the metadata is different, and parameters like ?rev= don't work (see builtins.getFlake doesn’t use git+file protocol if available #5670).
2. The documentation says that specifying just <path> by itself is the same as path:<path> (in that the specified URL form here is [path:]<path>(?<params>)?). This is true for builtins.getFlake and for the registry (and presumably for flake inputs but I haven't tested that variant), but is not true for arguments given on the command-line.

I do think that it's useful to have path: and git+file: behave differently for the same path, and so writing path:<path> explicitly should behave as it does now. But the bare absolute path form should infer git+file: everywhere instead of just when used as CLI arguments, and the documentation should be updated such that it does not claim that path:<path> is equivalent to git+file://<path>, does not list the path: prefix as optional, and includes explicit documentation about how bare paths are interpreted.

Steps To Reproduce

1. Navigate to a directory on your local filesystem that contains a flake.
2. Run nix flake metadata $PWD
3. Run nix flake metadata path:$PWD
4. Run nix flake metadata git+file://$PWD
5. Run nix registry add foo $PWD && nix flake metadata foo
6. Run nix repl
7. In the repl, run builtins.getFlake "/path/to/current/dir"
8. In the repl, run builtins.getFlake "path:/path/to/current/dir"
9. In the repl, run builtins.getFlake "git+file:///path/to/current/dir"

Expected behavior

Every means of evaluating a flake using an unprefixed path should behave identically to evaluating the flake using the git+file:// prefix (including resolved and locked URLs and other metadata). The flake evaluations using an explicit path: prefix should evaluate differently.

nix --version output

nix (Nix) 2.5.1

[thufschmitt]

The documentation is indeed wrong, but not exactly as you say: The intended (and as far as I can tell observed) behavior is that

• path:/some/dir will take the content of /some/dir as input
• git+file:///some/dir will fetch the git repo at /some/dir and use that as input
• /some/dir (or a relative path) will try to infer whether /some/dir is in a git repository and behave like path or git+file depending on that

This seems to be consistently the case in the CLI, in flake inputs and in builtins.getFlake (didn’t try the registry but it should be the same).

Does that behavior make sense to you?

[lilyball]

This is not my observed behavior with Nix 2.5.1.

At the CLI, /some/dir infers path: or git+file://. But this is not true of anywhere else. Here's a demonstration using a hacky flake that lets me inspect the flake with nix eval:

❯ ls
bar		flake.nix
❯ git status --short
 M flake.nix
?? bar
❯ cat flake.nix
{
  outputs = { self }: {
    legacyPackages.x86_64-darwin = self.sourceInfo;
    defaultPackage.x86_64-darwin = self.sourceInfo;
  };
}
❯ ls $(nix eval --raw .#outPath)
warning: Git tree '/Users/lily/Dev/Scratch/foo' is dirty
flake.nix
❯ ls $(nix eval --raw --impure --expr 'builtins.getFlake (toString ./.)')
bar		flake.nix
❯ nix registry add foo $PWD
❯ ls $(nix eval --raw foo#outPath)
bar		flake.nix
❯ nix registry list | grep foo
user   flake:foo path:/Users/lily/Dev/Scratch/foo

As you can see, I have a git repo with a tracked flake.nix and an untracked bar. If I examine the resulting outPath when using . at the CLI, it's just flake.nix, which demonstrates that it's using git+file://. If I examine the results of builtins.getFlake (toString ./.) then bar is present, demonstrating that it's using path:. And nix registry add foo $PWD also does path:, as can be seen both from the eval and from the resulting URL printed by nix registry list.

Besides the filesystem tree, the behavioral difference of unprefixed paths at the CLI vs builtins.getFlake can also be demonstrated by looking at the sourceInfo (e..g the presence of keys such as lastModified, submodules, rev (if my tree were not dirty), etc). And of course the fact that nix eval . produces a dirty warning whereas the builtins.getFlake version doesn't. Also the use of relative vs absolute paths doesn't matter (e.g. I could use $PWD instead of . at the CLI for the same behavior; builtins.getFlake needs the absolute path hence the use of toString ./. as a more convenient alternative to interpolating $PWD).

So my original assertion remains true. /some/dir can only ever infer git+file:// when used at the CLI, and cannot do so when used with builtins.getFlake or with nix registry. As well as the fact that the docs say that path: is equivalent to git+file:// in git repos, which is completely wrong.

[thufschmitt]

Oh you’re right, I tested things a bit too fast. So https://gist.github.com/thufschmitt/5ad00ed77d751b4db309b666d448f779 should test all the interesting behaviors of passing a direct path as a flake url.

As you say, builtins.getFlake and the registry always treat /foo/bar is a path: url, but the CLI and flake inputs (not going through the registry) will try to use a git+file input.

Regardless of the documentation issue (which will have to be fixed ofc), I’m actually not sure what the correct behavior should be: On the one hand, having /foo/bar infer a git url is very handy, but otoh there’s a lot of value in enforcing explicitness for this kind of things outside of the CLI. So I think there’s three possible paths:

1. Keep the current behavior. I’m not utterly fond of the fact that inputs.A.url = "/foo/bar" and inputs.A.url = "A" where A is a registry entry pointing to /foo/bar behave differently but it’s not atrocious as long as we fix the documentation;
2. Make all the places try to infer what /foo/bar is. Nice and unified, but maybe a bit too magical;
3. Only leave the magic on the CLI, and enforce the use of git+file on flake inputs, the registry and getFlake. Slightly more verbose, but explicit and straightforward

[lilyball]

I would prefer to have a bare /foo/bar infer wherever possible. Having flake references in the CLI behave differently than elsewhere is a recipe for confusion. In addition, inferring path:/foo/bar when inferring git+file:///foo/bar is possible is a footgun, as it can mean accidentally copying things like build output into the nix store. This is actually the specific behavior I ran into that motivated this issue in the first place.

So I think there's two tasks here:

1. Update the documentation accordingly. This includes both changing the docs on path: to stop claiming to be equivalent to git+file: but instead to have a separate doc entry for unprefixed /path/foo that explains how it infers. Also perhaps make a note on nix registry add that unprefixed paths are resolved at the time the entry is added to the registry rather than when they're resolved.
2. Everywhere that we currently infer /foo/bar → path:/foo/bar we should run the logic to check for git+file: too.

[thufschmitt]

inferring path:/foo/bar when inferring git+file:///foo/bar is possible is a footgun

Mh you’re right, it’s not just about being more strict outside of the CLI, it’s a different semantics. So yes, it’d probably make sense to infer things the same way everywhere. @edolstra any thoughts on making /foo/bar possibly resolve to git+file:///foo/bar everywhere? Alternatively, we could simply forbid the path syntax everywhere but in the CLI, and enforce an explicit URI scheme

[erikarvstedt]

+1 for letting builtins.getFlake handle flake paths like the CLI.
As already mentioned, this is a footgun. It has affected me personally and I've seen some nix-based projects failing to use getFlake correctly.
For example, the repl command from flake-utils-plus always the copies the user's system flake into the store, including its version control dir (like .git), when starting a REPL.
This leads to degraded startup performance and is caused by getFlake being called with a regular path that is created here or here. Fixing this would entail detecting the version control system of the user's system flake and constructing an appropriate flake URL, which is very inconvenient.

[erikarvstedt]

I've just noticed that path inputs (inputs.myinput.url = "/my/path") in flake.nix files are handled like in the CLI, i.e. they are automatically resolved to type git.
It really makes sense for getFlake and registries to work the same way.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/flakes-re-locking-necessary-at-each-evaluation-when-import-sub-flake-by-path/34465/7

[schmittlauch]

I found a scenario where the interpretation of whether url = "a/relative/path/flake.nix"; is interpreted as url = "path:a/relative/path/flake.nix"; or not. If it resolves to an absolute nix store path reference that is stored in the lock file, this can lead to infinite re-locking behaviour at each new evaluation.
As this is a rather specific and possibly interwoven phenomenon, I opened #9339 as a dedicated issue.

[lf-]

equivalent lix bug https://git.lix.systems/lix-project/lix/issues/28