Flakes: Re-locking necessary at each evaluation when import sub-flake by path

[schmittlauch]

Describe the bug

I have a flake dev-env that itself references a sub-flake myNixpkgs from the same local git repo by relative path.
As the sub-flake input is also locked in the main flake.lock file, its path is recorded in there.

Unfortunately I've discovered that each evaluation of the flake now changes the flake.lock file again.

$ nix develop
warning: Git tree '[…]' is dirty
warning: updating lock file '[…]/flake.lock':                                                                                                           
• Updated input 'myNixpkgs':                                                                                                                                                                 
    'path:/nix/store/whdbxck4f86n3244v9pqmvl81p1qsg65-source/flakeSupport/myNixpkgs?lastModified=1&narHash=sha256-6+DqW6+6Bf1Ea3t8SaGhB9zfx2fkl3ZtolWM8LC7Oto=' (1970-01-01)
  → 'path:/nix/store/3y3cfmdy6vnv4pgl30r883j37p98nikv-source/flakeSupport/myNixpkgs?lastModified=1&narHash=sha256-6+DqW6+6Bf1Ea3t8SaGhB9zfx2fkl3ZtolWM8LC7Oto=' (1970-01-01)
warning: Git tree '[…]' is dirty

$ nix develop
warning: Git tree '[…]' is dirty
warning: updating lock file '[…]/flake.lock':
• Updated input 'myNixpkgs':
    'path:/nix/store/3y3cfmdy6vnv4pgl30r883j37p98nikv-source/flakeSupport/myNixpkgs?lastModified=1&narHash=sha256-6+DqW6+6Bf1Ea3t8SaGhB9zfx2fkl3ZtolWM8LC7Oto=' (1970-01-01)
  → 'path:/nix/store/69skdc3dhdmf5g0iq7gh6r7pr2jic1r7-source/flakeSupport/myNixpkgs?lastModified=1&narHash=sha256-6+DqW6+6Bf1Ea3t8SaGhB9zfx2fkl3ZtolWM8LC7Oto=' (1970-01-01)
warning: Git tree '[…]' is dirty

Especially with direnv this is a no-go behaviour, as the flake is re-evaluated after each command run in the shell.

The reason for this is quite clear: For pure evaluation, the whole git repo of the flake is copied to the nix store. Its out path there depends on the repo's content. So the following happens:

1. nix copies the whole repo to store
2. nix detects a path mismatch of the relatively referenced flake file
3. nix updates the path in the lock file
4. this causes the repo contents to change…
5. …causing the out path of the repo to change when it is copied to the store at the next evaluation. GOTO 1.

I mainly wonder whether I am holding this wrong and there is a better solution to this? Or is this still one of the cases why flakes are experimental and the issue about relative flake path references is still open? #3978

I'd be so bold to say that my approach is a rather obvious one, so even if there is a better approach, it is easy to do things the wrong way.

---

the flake.nix at the repo top-level:

{
  inputs = {
    myNixpkgs = {
      url = "./flakeSupport/myNixpkgs";
      inputs.nixpkgs.follows = "nixpkgs";
    };
    nixpkgs = {
      #url = "github:NixOS/nixpkgs/nixos-23.05";
      url = "nixpkgs";
    };
    nixpkgsOld.url = "github:NixOS/nixpkgs/nixos-21.05";
    poetry2nix = {
      url = "github:nix-community/poetry2nix";
      inputs.nixpkgs.follows = "myNixpkgs";
    };
  };

  outputs = { self, myNixpkgs, poetry2nix, ... }:
    let
      inherit (myNixpkgs) lib;
      supportedSystems = [ "x86_64-linux" "x86_64-darwin" "aarch64-linux" "aarch64-darwin" ];
      forAllSystems = lib.genAttrs supportedSystems;
      pkgs = forAllSystems (system: myNixpkgs.legacyPackages.${system});
      p2n = forAllSystems (system: poetry2nix.legacyPackages.${system});
    in
    {
      devShells = forAllSystems (system: {
        default = pkgs.${system}.mkShellNoCC {
          packages = with pkgs.${system}; [
            (p2n.${system}.mkPoetryEnv poetryCommonOpts.${system})
            pkgs.${system}.python38
            poetry
            openssl_1_1
          ];
        };
      });
    };
}

The sub-flake flakeSupport/myNixpkgs/flake.nix:

# vaguely inspired by https://github.com/numtide/nixpkgs-unfree/blob/9545d844027c1b91b14b19d225856efc931b22b2/flake.nix
{
  description = "nixpkgs reexported with some required configuration, e.g. permitted insecure packages";

  outputs = inputs@{self, nixpkgs}:
  let
    inherit (nixpkgs) lib;
    # re-use same supported systems as in the upstream nixpkgs
    systems = lib.systems.flakeExposed;
    forEachSystem = lib.genAttrs systems;

    nixpkgsConfig = {
      permittedInsecurePackages = [
        "openssl-1.1.1w"
      ];
    };
  in
  nixpkgs // { legacyPackages = forEachSystem (system:
    import nixpkgs {
      inherit system;
      config = nixpkgsConfig;
    });};
}

Expected behavior

As long as the referenced sub-flake file does not change, the lock file shall not change as well and thus no re-evaluation or automatic re-locking shall be necessary.

nix-env --version output: nix-env (Nix) 2.13.6

Additional context

This issues has been initially discussed in the NixOS Discourse forum. We came to the conclusion that this behaviour can be seen as a Nix bug.
Possible workarounds are referencing the sub-flake either implicitly via git+file: or explicitly marking it as a path:.

This gives links to some other issues:
The general use case of referencing sub-flakes by path is discussed in #3978.
The actual interpretation of the path to the sub-flake is an important cause of the issue, as the plain relative path not explicitly marked as path: resolves to the full absoulte nix-store path of the flake's repo:

url = "path:a/relative/path" causes this path to still land as relative in the lock file, while url = "./a/relative/path/" causes it to be converted into an absolute path at locking time – which can change and then result in a behaviour as I've described.

But how does this fit the nix docs?

Flakes corresponding to a local path can also be referred to by a direct path reference, either /absolute/path/to/the/flake or ./relative/path/to/the/flake (note that the leading ./ is mandatory for relative paths to avoid any ambiguity).

The semantic of such a path is as follows:

If the directory is part of a Git repository, then the input will be treated as a git+file: URL, otherwise it will be treated as a path: url;

This problem is already discussed in #5836.

Priorities

Add 👍 to issues you find important.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/flakes-re-locking-necessary-at-each-evaluation-when-import-sub-flake-by-path/34465/10

[tomberek]

When referring to a relative subflake, it seems there are a few related design issues:

• subflake is fully independent - contents can be part of another commit, own lockfile
• subflake is independently locked - contents match the root, but has its own lockfile
• subflake is dependently locked - re-uses some/all of the lock entries from the parent

There are use cases for each of these, but we use the same name for all of them, and the features and behavior is different. The initial design was more along the first, but then people want to use it via the second approach.

Perhaps we try to clarify the various cases, give them names, define the expected behavior, and document the recommended way for users to achieve it. We may learn along the way that there is a way to unify them, or simplify the cases.

Another way to talk about them:

1. Subflake is a reference only.
2. Subflake is part of the same git commit
3. Subflake shares lockfile with parent.

[schmittlauch]

The initial design was more along the first, but then people want to use it via the second approach.

For this, it might make sense to clarify what I actually wanted to achieve there: I was trying to re-configure nixpkgs to permit some insecure packages, and then ensure that my flake and the poetry2nix flake actually use this configuration instead of using their internal nixpkgs instance. nix-community/poetry2nix#1354

This was the only idea I came up with, and at least it seems possible within the flake semantics.
So after going through the different cases, we might also be able to figure out cases that are somehow possible either way, but should really be done only in one of the ways.