Fetching paths from cache.nixos.org can get 'stuck'

[raboof]

Describe the bug

Sometimes, nix gets stuck fetching paths:

$ nix-shell -p sbt
these 2 paths will be fetched (408.08 MiB download, 694.33 MiB unpacked):
  /nix/store/0kg8zm25k6av8hgf177kwnp9q5j6yrf3-openjdk-17.0.1+12
  /nix/store/wj7x8ik896h0k9r6fk0bns21xw5c18yh-sbt-1.6.2
copying path '/nix/store/0kg8zm25k6av8hgf177kwnp9q5j6yrf3-openjdk-17.0.1+12' from 'https://cache.nixos.org'...

Stalling without any network traffic.

Interestingly this seems to be somehow tied to the specific path: nix can download other paths just fine, but always gets stuck on (in this case) /nix/store/0kg8zm25k6av8hgf177kwnp9q5j6yrf3-openjdk-17.0.1+12.

It has something to do with the system state: rebooting the machine makes it possible to fetch the path again, but killing all nix-daemon processes without rebooting doesn't seem to help.

edit: it does seem to be something around nix-daemon though, because when rebooting I see systemd-shutdown waiting for several nix-daemon processes to shut down which takes quite a while.

If anyone can suggest ways to further diagnose the problem I'd be happy to try things out!

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 5.16.5, NixOS, 22.05 (Quokka)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.6.0`
 - channels(root): `"nixos-20.03pre194293.2436c27541b"`
 - nixpkgs: `/home/aengelen/nixpkgs`

(current profile is at 9f697d6, nixpkgs is at 48d63e9)

[kenranunderscore]

I'm seeing the same just now. For me lots of builds worked today, but I get stuck at /nix/store/062izp7vafcszxbip77mwya90rzkcg19-mbrola-3.3 specifically, also after killing the nix-daemon processes.

 - system: `"x86_64-linux"`
 - host os: `Linux 5.10.95, NixOS, 22.05 (Quokka)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.7.0pre20220127_558c4ee`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

[06kellyjac]

It seems to be related to remote builds. I have to sudo kill -9 the processes

https://discourse.nixos.org/t/remote-building-experienence-massively-degraded-since-2-3-months/16950

[raboof]

It seems to be related to remote builds.

Hmm, I don't use remote builds though

I have to sudo kill -9 the processes

Which processes? killall -9 nix-daemon did kill/restart them, but didn't solve the problem for me.

[06kellyjac]

if you run that as a normal user it'll ignore you

[kenranunderscore]

I ran it with sudo pkill -9 nix-daemon (or some such) and it didn't fix my problem either. Also no remote builds here.
Even a reboot didn't fix it at first.

After the reboot and failed attempt I tried to nix-collect-garbage -d && sudo nix-collect-garbage -d, after which it started working again.

[raboof]

if you run that as a normal user it'll ignore you

(I sudo'ed, and verified in ps aux that the processes were really gone/replaced)

[06kellyjac]

ohh, did kill them. I misread.

Well after I ran into this problem a couple times and removed remote builders I've not encountered it again since.
I did also notice during those period that ping-ing cache.nixos.org also hung for like 10 seconds plus

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/stuck-at-copying-path-pkg-path-from-https-cache-nixos-org-even-after-turning-off-network/17564/5

[Artturin]

I can repro

@edolstra @roberth

[roberth]

@Artturin

I can repro

How?

Here's a fix for a deadlock involving CurlDownloader and the interrupt handlers. May or may not be related. Does it repro with this fix applied?

• Fix deadlocked nix-daemon zombies on darwin #3294 nix#6052

[Artturin]

@Artturin

I can repro

How?

Here's a fix for a deadlock involving CurlDownloader and the interrupt handlers. May or may not be related. Does it repro with this fix applied?

* [Fix deadlocked nix-daemon zombies on darwin #3294 nix#6052](https://github.com/NixOS/nix/pull/6052)

Sorry I should have said that I have the same issue.

I'll apply that pr and see if I get this issue anymore

[Artturin]

i've been running nix with this overlay since 5 days ago and i haven't gotten the same problem

      (self: super: {
        nixUnstable = super.nixUnstable.overrideAttrs (old: {
          src = super.fetchFromGitHub {
            owner = "nixos";
            repo = "nix";
            rev = "a768e85e2fb3b0500829bc42cdc137176481bedf";
            sha256 = "sha256-XtjveEPLGhHPh95+yequcLajtyCskSUTx+XZsP4UqA0=";
          };
          patches = (old.patches or [ ]) ++ [
            (super.fetchpatch {
              url = "https://github.com/NixOS/nix/commit/c3b942e0fc4777f9033f614b6b1f462c0f8c473e.patch";
              sha256 = "sha256-LQ5zkXwv1/3DoZz6CevyipdHTdxX7XlXv3nbmgGsngA=";
            })
          ];
        });
      })

updated #158455 to the newest nix commit to include the pr

[miallo]

@Artturin Just as a warning that you might just be lucky: I also ran into the stuck downloads once and after following the instructions by @kenranunderscore (nix-collect-garbage) I could do the updates and for the last 7 days it also worked for me without the overlay.

Did anyone figure out how to reproduce this bug?

[raboof]

I also haven't seen stuck downloads for the past week or so...