Binaries R2 - s3 out of sync

[malfet]

🐛 Describe the bug

Reporting on behalf of @jameslamb

Running

docker run --rm python:3.11 /bin/bash -c "pip --version && pip download --no-deps --index-url https://download.pytorch.org/whl/cu129 'torch==2.9.0+cu129'"

Fails with:

pip 24.0 from /usr/local/lib/python3.11/site-packages/pip (python 3.11)

Looking in indexes: https://download.pytorch.org/whl/cu129
Collecting torch==2.9.0+cu129
  Downloading https://download-r2.pytorch.org/whl/cu129/torch-2.9.0%2Bcu129-cp311-cp311-manylinux_2_28_x86_64.whl.metadata (30 kB)
ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    torch==2.9.0+cu129 from https://download-r2.pytorch.org/whl/cu129/torch-2.9.0%2Bcu129-cp311-cp311-manylinux_2_28_x86_64.whl#sha256=78005065547658b8c3fd2dd4d68f34625b0543494c947001b5303d253b22da5f:
        Expected sha256 05df84ccec407908cb70f89d6c2675b8220661f23d7de0cf899f4401f8ab2798
             Got        2b0a3a5d37a8d7447e56e7e4e27280f881e805fbae79130fa8874bcfe6eae333

Likely because binary got corrupted when migrated from CloudFront to R2

$ curl -OL https://download-r2.pytorch.org/whl/cu129/torch-2.9.0%2Bcu129-cp311-cp311-manylinux_2_28_x86_64.whl
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1196M  100 1196M    0     0   125M      0  0:00:09  0:00:09 --:--:--  103M
$ shasum -a 256 torch-2.9.0%2Bcu129-cp311-cp311-manylinux_2_28_x86_64.whl 
aead92a82a01b859a54c933667fe5208598f1c58361b377f1c7e4c898702b341  torch-2.9.0%2Bcu129-cp311-cp311-manylinux_2_28_x86_64.whl
$ curl -OL https://download.pytorch.org/whl/cu129/torch-2.9.0%2Bcu129-cp311-cp311-manylinux_2_28_x86_64.whl
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1196M  100 1196M    0     0  44.7M      0  0:00:26  0:00:26 --:--:-- 32.1M
$ shasum -a 256 torch-2.9.0%2Bcu129-cp311-cp311-manylinux_2_28_x86_64.whl 
78005065547658b8c3fd2dd4d68f34625b0543494c947001b5303d253b22da5f  torch-2.9.0%2Bcu129-cp311-cp311-manylinux_2_28_x86_64.whl

Versions

¯\(ツ)/¯

cc @ezyang @gchanan @kadeng @msaroufim @seemethere @atalman @tinglvv @nWEIdia

[atalman]

@malfet There was no corruption during the migration. The cu129 binaries where not part of the official release:
https://github.com/pytorch/pytorch/blob/main/RELEASE.md
Stable CUDA: CUDA 12.6 (CUDNN 9.10.2.21), CUDA 12.8 (CUDNN 9.10.2.21)
Experimental CUDA: CUDA 13.0 (CUDNN 9.13.0.50)

However they where rebuilt after the release on
October 16: https://hud.pytorch.org/pytorch/pytorch/commit/e9e3db62feb524d93664ce45da8d84904d154dbd
October 18: https://hud.pytorch.org/pytorch/pytorch/commit/cbe1a35dbdfdc9b7490212bdd812f3e2f4ce9e2d

Post official release date that was done on October 15: https://pypi.org/project/torch/2.9.0/

The hash mismatch is due to these two versions being released - October 16 vs October 18 version.

[malfet]

@atalman sorry but I don't fully understand this release argument here. R2 and CloudFront should serve identical files regardless of whether they are in whl/, whl/test or whl/nightly folder, shouldn't it?

[atalman]

Yes, will run comparison across both Mirrors now and make sure its synced

[atalman]

Run sync and cache purge for 2.9.0 cu129. Whls are in sync:

INFO: Found 42 objects to check for torch-2.9.0+cu129 in whl/

  OK          whl/cu129/torch-2.9.0+cu129-cp310-cp310-manylinux_2_28_aarch64.whl.metadata
  OK          whl/cu129/torch-2.9.0+cu129-cp310-cp310-win_amd64.whl.metadata
  OK          whl/cu129/torch-2.9.0+cu129-cp310-cp310-manylinux_2_28_x86_64.whl.metadata
  OK          whl/cu129/torch-2.9.0+cu129-cp311-cp311-manylinux_2_28_aarch64.whl.metadata
  OK          whl/cu129/torch-2.9.0+cu129-cp311-cp311-manylinux_2_28_x86_64.whl.metadata
  OK          whl/cu129/torch-2.9.0+cu129-cp310-cp310-manylinux_2_28_aarch64.whl
  OK          whl/cu129/torch-2.9.0+cu129-cp311-cp311-win_amd64.whl.metadata
  OK          whl/cu129/torch-2.9.0+cu129-cp311-cp311-manylinux_2_28_aarch64.whl
  OK          whl/cu129/torch-2.9.0+cu129-cp312-cp312-manylinux_2_28_aarch64.whl.metadata
  OK          whl/cu129/torch-2.9.0+cu129-cp312-cp312-manylinux_2_28_aarch64.whl
  OK          whl/cu129/torch-2.9.0+cu129-cp312-cp312-manylinux_2_28_x86_64.whl.metadata
  OK          whl/cu129/torch-2.9.0+cu129-cp310-cp310-manylinux_2_28_x86_64.whl
  OK          whl/cu129/torch-2.9.0+cu129-cp312-cp312-win_amd64.whl.metadata
  OK          whl/cu129/torch-2.9.0+cu129-cp312-cp312-manylinux_2_28_x86_64.whl
  OK          whl/cu129/torch-2.9.0+cu129-cp313-cp313-manylinux_2_28_aarch64.whl.metadata
  OK          whl/cu129/torch-2.9.0+cu129-cp311-cp311-manylinux_2_28_x86_64.whl
  OK          whl/cu129/torch-2.9.0+cu129-cp313-cp313-manylinux_2_28_x86_64.whl.metadata
  OK          whl/cu129/torch-2.9.0+cu129-cp313-cp313-manylinux_2_28_aarch64.whl
  OK          whl/cu129/torch-2.9.0+cu129-cp313-cp313-win_amd64.whl.metadata
  OK          whl/cu129/torch-2.9.0+cu129-cp310-cp310-win_amd64.whl
  OK          whl/cu129/torch-2.9.0+cu129-cp313-cp313t-manylinux_2_28_aarch64.whl.metadata
  OK          whl/cu129/torch-2.9.0+cu129-cp313-cp313t-manylinux_2_28_aarch64.whl
  OK          whl/cu129/torch-2.9.0+cu129-cp313-cp313t-manylinux_2_28_x86_64.whl.metadata
  OK          whl/cu129/torch-2.9.0+cu129-cp311-cp311-win_amd64.whl
  OK          whl/cu129/torch-2.9.0+cu129-cp313-cp313t-win_amd64.whl.metadata
  OK          whl/cu129/torch-2.9.0+cu129-cp313-cp313-manylinux_2_28_x86_64.whl
  OK          whl/cu129/torch-2.9.0+cu129-cp314-cp314-manylinux_2_28_aarch64.whl.metadata
  OK          whl/cu129/torch-2.9.0+cu129-cp314-cp314-manylinux_2_28_aarch64.whl
  OK          whl/cu129/torch-2.9.0+cu129-cp314-cp314-manylinux_2_28_x86_64.whl.metadata
  OK          whl/cu129/torch-2.9.0+cu129-cp312-cp312-win_amd64.whl
  OK          whl/cu129/torch-2.9.0+cu129-cp314-cp314-win_amd64.whl.metadata
  OK          whl/cu129/torch-2.9.0+cu129-cp313-cp313t-manylinux_2_28_x86_64.whl
  OK          whl/cu129/torch-2.9.0+cu129-cp314-cp314t-manylinux_2_28_aarch64.whl.metadata
  OK          whl/cu129/torch-2.9.0+cu129-cp314-cp314t-manylinux_2_28_aarch64.whl
  OK          whl/cu129/torch-2.9.0+cu129-cp314-cp314t-manylinux_2_28_x86_64.whl.metadata
  OK          whl/cu129/torch-2.9.0+cu129-cp314-cp314-manylinux_2_28_x86_64.whl
  OK          whl/cu129/torch-2.9.0+cu129-cp314-cp314t-win_amd64.whl.metadata
  OK          whl/cu129/torch-2.9.0+cu129-cp313-cp313-win_amd64.whl
  OK          whl/cu129/torch-2.9.0+cu129-cp314-cp314-win_amd64.whl
  OK          whl/cu129/torch-2.9.0+cu129-cp314-cp314t-manylinux_2_28_x86_64.whl
  OK          whl/cu129/torch-2.9.0+cu129-cp313-cp313t-win_amd64.whl

@jameslamb Please test this issue should be resolved

[jameslamb]

Tested with my repro from #145501 (comment) (Python 3.11, x86_64, pip 24.0) and looks like it is resolved!

Thanks so much for the quick resolution 😊