What happened?
This is continuation of #4029 - while .sdist and AMD linux whl packages have been uploaded, the ARM linux whl packages are still missing - and it causes problems in CI systems that run tests on ARM in linux, also it adds a lot of overhead for anyone running linux docker on Mac OS ARM devices (i.e. pretty much all of modern Mac OS).
This is quite a big issue for Airflow - we not only run our CI with ARM, but we also run tests in parallel and a lot of our contributors use ARM MacOS devices - and we have there breeze development environment that uses docker-compose to be able to reproduce CI environment and we are using debian-bookworm based images for this environment.
The effect of runnig CI in parallel for deltalake is that when we are building several builds in parallel, and they want to build deltalake in parallel, we are sharing uv cache between the processes and in such case, only one process gets a lock and rebuilds deltalake - this lock however is kept for the whole time of building deltalake and - with multiple processes - the build takes more than default uv 300 seconds and build fails:
Example here: https://github.com/apache/airflow/actions/runs/20687292566/job/59390484354
× Failed to download and build `deltalake==1.3.0`
├─▶ Failed to acquire lock on the distribution cache
├─▶ Could not acquire lock
╰─▶ Timeout (300s) when waiting for lock on
`/root/.cache/uv/sdists-v9/pypi/deltalake/1.3.0` at
`/root/.cache/uv/sdists-v9/pypi/deltalake/1.3.0/.lock`, is another uv
process running? You can set `UV_LOCK_TIMEOUT` to increase the timeout.
We could probably increase the lock timeout, but actually this is an indication that not having ARM whls increases the time to run our CI tests quite dramatically - it's not cool to delay most of our build CI by more than 300 seconds (5 mins) just because a .whl file is missing.
For now we simply exclude 1.3.0 version of deltalake, and hope that if not for 1.3.0, this will be fixed for 1.3.1
Expected behavior
Whl file should be present for ARM linux and pip install deltalake or uv pip install deltalake should not take several minutes.
Operating System
Linux
Binding
Python
Bindings Version
1.3.0
Steps to reproduce
- Use MacOS ARM device and have docker desktop or similar installed.
- Run
docker run -it python:3.12 bash
- In docker run
pip install deltalake==1.2.1 and observe that it installs quickly from whl (seconds)
- In docker run
pip install deltalake==1.3.0 and observe that it downloads .tar.gz file and builds the whl which takes way longer (few minutes).
The step 4 takes at least 4-5 minutes on my very powerful M4 machine and all the procesors are busy while this is happening - most of the time it uses all processors while running the build for those few minutes - which means that it will be way slower on a smaller runner machine with 4 cores (which is default on public runners of GitHub in GitHub actions):
I think this is quite bad experience for users to make them wait when all that is needed is to upload a pre-built file, especially that MacOS ARM + Docker linux is pretty common configuration.
Relevant logs
⌁ [jarekpotiuk:~/code/airflow] again-limit-deltalake(+3/-1)+ 9s 127 ± docker run -it python:3.12 bash
root@884263eb50cc:/# pip install deltalake==1.2.1
Collecting deltalake==1.2.1
Downloading deltalake-1.2.1-cp39-abi3-manylinux_2_28_aarch64.whl.metadata (5.3 kB)
Collecting arro3-core>=0.5.0 (from deltalake==1.2.1)
Downloading arro3_core-0.6.5-cp311-abi3-manylinux_2_24_aarch64.whl.metadata (363 bytes)
Collecting deprecated>=1.2.18 (from deltalake==1.2.1)
Downloading deprecated-1.3.1-py2.py3-none-any.whl.metadata (5.9 kB)
Collecting wrapt<3,>=1.10 (from deprecated>=1.2.18->deltalake==1.2.1)
Downloading wrapt-2.0.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl.metadata (9.0 kB)
Downloading deltalake-1.2.1-cp39-abi3-manylinux_2_28_aarch64.whl (45.4 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 45.4/45.4 MB 1.6 MB/s eta 0:00:00
Downloading arro3_core-0.6.5-cp311-abi3-manylinux_2_24_aarch64.whl (2.5 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.5/2.5 MB 595.9 kB/s eta 0:00:00
Downloading deprecated-1.3.1-py2.py3-none-any.whl (11 kB)
Downloading wrapt-2.0.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl (123 kB)
Installing collected packages: wrapt, arro3-core, deprecated, deltalake
Successfully installed arro3-core-0.6.5 deltalake-1.2.1 deprecated-1.3.1 wrapt-2.0.1
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
[notice] A new release of pip is available: 25.0.1 -> 25.3
[notice] To update, run: pip install --upgrade pip
root@884263eb50cc:/# pip install deltalake==1.3.0
Collecting deltalake==1.3.0
Downloading deltalake-1.3.0.tar.gz (5.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.2/5.2 MB 832.3 kB/s eta 0:00:00
Installing build dependencies ... done
Getting requirements to build wheel ... done
Installing backend dependencies ... done
Preparing metadata (pyproject.toml) ...
Created wheel for deltalake: filename=deltalake-1.3.0-cp310-abi3-linux_aarch64.whl size=251317724 sha256=80b3faa012d9f92157c949577dffe55a546e582e2d759a6a8dd3b28f516fef84
Stored in directory: /root/.cache/pip/wheels/1a/42/86/50035d80787b970e63cce6b699a1601c70b0ed2e3fb07fd4e5
Successfully built deltalake
Installing collected packages: deltalake
Attempting uninstall: deltalake
Found existing installation: deltalake 1.2.1
Uninstalling deltalake-1.2.1:
Successfully uninstalled deltalake-1.2.1
Successfully installed deltalake-1.3.0
What happened?
This is continuation of #4029 - while .sdist and AMD linux whl packages have been uploaded, the ARM linux whl packages are still missing - and it causes problems in CI systems that run tests on ARM in linux, also it adds a lot of overhead for anyone running linux docker on Mac OS ARM devices (i.e. pretty much all of modern Mac OS).
This is quite a big issue for Airflow - we not only run our CI with ARM, but we also run tests in parallel and a lot of our contributors use ARM MacOS devices - and we have there
breezedevelopment environment that uses docker-compose to be able to reproduce CI environment and we are using debian-bookworm based images for this environment.The effect of runnig CI in parallel for deltalake is that when we are building several builds in parallel, and they want to build deltalake in parallel, we are sharing
uvcache between the processes and in such case, only one process gets a lock and rebuilds deltalake - this lock however is kept for the whole time of building deltalake and - with multiple processes - the build takes more than defaultuv300 seconds and build fails:Example here: https://github.com/apache/airflow/actions/runs/20687292566/job/59390484354
We could probably increase the lock timeout, but actually this is an indication that not having ARM whls increases the time to run our CI tests quite dramatically - it's not cool to delay most of our build CI by more than 300 seconds (5 mins) just because a .whl file is missing.
For now we simply exclude 1.3.0 version of deltalake, and hope that if not for 1.3.0, this will be fixed for 1.3.1
Expected behavior
Whl file should be present for ARM linux and
pip install deltalakeoruv pip install deltalakeshould not take several minutes.Operating System
Linux
Binding
Python
Bindings Version
1.3.0
Steps to reproduce
docker run -it python:3.12 bashpip install deltalake==1.2.1and observe that it installs quickly from whl (seconds)pip install deltalake==1.3.0and observe that it downloads .tar.gz file and builds the whl which takes way longer (few minutes).The step 4 takes at least 4-5 minutes on my very powerful M4 machine and all the procesors are busy while this is happening - most of the time it uses all processors while running the build for those few minutes - which means that it will be way slower on a smaller runner machine with 4 cores (which is default on public runners of GitHub in GitHub actions):
I think this is quite bad experience for users to make them wait when all that is needed is to upload a pre-built file, especially that MacOS ARM + Docker linux is pretty common configuration.
Relevant logs