Python bindings fail to build without internet connection

[yurivict]

Description

They build with the connection, but not in an isolated VM:

===>  Building for py39-lightgbm-4.0.0
building lightgbm
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: build>=0.10.0 in /usr/local/lib/python3.9/site-packages (0.10.0)
Requirement already satisfied: packaging>=19.0 in /usr/local/lib/python3.9/site-packages (from build>=0.10.0) (23.1)
Requirement already satisfied: pyproject_hooks in /usr/local/lib/python3.9/site-packages (from build>=0.10.0) (1.0.0)
Requirement already satisfied: tomli>=1.1.0 in /usr/local/lib/python3.9/site-packages (from build>=0.10.0) (2.0.1)
--- building wheel ---
* Creating venv isolated environment...
* Installing packages in isolated environment... (scikit-build-core>=0.4.4)
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x86f60fb80>: Failed to establish a new connection: [Errno 8] Name does not resolve')': /simple/scikit-build-core/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x86f60feb0>: Failed to establish a new connection: [Errno 8] Name does not resolve')': /simple/scikit-build-core/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x86f60fee0>: Failed to establish a new connection: [Errno 8] Name does not resolve')': /simple/scikit-build-core/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x8705b42e0>: Failed to establish a new connection: [Errno 8] Name does not resolve')': /simple/scikit-build-core/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x8705b4490>: Failed to establish a new connection: [Errno 8] Name does not resolve')': /simple/scikit-build-core/
ERROR: Could not find a version that satisfies the requirement scikit-build-core>=0.4.4 (from versions: none)
ERROR: No matching distribution found for scikit-build-core>=0.4.4

scikit-build-core-0.4.7 is already installed, which should be sufficient.

All packages are built in isolated VMs in most distros. For security reasons downloads aren't allowed.

How can we build Python bindings without internet connection?

Environment info

LightGBM version or commit hash: 4.0.0
Python-3.9
FreeBSD 13.2

[jameslamb]

Thanks for the report! I actually just experienced the same as well two days ago, trying to build while on a plane.

I'll put up a PR momentarily to allow optionally passing through --no-build-isolation to pip install and --no-isolation to build. Sorry for the inconvenience.

To make progress without waiting for that, you could patch your copy of this project's build-python.sh like this:

--- a/build-python.sh
+++ b/build-python.sh
@@ -179,7 +179,7 @@ while [ $# -gt 0 ]; do
   shift
 done

-pip install --prefer-binary 'build>=0.10.0'
+# pip install --prefer-binary 'build>=0.10.0'

 # create a new directory that just contains the files needed
 # to build the Python package
@@ -344,6 +344,7 @@ if test "${BUILD_WHEEL}" = true; then
     echo "--- building wheel ---"
     rm -f ../dist/*.whl || true
     python -m build \
+        --no-isolation \
         --wheel \
         --outdir ../dist \
         ${BUILD_ARGS} \

And then build as mentioned in #5977 (comment)

sh build-python.sh bdist_wheel

[jameslamb]

And thanks for coming to the repo to report these things, and for helping to make LightGBM usable by more people!

We did not know that there were these FreeBSD ports of LightGBM and the lightgbm Python package:

• https://cgit.freebsd.org/ports/tree/misc/lightgbm
• https://cgit.freebsd.org/ports/tree/misc/py-lightgbm

[jameslamb]

@yurivict I've put up #6042 with fixes to LightGBM's build system which I hope will resolve this issue for you.

If you have time, please do look at that PR and comment if it doesn't meet what you're looking for.

[yurivict]

With this PR it still creates venv and fails because it can't download packages:

===>  Building for py39-lightgbm-4.0.0
building lightgbm
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: build>=0.10.0 in /usr/local/lib/python3.9/site-packages (0.10.0)
Requirement already satisfied: packaging>=19.0 in /usr/local/lib/python3.9/site-packages (from build>=0.10.0) (23.1)
Requirement already satisfied: pyproject_hooks in /usr/local/lib/python3.9/site-packages (from build>=0.10.0) (1.0.0)
Requirement already satisfied: tomli>=1.1.0 in /usr/local/lib/python3.9/site-packages (from build>=0.10.0) (2.0.1)
--- building wheel ---
* Creating venv isolated environment...
* Installing packages in isolated environment... (scikit-build-core>=0.4.4)
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x87759eb80>: Failed to establish a new connection: [Errno 8] Name does not resolve')': /simple/scikit-build-core/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x87759eeb0>: Failed to establish a new connection: [Errno 8] Name does not resolve')': /simple/scikit-build-core/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x87759eee0>: Failed to establish a new connection: [Errno 8] Name does not resolve')': /simple/scikit-build-core/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x8775c72e0>: Failed to establish a new connection: [Errno 8] Name does not resolve')': /simple/scikit-build-core/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x8775c7490>: Failed to establish a new connection: [Errno 8] Name does not resolve')': /simple/scikit-build-core/
ERROR: Could not find a version that satisfies the requirement scikit-build-core>=0.4.4 (from versions: none)
ERROR: No matching distribution found for scikit-build-core>=0.4.4

It should not create any isolated environment.

Somehow virtually all other Python packages don't have such problem.

[jameslamb]

What command did you run? I did not see that running the following in an environment without internet access.

sh build-python.sh bdist_wheel --no-isolation

Somehow virtually all other Python packages don't have such problem.

LightGBM is not just a Python package... it's a single source tree used to build a CLI, C++ shared library, Python package, R package, Java jar, and more... and as such, it's build system is different than pure-python projects

[yurivict]

It runs:

python3.9 -m build --no-isolation --wheel

[jameslamb]

It runs:

python3.9 -m build --no-isolation --wheel

That's not what I'm asking for.

What command did you run? To be as explicit as possible, here's what I just did to test this.

1. Started a shell in a python:3.9 container

docker run \
    --rm \
    -it python:3.9 \
    bash

2. In that shell, install all the build dependencies:

apt-get update -y
apt-get install -y \
    cmake

pip install --no-cache-dir \
    build \
    ninja \
    pathspec \
    pyproject_metadata \
    scikit-build-core \
    wheel

3. cloned in LightGBM from the branch in [python-package] make it possible to build wheels without internet connection (fixes #5979) #6042

git clone \
    --recursive \
    --branch python/no-internet \
    https://github.com/microsoft/LightGBM.git

cd ./LightGBM

4. disconnect internet access
5. Build wheel

sh ./build-python.sh bdist_wheel --no-isolation

For me, that succeeded and did not print any logs about "creating venv isolated environment":

building lightgbm
Requirement already satisfied: build>=0.10.0 in /usr/local/lib/python3.9/site-packages (0.10.0)
Requirement already satisfied: tomli>=1.1.0 in /usr/local/lib/python3.9/site-packages (from build>=0.10.0) (2.0.1)
Requirement already satisfied: pyproject_hooks in /usr/local/lib/python3.9/site-packages (from build>=0.10.0) (1.0.0)
Requirement already satisfied: packaging>=19.0 in /usr/local/lib/python3.9/site-packages (from build>=0.10.0) (23.1)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
--- building wheel ---
* Getting build dependencies for wheel...
* Building wheel...
2023-08-15 16:21:31,574 - scikit_build_core - INFO - CMake version: 3.25.1
*** scikit-build-core 0.4.8 using CMake 3.25.1 (wheel)
2023-08-15 16:21:31,578 - scikit_build_core - INFO - Build directory: /tmp/tmpmfpmgwgq/build
*** Configuring CMake...
2023-08-15 16:21:31,600 - scikit_build_core - INFO - Ninja version: 1.11.1
loading initial cache file /tmp/tmpmfpmgwgq/build/CMakeInit.txt
-- The C compiler identification is GNU 12.2.0
-- The CXX compiler identification is GNU 12.2.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found OpenMP_C: -fopenmp (found version "4.5") 
-- Found OpenMP_CXX: -fopenmp (found version "4.5") 
-- Found OpenMP: TRUE (found version "4.5")  
-- Performing Test MM_PREFETCH
-- Performing Test MM_PREFETCH - Success
-- Using _mm_prefetch
-- Performing Test MM_MALLOC
-- Performing Test MM_MALLOC - Success
-- Using _mm_malloc
-- Configuring done
-- Generating done
-- Build files have been written to: /tmp/tmpmfpmgwgq/build
*** Building project with Ninja...
[52/52] Linking CXX shared library /LightGBM/lightgbm-python/lib_lightgbm.so
*** Installing project into wheel...
-- Install configuration: "Release"
-- Installing: /tmp/tmpmfpmgwgq/wheel/platlib/lightgbm/lib/lib_lightgbm.so
*** Making wheel...
2023-08-15 16:22:45,009 - scikit_build_core - INFO - Discovered Python package at lightgbm
*** Created lightgbm-4.0.0.99-py3-none-manylinux_2_36_x86_64.whl...
Successfully built lightgbm-4.0.0.99-py3-none-manylinux_2_36_x86_64.whl
cleaning up

In an environment without access to the internet, and it built the wheel successfully.

[yurivict]

What command did you run?

I build the FreeBSD port misc/py-lightgbm using the FreeBSD ports framework. I run cd /usr/ports/misc/py-lightgbm && make. This command succeeds with internet access and fails without it.

The way how I test it without internet access - is running in a virtual machine.

Under the hood make runs the command from my previous message, which is the standard way to build Python ports.

[jameslamb]

Under the hood make runs the command from my previous message, which is the standard way to build Python ports

As of v4.0.0 of the project, LightGBM's Python package cannot just by built by cd-ing into a directory and running python -m build.

If you want to build a custom wheel from sources in this repo, invoke the shell script at the root of the repo exactly as I did at #5979 (comment).

standard way to build Python ports

I'm sorry that the design decisions we've made have made it difficult for you to build FreeBSD ports. I can try to explain in greater detail why this project's build system is as it currently is.

In response to changes outside of our control in upstream tools like build, pip, and setuptools, the build system for this project absolutely needed to change. In response, we chose a build toolchain that we felt balances complexity and correctness best for the majority of lightgbm's users... people who pip install or conda install from package repositories.

You can read about the history of those changes in the following places (and all the other places linked from them):

• https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html
• [RFC] [python] 'setup.py install' is deprecated #5061 (open as a request-for-comment since March 2022)
• [ci] [python-package] replace 'python setup.py' with a shell script #5837
• Replace setup.py with pyproject.toml dmlc/xgboost#9021

I'm happy to help and to try to accommodate less-common use cases we hadn't considered with changes to the interface like #6042, but to be clear we have no plans to support cd python-package && python -m build in the near future.

My top priority is ensuring that LightGBM publishes correct, highly-portable source distributions and wheels of its Python package to package repositories like PyPI and conda-forge, so that for the majority of users running pip install lightgbm "just works" and they can focus on the machine learning task at hand.

If you have a specific suggestion for how to do that in a way that reintroduces the ability to cd python-package && python -m build without introducing significant new maintenance burden, we'd be happy to hear it.