[python-package] [ci] Use manylinux image to produce wheels

[jameslamb]

Summary

Created from #5390 (comment).

The following changes should be made in LightGBM:

• move from supporting GLIBC >= 2.14 to GLIBC >= 2.23
• start producing Python wheels using one of the manylinux images (https://github.com/pypa/manylinux)
  • this implies ending official support for Ubuntu 14.04 in LightGBM

Motivation

It has been really difficult to maintain support for older operating systems and versions of GLIBC.

References:

• [ci] [dask] CI jobs failing with Dask 2022.7.1 #5390 (comment)
• [python] Start supporting Python 3.10 #4893 (comment)
• [python] Start supporting Python 3.10 #4893 (comment)

Moving to newer base image and higher floor for GLIBC and supported Ubuntu version should help to alleviate some maintenance burden.

Description

As described in #5390 (comment), LightGBM should upgrade to a manylinux base image supporting at least GLIBC==2.23 (and not too much higher) for producing wheels.

The manylinux README shows the following compatibility for wheels produced on those images (https://github.com/pypa/manylinux).

https://ubuntu.com/about/release-cycle describes the release cycle for versions of Ubuntu

Using an approach like the following

docker run \
    --rm \
    --entrypoint="" \
    -it quay.io/pypa/manylinux1_x86_64 \
    ldd --version

I found that the GLIBC version for the published manylinux images is as follows.

image	GLIBC
manylinux1	2.5
manylinux2010	2.12
manylinux2014	2.17
manylinux_2_28	2.28

References

Other related conversations:

• discussion from 2020 about possibly using manylinux_2010: Add arm64 wheel to travis-ci #3421 (comment)
• plan to migrate to PEP600 manylinux_x_y tags eventually: Python wheels not PEP 599 compliant; wheels should include libgomp.so.1 #4484 (comment)
• discussion about lightgbm's x86_64 wheel being labeled manylinux1 even though it is not fully compatible with that tag: [python-package] LightGBM wheel on PyPI seems to have the wrong platform tag #4788

[jameslamb]

Now that CI is working again, next I'm going to prioritize this issue, based on the discussion in #5390 (comment), so that we can upgrade to macOS-latest CI images before they are shut off one month from now.

I'll work on an image on the dev2 branch of https://github.com/guolinke/lightgbm-ci-docker that @StrikerRUS set up for me (#5390 (comment)).

[jameslamb]

Did some research tonight, and I think we should pursue manylinux_2_28 (PEP600) for lightgbm v4.0.0 wheels

It's a big jump (lightgbm currently publishes manylinux1 wheels for x84_64 architecture), but I think it's the right choice given:

• limited maintainer availability
• slow pace of releases in this project
• the fact that it will realistically probably still be at least 2 months until v4.0.0 is actually released
• Python 3.7 reaches end-of-life in 7 months (https://devguide.python.org/versions/)

Reasons to prefer manylinux_2_28

1. it's the oldest supported manylinux image with GLIBC 2.23+ (click me)

manylinux_2_24 is the only other official manylinux image meeting the requirement GLIBC >= 2.23, and it is deprecated (pypa/manylinux#1332) and will officially be unsupported on January 1st, 2023 (pypa/manylinux#1369)

2. Python 3.7 reaches end-of-life in 7 months, so not having wheel support for it isn't too bad (click me)

From https://devguide.python.org/versions/.

Reasons we might not want to support manylinux_2_28

1. Python 3.7 users might not be able to use these wheels (click me)

According to https://github.com/pypa/manylinux (see screenshot in this issue's description), manylinux_2_28 wheels will by default only be installable on Python 3.8.10+. That means users on older versions will have to build from source (which involves having CMake installed and recompiling).

According to https://pypistats.org/packages/lightgbm, at least half of all lightgbm installations from PyPI are still on Python 3.7.

2. similar projects like xgboost are still on manylinux2014 (click me)

Looked around at some other similar projects to see what tags they use for Linux wheels:

library	manylinux1	manylinux_2010	manylinux2014	manylinux_2_17
catboost (link)	✅
numpy (link)			✅	✅
scikit-learn (link)			✅	✅
xgboost (link)			✅	✅