[c++] support building with Ninja on Linux

[jameslamb]

Contributes to #5061.

Adds support for building with Ninja instead of make on Linux.

This is necessary (or at least, very very helpful) for integrating scikit-build-core as the backend for building the Python package (as I'm working on in #5759).

What is Ninja?

Ninja is an open-source alternative to make, originally developed to support building Google Chrome on Linux ("Google man open sources Chrome build system").

It's often much faster than make, in exchange for being simpler (i.e. supporting a stricter, harder-to-customize API).

Because of its build speeds, Ninja is the strongly-preferred default build tool for macOS and Linux in scikit-build-core. See https://github.com/scikit-build/scikit-build-core.

How do these changes relate to supporting it?

Try the following on a Linux system, using latest master (36fe73a).

rm -rf ./build
mkdir ./build
cd ./build
cmake \
    -G 'Ninja' \
    -D__INTEGRATE_OPENCL=ON \
    ..

ninja -j2 _lightgbm

You'll see the following error.

ninja: error: 'Boost/source/stage/lib/libboost_filesystem.a', needed by '/home/jlamb/repos/LightGBM/lib_lightgbm.so', missing and no known rule to make it

This is because of the use of ExternalProject_Add(... BUILD_COMMAND ...) in the CMake script used to build the integrated-OpenCL wheels. Unlike with make, the Ninja CMake generator requires that you explicitly list the outputs of such commands, so the generated .ninja files will contain the appropriate references to them. See the following for documentation on that:

• https://cmake.org/cmake/help/latest/module/ExternalProject.html
• https://stackoverflow.com/a/65803911/3986677

This PR modifies that script by adding an explicit list of the expected output files.

Should we add new CI jobs?

I don't think that's necessary. Once #5759 is merged, the Python-package tests on Linux and macOS will be testing compilation with Ninja.

Notes for Reviewers

Thanks for your time and consideration. I know the progress towards fixing #5061 has been slow and has involved a lot of building/packaging changes. I really appreciate your thoughtful reviews.

[jameslamb]

cc-ing main authors of the integrated OpenCL wheels, in case you're interested: @jgiannuzzi @tpboudreau

and thanks to @csegarragonz for the very helpful Stack Overflow answer 😊

[jameslamb]

@guolinke @shiyu1994 could I get a review on this this week, if possible? Sorry for the @, but I think this is an important step towards #5061 via its support of #5759 ... and therefore an important step towards the v4.0 release (#5153).

[jameslamb]

Thank you!

[github-actions]

This pull request has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.