List of dependencies that should be updated from time to time

[StrikerRUS]

Here is a list of different places in LightGBM's GitHub repo where we specify some dependencies or helpers. Quite often we should specify a particular version of such software. And these versions are tend to obsolete with time. If you see that there is a newer version comparing to what we have, please feel free to propose a PR with updates or simply leave a comment here.

• SHAs of commits of third-party dependencies in the external_libs folder. If there are some checkpoints (markers of stable versions) of these libraries, e.g. Releases at GitHub, the most recent stable release should be used (not the unstable latest master commit). If there are no such indicators of stable commit, use the latest available one. How to update GitHub submodules.
• SHAs of commits OpenCL libraries integration (same notes about versions as above):
  1.https://github.com/microsoft/LightGBM/blob/f997a0692ca0f26740d2bdef2695c3e881d4e918/CMakeIntegratedOpenCL.cmake#L5-L6
  2.https://github.com/microsoft/LightGBM/blob/f997a0692ca0f26740d2bdef2695c3e881d4e918/CMakeIntegratedOpenCL.cmake#L8-L9
  3.https://github.com/microsoft/LightGBM/blob/f997a0692ca0f26740d2bdef2695c3e881d4e918/CMakeIntegratedOpenCL.cmake#L11-L12
• Visual Studio version for OpenCL libraries integration https://github.com/microsoft/LightGBM/blob/f997a0692ca0f26740d2bdef2695c3e881d4e918/CMakeIntegratedOpenCL.cmake#L66-L68
• MinGW w64 version at Appveyor https://github.com/microsoft/LightGBM/blob/f997a0692ca0f26740d2bdef2695c3e881d4e918/.appveyor.yml#L26
• PublishBuildArtifacts Azure Pipelines task (multiple occurrences, do search the file for PublishBuildArtifacts keyword)
• macOS image version at Azure Pipelines (Attention! Should be the oldest available, not the newest one) https://github.com/microsoft/LightGBM/blob/f997a0692ca0f26740d2bdef2695c3e881d4e918/.vsts-ci.yml#L144
• Xcode version in Azure Pipelines macOS image (Attention! Should be the oldest available, not the newest one), also make updates in the Python installation guide about that this version is used for Python-package wheels generation https://github.com/microsoft/LightGBM/blob/f2695daba2506d047831e39aeb521e94adec5e8d/.ci/setup.sh#L7
• DownloadBuildArtifacts Azure Pipelines task https://github.com/microsoft/LightGBM/blob/f997a0692ca0f26740d2bdef2695c3e881d4e918/.vsts-ci.yml#L220
• NuGetCommand Azure Pipelines task https://github.com/microsoft/LightGBM/blob/f997a0692ca0f26740d2bdef2695c3e881d4e918/.vsts-ci.yml#L228
• PublishBuildArtifacts Azure Pipelines task https://github.com/microsoft/LightGBM/blob/f997a0692ca0f26740d2bdef2695c3e881d4e918/.vsts-ci.yml#L233
• GitHubRelease Azure Pipelines task https://github.com/microsoft/LightGBM/blob/f997a0692ca0f26740d2bdef2695c3e881d4e918/.vsts-ci.yml#L238
• CMake minimum required version and minimum supported versions of different compilers (including CUDA) in CMakeLists.txt in case of adding new breaking changes in the CPP code of LightGBM (also, make updates in the installation guide and in the Python installation guide)
• Regenerate files in Visual Studio solution folder in case of bumping minimum required version of MSVC++
• Checkout repository GitHub Action (multiple occurrences, do search in the .github/workflows folder for actions/checkout keyword in all files)
• NVIDIA CUDA Docker versions https://github.com/microsoft/LightGBM/blob/65349b41c20116007e602c17a68e375869a7e594/.github/workflows/cuda.yml#L25-L37
• upload-artifact GitHub Action https://github.com/microsoft/LightGBM/blob/f2695daba2506d047831e39aeb521e94adec5e8d/.github/workflows/r_artifacts.yml#L48
• setup-pandoc GitHub Action https://github.com/microsoft/LightGBM/blob/f2695daba2506d047831e39aeb521e94adec5e8d/.github/workflows/r_package.yml#L119
• Versions of Visual Studio for R-package tests (two latest available versions) https://github.com/microsoft/LightGBM/blob/f2695daba2506d047831e39aeb521e94adec5e8d/.github/workflows/r_package.yml#L69-L70 https://github.com/microsoft/LightGBM/blob/f2695daba2506d047831e39aeb521e94adec5e8d/.github/workflows/r_package.yml#L76-L77
• Version of the GitHub REST API (multiple occurrences, do search for application/vnd.github keyword in the repo)
• List of linters for R-package linting script file
• Entire Dockerfile(s) in the lightgbm-ci-docker repository that are used in our CI environments
• OpenCL AMD App SDK for Windows (new file should be uploaded to Releases page of the repo) https://github.com/microsoft/LightGBM/blob/f2695daba2506d047831e39aeb521e94adec5e8d/.ci/install_opencl.ps1#L3
• OpenCL AMD App SDK for Linux (new file should be uploaded to Releases page of the repo) https://github.com/microsoft/LightGBM/blob/f2695daba2506d047831e39aeb521e94adec5e8d/.ci/setup.sh#L72
• SWIG for Windows (new file should be uploaded to Releases page of the repo) https://github.com/microsoft/LightGBM/blob/f2695daba2506d047831e39aeb521e94adec5e8d/.ci/test_windows.ps1#L44
• libboost from Martin Hierholzer's ppa https://github.com/microsoft/LightGBM/blob/f2695daba2506d047831e39aeb521e94adec5e8d/.ci/setup.sh#L70
• gcc version that comes from brew on macOS https://github.com/microsoft/LightGBM/blob/f2695daba2506d047831e39aeb521e94adec5e8d/.ci/test.sh#L4-L5
• Three latest available macOS versions in the Python wheel filename https://github.com/microsoft/LightGBM/blob/f2695daba2506d047831e39aeb521e94adec5e8d/.ci/test.sh#L110
• R 4.x version for Linux from R-project ppa https://github.com/microsoft/LightGBM/blob/f2695daba2506d047831e39aeb521e94adec5e8d/.ci/test_r_package.sh#L30-L32
• R 4.x version for Windows from official CRAN site https://github.com/microsoft/LightGBM/blob/f2695daba2506d047831e39aeb521e94adec5e8d/.ci/test_r_package_windows.ps1#L49
• Rtools 4.x version for Windows (new file should be uploaded to Releases page of the repo) https://github.com/microsoft/LightGBM/blob/f2695daba2506d047831e39aeb521e94adec5e8d/.ci/test_r_package_windows.ps1#L48
• MiKTeX for Windows (new file should be uploaded to Releases page of the repo) https://github.com/microsoft/LightGBM/blob/f2695daba2506d047831e39aeb521e94adec5e8d/.ci/test_r_package_windows.ps1#L107
• Minimum Autoconf version for R-package https://github.com/microsoft/LightGBM/blob/d107872a5342442420edaa1d1849901f865f2331/R-package/configure.ac#L6
• Autoconf version for R-package
• Ubuntu version in which Autoconf should be run to generate configure file for R-package: https://github.com/microsoft/LightGBM/tree/master/R-package#changing-the-cran-package and https://github.com/microsoft/LightGBM/blob/9388b2ecc1649d0a68d3c1ff053775ff00d4d288/.github/workflows/r_configure.yml#L12
• Versions of R-package dependencies https://github.com/microsoft/LightGBM/blob/d107872a5342442420edaa1d1849901f865f2331/R-package/DESCRIPTION#L45-L57
• Version of {roxygen2} used in R-package docs
  • https://github.com/microsoft/LightGBM/blob/fb37e5074ad0163afd9d88366b58075899775cfa/R-package/DESCRIPTION#L66
  • https://github.com/microsoft/LightGBM/blob/fb37e5074ad0163afd9d88366b58075899775cfa/docs/env.yml#L15
• Supported versions of Visual Studio in R-package https://github.com/microsoft/LightGBM/blob/d107872a5342442420edaa1d1849901f865f2331/R-package/src/install.libs.R#L66-L70
• Minimum required versions of system dynamic libraries LightGBM links to in case of adding new breaking changes in the CPP code of LightGBM (also, make updates in the installation guide and in the Python installation guide)
• Azure Pipelines REST API version for artifacts downloading link https://github.com/microsoft/LightGBM/blob/d107872a5342442420edaa1d1849901f865f2331/docs/_static/js/script.js#L41
• Minimum required version of libboost for GPU-version in CMakeLists.txt (also, make updates in the installation guide) https://github.com/microsoft/LightGBM/blob/d107872a5342442420edaa1d1849901f865f2331/CMakeLists.txt#L137
• Minimum required version of OpenCL for GPU-version (multiple occurrences, do search for opencl keyword in the installation guide)
• Visual Studio mapping for Boost binaries in the installation guide
• Supported CUDA architectures in CMakeLists.txt for CUDA-version https://github.com/microsoft/LightGBM/blob/d107872a5342442420edaa1d1849901f865f2331/CMakeLists.txt#L159
• Read the Docs config file (version of config file; build section; adding new and removing deprecated config values)
• Minimum required version of Sphinx in case of adding new breaking changes in docs generating process https://github.com/microsoft/LightGBM/blob/d107872a5342442420edaa1d1849901f865f2331/docs/conf.py#L77
• conda environment for building docs https://github.com/microsoft/LightGBM/blob/master/docs/env.yml
• Minimum required version of sphinx_rtd_theme for documentation generation in docs' folder README file (search for sphinx_rtd_theme; version should be the same as in conda environment)
• Suggested tag for RTD Docker image for generating documentation locally in docs' folder README file (search for readthedocs/build; tag should be the most recently released)
• Entire Dockerfiles in the docker folder
• Versions of Python-package dependencies https://github.com/microsoft/LightGBM/blob/9cc3777c8a892c7ca253864839a926fd35f2b775/python-package/setup.py#L338-L350
• Supported versions of Visual Studio in Python-package https://github.com/microsoft/LightGBM/blob/9cc3777c8a892c7ca253864839a926fd35f2b775/python-package/setup.py#L183
• Supported versions of Visual Studio Platform Toolsets in Python-package https://github.com/microsoft/LightGBM/blob/9cc3777c8a892c7ca253864839a926fd35f2b775/python-package/setup.py#L168
• Supported Python versions (see [python][ci] start to support Python 3.8 #2713 for example how to start running tests with new Python version) https://github.com/microsoft/LightGBM/blob/9cc3777c8a892c7ca253864839a926fd35f2b775/python-package/setup.py#L373-L376
• The oldest supported Python version (shouldn't be too strict just because that version reached EOL; it should reflect only known incompatibilities)
  https://github.com/microsoft/LightGBM/blob/d31346f6c265e994c54ebddd19ed1fb6445ba0c9/python-package/setup.py#L343
• Google Test release tag https://github.com/microsoft/LightGBM/blob/337103d39158001b695f70eb2b2fca6180c7868a/CMakeLists.txt#L475-L476

[StrikerRUS]

@jameslamb Please help to clarify. Should we monitor new releases for R 3.x stuff or they have been frozen forever?

[jameslamb]

@jameslamb Please help to clarify. Should we monitor new releases for R 3.x stuff or they have been frozen forever?

No R 3.x resources will change.

Rtools for R 3.x is frozen forever. https://cran.r-project.org/bin/windows/Rtools/history.html

We also do not need to look for new versions of R itself. R doesn't have a concept of "long-term support" releases like other projects. So you won't see a bugfix that results in, say, R 4.1.1 and R 3.6.4 at the same time.

You can see the history of release dates at https://cran.r-project.org/bin/windows/base/old/.

[StrikerRUS]

I'm so sorry for my English, but I can read this answer in two ways. Which one is correct? 😄

No, R 3.x resources will change.

No R 3.x resources will change.

---

Rtools for R 3.x is frozen forever.

Ah, I see, thanks! Removed from the list.

You can see the history of release dates at https://cran.r-project.org/bin/windows/base/old/.

I see there are

R 4.0.3 (October, 2020)
R 4.0.2 (June, 2020)
R 4.0.1 (June, 2020)
R 4.0.0 (April, 2020)

What about 4.0.4? Why we shouldn't wait for it?

[jameslamb]

Oh sorry, let me rephrase.

• there will never be a new R 3.x release
• there will never be a new Rtools 3.x release

For 4.x series, there will continue to be new releases. They tend to do 4 or 5 per year. The next release of R will be R 4.1.0. That might or might not involve a new Rtools 4.x release. I use this mirror of the R source (R development is not actually done on GitHub) to keep track: https://github.com/wch/r-source/blob/trunk/VERSION

[StrikerRUS]

Thank you very much!
Then I'm removing R 3.x from the list as well.

The next release of R will be R 4.1.0. That might or might not involve a new Rtools 4.x release.

Got it, great! Then I think we can have them both in the list.

They tend to do 4 or 5 per year.

Quite frequent. For instance,

Visual Studio version for OpenCL libraries integration

VS releases have 2-year cycle, but I still find it useful to have VS mentioned here.

[jameslamb]

I want to note here that R 4.1.0 is planned for release on May 18, 2021.

https://twitter.com/pdalgd/status/1383008275545919488

You can see a full list of changes in that version at https://stat.ethz.ch/R-manual/R-devel/doc/html/NEWS.html. There is only one that looks immediately relevant to {lightgbm}, although it's hard to tell since some of the changes are deep in the internals of R and {lightgbm} may indirectly rely on them.

The default C++ standard has been changed to C++14 where available (which it is on all currently checked platforms): if not (as before) C++11 is used if available otherwise C++ is not supported.

Packages which specify C++11 will still be installed using C++11.

Either specify C++11 (see ‘Writing R Extensions’) or modernize the code and if needed specify C++14

I don't think {lightgbm} needs to worry about this, because it already declares its use of C++11 in the way that is recommended by "Writing R Extensions"

• guidelines saying to set CXX_STD in Makevars: https://cran.r-project.org/doc/manuals/r-release/R-exts.html#Using-C_002b_002b-code
  • Makevars.in: https://github.com/microsoft/LightGBM/blob/7ea2bc4dbca53c03eaae2aa2b22f9b30447caea4/R-package/src/Makevars.in#L1
  • Makevars.win.in: https://github.com/microsoft/LightGBM/blob/7ea2bc4dbca53c03eaae2aa2b22f9b30447caea4/R-package/src/Makevars.win.in#L1

I don't think that any action is required at this time to prepare for R 4.1.0. But that release will be a big one so I just wanted to note that it's coming so we're prepared.