[ci] [R-package] clang18 jobs failing: _mm_malloc compilation errors

[jameslamb]

Description

The clang18 R-package job is failing across multiple PRs, with errors like this:

In file included from /usr/lib/llvm-18/lib/clang/18/include/xmmintrin.h:31:
/usr/lib/llvm-18/lib/clang/18/include/mm_malloc.h:32:40: error: 'alloc_align' attribute parameter 1 is out of bounds
   32 |                                        __alloc_align__(2)))
      |                                        ^               ~
/usr/lib/llvm-18/lib/clang/18/include/mm_malloc.h:33:1: error: static declaration of 'malloc' follows non-static declaration
   33 | _mm_malloc(size_t __size, size_t __align) {
      | ^
./include/LightGBM/utils/common.h:53:26: note: expanded from macro '_mm_malloc'
   53 | #define _mm_malloc(a, b) malloc(a)
      |                          ^
/usr/include/stdlib.h:540:14: note: previous declaration is here
  540 | extern void *malloc (size_t __size) __THROW __attribute_malloc__

(build link)

full log (click me)

clang++-18 -stdlib=libc++ -std=gnu++17 -I"/opt/R/devel/lib/R/include" -DNDEBUG -I./include -DEIGEN_MPL2_ONLY -DEIGEN_DONT_PARALLELIZE -DUSE_SOCKET -DLGB_R_BUILD  -I/usr/local/include   -fopenmp -pthread -fpic  -O3 -Wall -pedantic -frtti -Wp,-D_FORTIFY_SOURCE=3  -c treelearner/tree_learner.cpp -o treelearner/tree_learner.o
In file included from treelearner/linear_tree_learner.cpp:7:
In file included from ./include/Eigen/Dense:1:
In file included from ./include/Eigen/Core:22:
In file included from ./include/Eigen/src/Core/util/ConfigureVectorization.h:346:
In file included from /usr/lib/llvm-18/lib/clang/18/include/emmintrin.h:17:
In file included from /usr/lib/llvm-18/lib/clang/18/include/xmmintrin.h:31:
/usr/lib/llvm-18/lib/clang/18/include/mm_malloc.h:32:40: error: 'alloc_align' attribute parameter 1 is out of bounds
   32 |                                        __alloc_align__(2)))
      |                                        ^               ~
/usr/lib/llvm-18/lib/clang/18/include/mm_malloc.h:33:1: error: static declaration of 'malloc' follows non-static declaration
   33 | _mm_malloc(size_t __size, size_t __align) {
      | ^
./include/LightGBM/utils/common.h:53:26: note: expanded from macro '_mm_malloc'
   53 | #define _mm_malloc(a, b) malloc(a)
      |                          ^
/usr/include/stdlib.h:540:14: note: previous declaration is here
  540 | extern void *malloc (size_t __size) __THROW __attribute_malloc__
      |              ^
In file included from treelearner/linear_tree_learner.cpp:7:
In file included from ./include/Eigen/Dense:1:
In file included from ./include/Eigen/Core:22:
In file included from ./include/Eigen/src/Core/util/ConfigureVectorization.h:346:
In file included from /usr/lib/llvm-18/lib/clang/18/include/emmintrin.h:17:
In file included from /usr/lib/llvm-18/lib/clang/18/include/xmmintrin.h:31:
/usr/lib/llvm-18/lib/clang/18/include/mm_malloc.h:34:7: error: use of undeclared identifier '__align'; did you mean '__asin'?
   34 |   if (__align == 1) {
      |       ^
/usr/include/x86_64-linux-gnu/bits/mathcalls.h:55:1: note: '__asin' declared here
   55 | __MATHCALL_VEC (asin,, (_Mdouble_ __x));
      | ^
/usr/include/math.h:280:3: note: expanded from macro '__MATHCALL_VEC'
  280 |   __MATHCALL (function, suffix, args)
      |   ^
/usr/include/math.h:287:3: note: expanded from macro '__MATHCALL'
  287 |   __MATHDECL (_Mdouble_,function,suffix, args)
      |   ^
/usr/include/math.h:290:22: note: expanded from macro '__MATHDECL'
  290 |   __MATHDECL_1(type, __CONCAT(__,function),suffix, args)
      |                      ^
/usr/include/x86_64-linux-gnu/sys/cdefs.h:124:23: note: expanded from macro '__CONCAT'
  124 | #define __CONCAT(x,y)   x ## y
      |                         ^
<scratch space>:125:1: note: expanded from here
  125 | __asin
      | ^
In file included from treelearner/linear_tree_learner.cpp:7:
In file included from ./include/Eigen/Dense:1:
In file included from ./include/Eigen/Core:22:
In file included from ./include/Eigen/src/Core/util/ConfigureVectorization.h:346:
In file included from /usr/lib/llvm-18/lib/clang/18/include/emmintrin.h:17:
In file included from /usr/lib/llvm-18/lib/clang/18/include/xmmintrin.h:31:
/usr/lib/llvm-18/lib/clang/18/include/mm_malloc.h:34:15: error: comparison between pointer and integer ('double (*)(double) noexcept(true)' and 'int')
   34 |   if (__align == 1) {
      |       ~~~~~~~ ^  ~
/usr/lib/llvm-18/lib/clang/18/include/mm_malloc.h:38:9: error: use of undeclared identifier '__align'; did you mean '__asin'?
   38 |   if (!(__align & (__align - 1)) && __align < sizeof(void *))
      |         ^
/usr/include/x86_64-linux-gnu/bits/mathcalls.h:55:1: note: '__asin' declared here
   55 | __MATHCALL_VEC (asin,, (_Mdouble_ __x));
      | ^
/usr/include/math.h:280:3: note: expanded from macro '__MATHCALL_VEC'
  280 |   __MATHCALL (function, suffix, args)
      |   ^
/usr/include/math.h:287:3: note: expanded from macro '__MATHCALL'
  287 |   __MATHDECL (_Mdouble_,function,suffix, args)
      |   ^
/usr/include/math.h:290:22: note: expanded from macro '__MATHDECL'
  290 |   __MATHDECL_1(type, __CONCAT(__,function),suffix, args)
      |                      ^
/usr/include/x86_64-linux-gnu/sys/cdefs.h:124:23: note: expanded from macro '__CONCAT'
  124 | #define __CONCAT(x,y)   x ## y
      |                         ^
<scratch space>:125:1: note: expanded from here
  125 | __asin
      | ^
In file included from treelearner/linear_tree_learner.cpp:7:
In file included from ./include/Eigen/Dense:1:
In file included from ./include/Eigen/Core:22:
In file included from ./include/Eigen/src/Core/util/ConfigureVectorization.h:346:
In file included from /usr/lib/llvm-18/lib/clang/18/include/emmintrin.h:17:
In file included from /usr/lib/llvm-18/lib/clang/18/include/xmmintrin.h:31:
/usr/lib/llvm-18/lib/clang/18/include/mm_malloc.h:38:20: error: use of undeclared identifier '__align'; did you mean '__asin'?
   38 |   if (!(__align & (__align - 1)) && __align < sizeof(void *))
      |                    ^
/usr/include/x86_64-linux-gnu/bits/mathcalls.h:55:1: note: '__asin' declared here
   55 | __MATHCALL_VEC (asin,, (_Mdouble_ __x));
      | ^
/usr/include/math.h:280:3: note: expanded from macro '__MATHCALL_VEC'
  280 |   __MATHCALL (function, suffix, args)
      |   ^
/usr/include/math.h:287:3: note: expanded from macro '__MATHCALL'
  287 |   __MATHDECL (_Mdouble_,function,suffix, args)
      |   ^
/usr/include/math.h:290:22: note: expanded from macro '__MATHDECL'
  290 |   __MATHDECL_1(type, __CONCAT(__,function),suffix, args)
      |                      ^
/usr/include/x86_64-linux-gnu/sys/cdefs.h:124:23: note: expanded from macro '__CONCAT'
  124 | #define __CONCAT(x,y)   x ## y
      |                         ^
<scratch space>:125:1: note: expanded from here
  125 | __asin
      | ^
In file included from treelearner/linear_tree_learner.cpp:7:
In file included from ./include/Eigen/Dense:1:
In file included from ./include/Eigen/Core:22:
In file included from ./include/Eigen/src/Core/util/ConfigureVectorization.h:346:
In file included from /usr/lib/llvm-18/lib/clang/18/include/emmintrin.h:17:
In file included from /usr/lib/llvm-18/lib/clang/18/include/xmmintrin.h:31:
/usr/lib/llvm-18/lib/clang/18/include/mm_malloc.h:38:28: error: arithmetic on a pointer to the function type 'double (double) noexcept(true)'
   38 |   if (!(__align & (__align - 1)) && __align < sizeof(void *))
      |                    ~~~~~~~ ^
/usr/lib/llvm-18/lib/clang/18/include/mm_malloc.h:38:37: error: use of undeclared identifier '__align'
   38 |   if (!(__align & (__align - 1)) && __align < sizeof(void *))
      |                                     ^
/usr/lib/llvm-18/lib/clang/18/include/mm_malloc.h:39:5: error: use of undeclared identifier '__align'
   39 |     __align = sizeof(void *);
      |     ^
/usr/lib/llvm-18/lib/clang/18/include/mm_malloc.h:47:41: error: use of undeclared identifier '__align'
   47 |   if (posix_memalign(&__mallocedMemory, __align, __size))
      |                                         ^
/usr/lib/llvm-18/lib/clang/18/include/mm_malloc.h:55:1: error: static declaration of 'free' follows non-static declaration
   55 | _mm_free(void *__p)
      | ^
./include/LightGBM/utils/common.h:54:21: note: expanded from macro '_mm_free'
   54 | #define _mm_free(a) free(a)
      |                     ^
/usr/include/stdlib.h:555:13: note: previous declaration is here
  555 | extern void free (void *__ptr) __THROW;
      |             ^
clang++-18 -stdlib=libc++ -std=gnu++17 -I"/opt/R/devel/lib/R/include" -DNDEBUG -I./include -DEIGEN_MPL2_ONLY -DEIGEN_DONT_PARALLELIZE -DUSE_SOCKET -DLGB_R_BUILD  -I/usr/local/include   -fopenmp -pthread -fpic  -O3 -Wall -pedantic -frtti -Wp,-D_FORTIFY_SOURCE=3  -c treelearner/voting_parallel_tree_learner.cpp -o treelearner/voting_parallel_tree_learner.o
11 errors generated.
make: *** [/opt/R/devel/lib/R/etc/Makeconf:204: treelearner/linear_tree_learner.o] Error 1
make: *** Waiting for unfinished jobs....
ERROR: compilation failed for package ‘lightgbm’
* removing ‘/tmp/Rtmp4131Uu/Rinst320155bc02f4/lightgbm’
      -----------------------------------
ERROR: package installation failed

The root cause appears to be this from further up in the logs.

** using staged installation
checking location of R... /opt/R/devel/lib/R
checking whether MM_PREFETCH works... ./conftest: error while loading shared libraries: libc++.so.1: cannot open shared object file: No such file or directory
no
checking whether MM_MALLOC works... ./conftest: error while loading shared libraries: libc++.so.1: cannot open shared object file: No such file or directory
no
configure: creating ./config.status
config.status: creating src/Makevars

Reproducible example

This can be reproduced with docker.

docker run \
    --rm \
    --env MAKEFLAGS=-j4 \
    --platform linux/amd64 \
    -v $(pwd):/opt/LightGBM \
    -w /opt/LightGBM \
    -it ghcr.io/r-hub/containers/clang18:latest \
    bash

sh build-cran-package.sh --no-build-vignettes

Rscript -e "install.packages(c('R6', 'data.table', 'jsonlite'), repos = 'https://cran.r-project.org')"
R CMD INSTALL ./lightgbm_4.4.0.99.tar.gz

Environment info

LightGBM version or commit hash: f8ec57b

Additional Comments

N/A

[jameslamb]

I do see one copy of libstdc++.so.1.

find / -name 'libc++.so.1'
# /usr/lib/llvm-18/lib/libc++.so.1

And it looks like clang++-18 is a symlink to that location.

which clang++-18
# /usr/bin/clang++-18

stat /usr/bin/clang++-18

  File: /usr/bin/clang++-18 -> ../lib/llvm-18/bin/clang++
  Size: 26              Blocks: 0          IO Block: 4096   symbolic link
Device: 65h/101d        Inode: 1428380     Links: 1
Access: (0777/lrwxrwxrwx)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-07-14 19:13:14.000000000 +0000
Modify: 2024-07-14 19:13:14.000000000 +0000
Change: 2024-07-18 01:22:36.113571002 +0000
 Birth: 2024-07-18 01:22:36.113571002 +0000

Helping the linker find that library resolves the issues.

LD_LIBRARY_PATH="/usr/lib/llvm-18/lib:${LD_LIBRARY_PATH}" \
R CMD INSTALL ./lightgbm_4.4.0.99.tar.gz

But that really shouldn't be necessary (and isn't in any of the other clang{nn} images from r-hub). I'll put up a PR here to temporarily resolve it and unblock CI, and report this in the repo where the images are built.

[jameslamb]

upstream report: r-hub/containers#70

PR to unblock CI: #6554

[github-actions]

This issue has been automatically locked since there has not been any recent activity since it was closed.
To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.