vm.mmap_rnd_bits=32 cause issues for MSAN, LSAN and ASAN

[chitao1234]

Command to generate a minimal program for test

echo 'int main(){}' | gcc -fsanitize=address -x c++ -
echo 'int main(){}' | clang -fsanitize=address -x c++ -

For ASAN on gcc, it will output AddressSanitizer:DEADLYSIGNAL repetedly.
For MSAN on clang, it will output the following

FATAL: Code 0x6412707d8410 is out of application range. Non-PIE build?
FATAL: MemorySanitizer can not mmap the shadow memory.
FATAL: Make sure to compile with -fPIE and to link with -pie.
FATAL: Disabling ASLR is known to cause this error.
FATAL: If running under GDB, try 'set disable-randomization off'.
==22584==Process memory map follows:
	0x64127078e000-0x6412707af000	/home/chi/a.out
	0x6412707af000-0x641270836000	/home/chi/a.out
	0x641270836000-0x641270862000	/home/chi/a.out
	0x641270862000-0x641270863000	/home/chi/a.out
	0x641270863000-0x641270866000	/home/chi/a.out
	0x641270866000-0x6412721ba000	
	0x79c83dd00000-0x79c83de00000	
	0x79c83df00000-0x79c83e000000	
	0x79c83e100000-0x79c83e200000	
	0x79c83e300000-0x79c83e400000	
	0x79c83e473000-0x79c83e814000	
	0x79c83e814000-0x79c83e83a000	/usr/lib/x86_64-linux-gnu/libc.so.6
	0x79c83e83a000-0x79c83e98f000	/usr/lib/x86_64-linux-gnu/libc.so.6
	0x79c83e98f000-0x79c83e9e2000	/usr/lib/x86_64-linux-gnu/libc.so.6
	0x79c83e9e2000-0x79c83e9e6000	/usr/lib/x86_64-linux-gnu/libc.so.6
	0x79c83e9e6000-0x79c83e9e8000	/usr/lib/x86_64-linux-gnu/libc.so.6
	0x79c83e9e8000-0x79c83e9f5000	
	0x79c83e9f5000-0x79c83e9f8000	/usr/lib/x86_64-linux-gnu/libgcc_s.so.1
	0x79c83e9f8000-0x79c83ea0f000	/usr/lib/x86_64-linux-gnu/libgcc_s.so.1
	0x79c83ea0f000-0x79c83ea13000	/usr/lib/x86_64-linux-gnu/libgcc_s.so.1
	0x79c83ea13000-0x79c83ea14000	/usr/lib/x86_64-linux-gnu/libgcc_s.so.1
	0x79c83ea14000-0x79c83ea15000	/usr/lib/x86_64-linux-gnu/libgcc_s.so.1
	0x79c83ea15000-0x79c83ea25000	/usr/lib/x86_64-linux-gnu/libm.so.6
	0x79c83ea25000-0x79c83ea98000	/usr/lib/x86_64-linux-gnu/libm.so.6
	0x79c83ea98000-0x79c83eaf2000	/usr/lib/x86_64-linux-gnu/libm.so.6
	0x79c83eaf2000-0x79c83eaf3000	/usr/lib/x86_64-linux-gnu/libm.so.6
	0x79c83eaf3000-0x79c83eaf4000	/usr/lib/x86_64-linux-gnu/libm.so.6
	0x79c83eafe000-0x79c83eb09000	
	0x79c83eb09000-0x79c83eb0a000	/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
	0x79c83eb0a000-0x79c83eb2f000	/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
	0x79c83eb2f000-0x79c83eb39000	/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
	0x79c83eb39000-0x79c83eb3b000	/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
	0x79c83eb3b000-0x79c83eb3d000	/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
	0x7ffdcfcd4000-0x7ffdcfcf5000	[stack]
	0x7ffdcfdb1000-0x7ffdcfdb5000	[vvar]
	0x7ffdcfdb5000-0x7ffdcfdb7000	[vdso]
==22584==End of process memory map.

For LSAN on both gcc and clang, and ASAN on clang, it will only output ​Segmentation fault.
It not all the time, but about 4 out of 10 times.
What worth noting is that it happens only when vm.mmap_rnd_bits=32, not any other value between 28 to 31.
Testing on Debian testing with gcc version

Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/12/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none:amdgcn-amdhsa
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Debian 12.2.0-14' --with-bugurl=file:///usr/share/doc/gcc-12/README.Bugs --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++,m2 --prefix=/usr --with-gcc-major-version-only --program-suffix=-12 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-plugin --enable-default-pie --with-system-zlib --enable-libphobos-checking=release --with-target-system-zlib=auto --enable-objc-gc=auto --enable-multiarch --disable-werror --enable-cet --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none=/build/gcc-12-bTRWOB/gcc-12-12.2.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-12-bTRWOB/gcc-12-12.2.0/debian/tmp-gcn/usr --enable-offload-defaulted --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 12.2.0 (Debian 12.2.0-14)

and clang version

Debian clang version 14.0.6
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/12
Selected GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/12
Candidate multilib: .;@m64
Selected multilib: .;@m64
​```

[thurstond]

As of April 2023 (https://reviews.llvm.org/D148280, https://reviews.llvm.org/D148193), 32-bits of entropy shouldn't be a problem for ASan and LSan anymore (they support 32 bits of entropy on x86-64 and 33 bits on aarch64).

[thurstond]

As of January 2024, TSan will re-exec without ASLR if the entropy is too high: llvm/llvm-project@0784b1e (it is difficult to support higher entropy due to pointer compression and the overhead of shadow mappings).