Integration of Clear Linux patches

[InBetweenNames]

Clear Linux maintains a number of performance related patches for open source projects. There are quite a few: https://github.com/clearlinux-pkgs

It would be interesting to integrate these into GentooLTO somehow, either synced into the overlay or through a user install mechanism.

[aw1cks]

I didn't look thoroughly, but if I'm seeing it right, these are just patches in their git repo? It would be very easy to put the patches into /etc/portage/patches, is there some downside to this?

[InBetweenNames]

It appears that way, however I'm guessing they also fine tune compiler options as well. Also, I'm unsure if they use icc or gcc for their builds. I'm guessing it depends on the package. At the very least, we could include the source code patches, I think.

[InBetweenNames]

Example: https://github.com/clearlinux-pkgs/fftw/blob/master/fftw.spec

[InBetweenNames]

Interesting flag: -fno-semantic-interposition. I'll be adding this one to the defaults I think.

[InBetweenNames]

Other interesting things: they appear to enable -ffast-math on certain packages, or flags that -ffast-math enables at the very least. Also interesting: they force all functions to be aligned on 32 byte boundaries with -falign-functions=32. I wonder if they do that for AVX512 compatibility? It'd be interesting to see what the benefits are for overaligning functions, if any.

[InBetweenNames]

It appears that -falign-functions=32 has some kind of impact on autovectorization. By default on x86_64, it is set to 16.

[InBetweenNames]

I just tested an architecture that has AVX512 instructions:

> gcc -march=skylake-avx512 -flto -Ofast -Q --help=optimizer | grep falign-functions
  -falign-functions                     [disabled]
  -falign-functions=                    16

It seems by default on x86_64, -falign-functions is always set 16 (as long as -O2 or higher is specified).

[Althorion]

Phoronix claims that they build using GCC/Clang (differs from package to package).

[InBetweenNames]

Makes sense - icc probably doesn't support a lot of the GNU extensions that are used out there in the wild. Not to mention, GCC is highly competitive with ICC when the right options are used. I'm thinking I may update the recommendations for -falign-functions, not adding it by default but mentioning it in make.conf.lto. The reason being, I want to support more than just Intel processors, or even just x86_64, and this feels very much like an Intel-specific thing.

[funghetto]

I heard that also Solus is using some of their optimizations.

[wolfwood]

@InBetweenNames one thing that clear linux does that isn't really necessary for us is function multiversioning. I think the linker links different functions based on eg. AVX support. since everyone here is probably building with -march=native we can get smaller binaries than Clear can, may have better LTO opportunities, etc.

It would be interesting to see if we can get Michael to bench lto-overlay vs. Clear, esp. once we steal some of their fancy tricks.

regarding -falign-functions I'd expect this to be about aligned jumps/instruction cache line reads. maybe RIP relative addressing? but once you are executing in the function instruction alignment is going to be off no matter what you do thanks to variable length instructions. even if AVX cared about instruction alignment, I'm not sure this would help.

is it possible that there is a more compact way to load immediates if they are 32-aligned, so function call sites are smaller?

[InBetweenNames]

Agreed -- in fact we should do even better than function multi versioning since we're compiling our system exactly tailored for the system it's running on. This means more opportunities for LTO all around. Not to mention, this should be highly portable across many architectures.

If one could link their system using mainly static libraries, I bet the LTO benefits would be even more profound. You can link-optimize across static library boundaries, and you can't do that with shared objects. I don't believe this is possible as-is however, since Portage seems to really prefer shared objects, and configure scripts, etc, also prefer shared objects.

I was wondering the same about -falign-functions today, but there are other -falign-* flags that affect those other cases you mention. I hadn't considered immediate operands however. It might make a difference if there's some static storage for a function as well. I've been looking around all day for more uses of -falign-functions=32 and I've been having serious trouble. I found a slide deck:

http://hpac.rwth-aachen.de/teaching/sem-accg-16/slides/08.Schmitz-GGC_Autovec.pdf

I also found a StackOverflow question that indirectly touches on it:

https://stackoverflow.com/questions/19470873/why-does-gcc-generate-15-20-faster-code-if-i-optimize-for-size-instead-of-speed

If I pass g++ -O2 -falign-functions=16 -falign-loops=16 then everything is back to normal: I always get the fastest case and the time isn't sensitive to the -fno-omit-frame-pointer flag anymore. I can pass g++ -O2 -falign-functions=32 -falign-loops=32 or any multiples of 16, the code is not sensitive to that either.

Without delving in the GCC internals, I can't find many resources that recommend this flag. I'll see if the Intel guys will shed some light on it.

[InBetweenNames]

More goodies: -fno-common

Line 481: https://github.com/clearlinux/autospec/blame/master/autospec/specfiles.py

In that commit, -fno-common is mentioned.
Detailed here: https://gcc.gnu.org/onlinedocs/gcc/Code-Gen-Options.html -- seems beneficial when it would work.

I find it interesting they enable -fno-math-errno, -fno-trapping-math by default. It's not -ffast-math, but it's partway there.

>gcc -march=skylake-avx512 -flto -Ofast -Q --help=common | grep fcommon
  -fcommon                              [enabled]

Even with the most aggressive optimization package, this is on by default.

[InBetweenNames]

Found when it was added: https://github.com/clearlinux/autospec/blame/a5260d7ce751774d46e0a957786d179456a14275/autospec/buildpattern.py

It was added by @fenrus75 for "high speed cases". Interesting.

[InBetweenNames]

I notice that on packages that are optimized for size, they enable -ffunction-sections and -fdata-sections for dead code removal, along with a -Wl,--gc-sections. However, these are two flags I want to research more before enabling by default -- I'm unsure how these interact with LTO. I assumed that LTO kind of did dead code elimination on its own, since the entire program would be visible at link time (minus definitions in shared objects).