CMake Build flags incompatible with AMD processors

[aidanheerdegen]

Currently-xCORE-AVX2 flags are set in the CMake build:

https://github.com/ACCESS-NRI/MOM5/blob/master/cmake/CMakeLists.txt#L54-L58

This causes an error on AMD architectures. We have a Pawsey porting goal which uses a similar architecture, which would also trip over the same issue.

[access-hive-bot]

This issue has been mentioned on ACCESS Hive Community Forum. There might be relevant details there:

https://forum.access-hive.org.au/t/installing-access-om2-on-nesi-new-zealand-supercomputer/5566/38

[harshula]

ACCESS-NRI/ACCESS-OM2@0df41e7

* David Hutchinson had experienced runtime issues with the `-xCORE-AVX2`
  compiler flag on NeSI HPC in the past. His first run of ACCESS-OM2 built
  by the oneapi compiler had the runtime error:
  "Please verify that both the operating system and the processor support Intel(R) X87, CMOV, MMX, SSE, SSE2, SSE3, SSSE3, SSE4_1, SSE4_2, MOVBE, POPCNT, AVX, F16C, FMA, BMI, LZCNT and AVX2 instructions."

* I found the following advice: https://community.intel.com/t5/Intel-Fortran-Compiler/Compiler-flags-for-AMD-Epyc-processors/td-p/1150137:
  "The -x options (-xHost excepted), as the disclaimer notes, reserve some optimizations for Intel processors and add a check at program start that gives an error if the CPU type doesn't match. The -m or -arch options omit this check."

* I discussed this with my colleague, Manodeep Sinha, and he suggested using
  `-mavx2`.

[dougiesquire]

We have -x options set in a few different packages at a few different levels (CMake, SPRs). Is the recommended way forward to set -m/-march/-mtune flags in the spack.yaml? If so, I can start updating everything and checking that we can preserve answers.

ADDED: possibly this is a question for @manodeep

[harshula]

Hi @dougiesquire , I think it's time to remove -xCORE-AVX2 flag from the build system of each model component git repository. e.g. MOM5.

[manodeep]

We have -x options set in a few different packages at a few different levels (CMake, SPRs). Is the recommended way forward to set -m/-march/-mtune flags in the spack.yaml? If so, I can start updating everything and checking that we can preserve answers.

Yes to changing to the -x flags to -march. Both gcc and intel oneapi accept -mavx2 or -march=core-avx2; not sure about the aocc compilers on Setonix. Looks like gcc also sets the -mtune flag for core-avx2, but for maximal benefit mtune should be provided with a cpu name (e.g., haswell) rather than the generic instruction set name.

[~ @gadi-login-03]$ module purge
[~ @gadi-login-03]$ module load intel-compiler-llvm/2025.2.0
[~ @gadi-login-03]$ icx -O2 -march=core-avx2 -dM -E  - < /dev/null | grep -i avx
#define __AVX2__ 1
#define __AVX__ 1
[~ @gadi-login-03]$ icx -O2 -mavx2 -dM -E  - < /dev/null | grep -i avx
#define __AVX2__ 1
#define __AVX__ 1
[~ @gadi-login-03]$ module purge
[~ @gadi-login-03]$ module load gcc/15.1.0 
[~ @gadi-login-03]$ gcc -O2 -march=core-avx2 -dM -E  - < /dev/null | grep -i avx
#define __core_avx2 1
#define __core_avx2__ 1
#define __tune_core_avx2__ 1
#define __AVX__ 1
#define __AVX2__ 1
[~ @gadi-login-03]$ gcc -O2 -mavx2 -dM -E  - < /dev/null | grep -i avx
#define __AVX__ 1
#define __AVX2__ 1

On a related note, do we know that the NeSI CPUs are AMD - because then the failure is expected for an exe built with -x<arch> flag. The content from /proc/cpuinfo would help.

[harshula]

Still open from Feb 2023: ACCESS-NRI/ACCESS-OM2#12

[harshula]

... and ACCESS-NRI/FMS#1

[dougiesquire]

Thanks both. I was really trying to ask if the MDR spack.yaml is the right place to set these flags (as opposed to in the repo, e.g. CMakeLists, or in the SPRs)? We currently have a mixture of all three

[harshula]

Hi @dougiesquire , https://github.com/ACCESS-NRI/dev-docs/wiki/Spack#compiler-flags

[dougiesquire]

Thanks @harshula. I think my reading of that is that they should be set in the source package (although these are not required flags...)? Is that what you are saying?

[aidanheerdegen]

although these are not required flags

Aren't they though? If this is how we want to build the package then I'd say they're required.

[dougiesquire]

Well my thinking with that comment was that they might end up being different depending on where you want to build the package

[aidanheerdegen]

I thought that was covered in this sentence:

The source package can contain compiler flags that are optional and can be enabled/disabled by the user. These compiler flags can be manipulated from the SPR. Consider grouping the compiler flags as required, optional and optimisation flags. Allow the user to override the optional and optimisation compiler flags.

So location or hardware specific flags could be enabled as variants, or through compiler detection logic.

But if it isn't clear please do suggest a change to the wording to include the specific use-case of optimisation for different compiler/hardware targets.

[dougiesquire]

Ta, I misunderstood how cpu-restrictive the -mavx2 flag is (ie not restrictive) but @manodeep has set me straight