NCAR-20250225 PR

[jiandewang]

With one and half months of back and forth collaborations among NCAR, PSL and me, this PR (mom-ocean#1653) is reaching its final stage. However the ifx testing case on hercules failed to compile one of the newly added MOM6 MARBL code although it is able to be compiled and run on orion and c5. So it is very likely due to the ifx compiler version that cause the failure on hercules.

Since this PR has been on queue for long time and GFDL has one pending PR that are ready to be issued right after this. I suggest we disable ifx test cases on hercules temporary and move forward for this PR, otherwise MOM6 repo. updating will be stacked for the whole community. This will also give time for NCAR developers to dive in the MARBL code and hopefully they can figure out and make patch fixing at later stage.

@junwang-noaa @sanAkel @DusanJovic-NOAA @DeniseWorthen Please let me know your opinion

the compiling error can be found on hercules at
/work2/noaa/stmp/jiwang/stmp/jiwang/FV3_RT/rt_1743599/compile_datm_cdeps_intelllvm/err

/local/scratch/jiwang/5023824/ifx1893972899iIDK5a/ifxJNwLgY.i90: error #5633: Internal compiler error: segmentation violation signal raised
Please report this error along with the circumstances in which it occurred in a Software Problem Report. Note: File and line given may not be e
xplicit cause of this error.
compilation aborted for /work/noaa/marine/Jiande.Wang/HERCULES/MOM6-update/NCAR-20250225-fix-SP/ufs-weather-model/MOM6-interface/MOM6/src/tracer
/MARBL_tracers.F90 (code 3)

[DusanJovic-NOAA]

Instead of disabling it completely, we can move the test that fails to compile on Hercules to Orion.

[jiandewang]

@DusanJovic-NOAA good suggestion, let me take a try on orion

[jiandewang]

actually I had a typo in my original message, MOM6 code is not tested on orion for ifx but on c5. Let me try on c5

[mnlevy1981]

What version of ifx are you using? Searching for information about the error, I found a report about the same issue when building FMS in CESM; that particular issue was slated to be fixed in v2024.2, but if you are seeing this error in a more recent compiler version it might be worth reporting to intel.

Also from that thread - do you get the internal compiler error if you build with -O0? If that flag lets you build the module, does your build system support using that level of optimization for just the single file? I assume you are not setting USE_MARBL_TRACERS=TRUE in MOM_input, so I don't think you well ever call any of the routines in MARBL_tracers.F90 (the get_param() call for that variable is in MOM_tracer_flow_control.F90, and then that module should skip all the calls into MARBL_tracers since the parameter is set to FALSE)... so you won't affect performance by changing that particular module's optimization setting.

[jiandewang]

@mnlevy1981 you are right, USE_MARBL_TRACER is F in UFS.
I am testing @DusanJovic-NOAA 's suggestion by moving to gaea c5, hopefully they can be compiled and run. In this way we can move forward for this PR and buy time for you to dive in.
I am not familiar wih ifx part but Dusan shall be able to give you information on what you asked.

[jiandewang]

@DusanJovic-NOAA move ifx job from hercules to c5 worked. Thanks for your good idea. I will approve MOM6 PR to let it move forward.