AIX: Testing fails because it cannot "make genParallelList"

[adamfarley]

Summary
All testing fails due to this error:

22:18:23  + cd ./aqa-tests/TKG
22:18:23  + make genParallelList TEST=sanity.openjdk TEST_TIME=120 NUM_MACHINES=
22:18:23  "makefile", line 19: make: Dependency line needs colon or double colon operator.
22:18:23  "makefile", line 20: make: Dependency line needs colon or double colon operator.
22:18:23  "makefile", line 21: make: Dependency line needs colon or double colon operator.

Details
This happens consistently but only recently, implying a syntactic intolerance of a new machine setup.

Next step

One solution is to implement the fix found here: #621

Will raise a PR.

[adamfarley]

Examples
https://ci.adoptium.net/job/Test_openjdk11_hs_sanity.openjdk_ppc64_aix/697/console
https://ci.adoptium.net/job/Test_openjdk21_hs_sanity.system_ppc64_aix/139/console
https://ci.adoptium.net/job/Test_openjdk24_hs_special.openjdk_ppc64_aix/7/

[smlambert]

Thanks @adamfarley, can you add information about when this started failing so we can pinpoint the regression.

[adamfarley]

Sure thing Shelley.

Ok, I'm assuming that the change affected different hosts at different times, based on these jobs that happened on the 23rd of August:

test-osuosl-aix72-ppc64-1 - 10pm - Job Link - Result: PASSED.
test-osuosl-aix72-ppc64-4 - 11am - Job Link - Result: FAILED.

Most affected tests seem to be passing around the 15th-21st of August (example) and failing from the 23rd-28th of August example.

[adamfarley]

Tagging @Haroon-Khel as recommended in the recent Adoptium Community Call.

Haroon, Stewart suggested you may know something about the change that caused this?

May be related to a path change on the AIX machines.

[sxa]

The potential cause would be adoptium/infrastructure#3716
Haroon posted on the 20th in the infrastructure slack channel which would have been when the change started to take effect.

[Haroon-Khel]

Yeah the path change might have caused this, taking a look on the machines

[Haroon-Khel]

@sxa to clarify, we want to use /opt/freeware/bin/make? At the moment with the new PATH variable, as a result of adoptium/infrastructure#3716, it is using /usr/bin/make

[sxa]

It needs to be GNU make - /opt/freeware/bin/make should be GNU make so that would be the one to use. I'll be honest I thought we had that special cased most of those invocations fro AIX but maybe we haven't done all of them yet.

[Haroon-Khel]

Do we ever need to use /usr/bin/make for anything? Because we can simply force /usr/bin/make to point to /opt/freeware/bin/make, instead of forcing the tests to use gmake in #5550

[Haroon-Khel]

Current setup

root@adopt08:[/root]ls -la `which make`
lrwxrwxrwx    1 bin      bin              17 May 30 2023  /usr/bin/make -> /usr/ccs/bin/make
root@adopt08:[/root]ls -la `which gmake`
lrwxrwxrwx    1 root     system           27 May 30 2023  /usr/bin/gmake -> ../../opt/freeware/bin/make

[sxa]

Do we ever need to use /usr/bin/make for anything? Because we can simply force /usr/bin/make to point to /opt/freeware/bin/make, instead of forcing the tests to use gmake in #5550

We need to get away from changing default system behavior - that was part of why we made the change to the PATH - we don't want anything relying on the AIX system tools to see differences.

adoptium/infrastructure#1979 is related to that topic too :-)

[Haroon-Khel]

Then a solution could be to set the make path in the jenkins config for each machine to /opt/freeware/bin/make

EDIT: Or just set the PATH at the jenkins config level to have /opt/freeware/bin before /usr/bin

[Haroon-Khel]

I've added PATH=/opt/freeware/bin:$PATH to the jenkins config of https://ci.adoptium.net/computer/test-osuosl-aix72-ppc64-4/, on which the sanity system test failed to compile, #5542 (comment), and rebuilt the failing job

https://ci.adoptium.net/job/Test_openjdk21_hs_sanity.system_ppc64_aix/140/console

Looks like it finds the right make

[Haroon-Khel]

I've updated the jenkins config of the machines that did not have that PATH variable to have it