Release Triage for Temurin JDK21

[smlambert]

https://ci.adoptium.net/job/build-scripts/job/release-openjdk21-pipeline/1/ - all platforms except x86-64_mac
https://ci.adoptium.net/job/build-scripts/job/release-openjdk21-pipeline/2/ - x86-64_mac
https://ci.adoptium.net/job/build-scripts/job/release-openjdk21-pipeline/6/ - Alpine/aarch64

Platforms	Details	Status ✔️ or ❌
aarch64_linux	link	✔️
aarch64_mac	link	✔️
arm_linux	link	✔️
ppc64_aix	link	✔️
ppc64le_linux	link	✔️
s390x_linux	link	✔️
x86-64_alpine-linux	link	✔️
x86-64_linux	link	✔️
x86-64_mac	link	✔️
x86-64_windows	link	✔️
aarch64_alpine-linux	link	✔️

[smlambert]

Rerun some of the failing top-level targets again in AQA_Test_Pipeline, because the build pipelines were launched before cherry-picked backports of test exclusions were merged.

Pipeline	Targets rerun	Platforms
AQA_Test_Pipeline/176	extended.openjdk, extended.perf, sanity.system	x86-64_linux, x86-64_windows, aarch64_linux, aarch64_mac
AQA_Test_Pipeline/177	extended.openjdk, extended.perf, sanity.system	x86-64_mac
AQA_Test_Pipeline/178	extended.openjdk, extended.perf, sanity.system	x86-64_alpine-linux,arm_linux,ppc64_aix,s390x_linux,ppc64le_linux

• extended.perf passing as dacapo_avrora is now excluded in v0.9.8-release via (v0.9.8-release) - exclude test dacapo-avrora ver=21+ #4775
• sanity.system failures are due to Exclude java.lang.Compiler from classloading test on JDK 21+ aqa-systemtest#485 not being available in v0.9.8 tagged version of aqa-systemtest. We can update the testenv.properties file in v0.9.8-release branch to point at aqa-systemtest tag v0.9.8.1 to pick up this change.

[smlambert]

aarch64_linux - ignore the ClassLoadingTest_5m failure and failed jobs are as following:

Test_openjdk21_hs_extended.openjdk_aarch64_linux ⚠️ UNSTABLE ⚠️
Rerun failed

java -version
jdk_net_1 => deep history 0/1 passed | possible issues

timeout rerun https://ci.adoptium.net/job/Grinder/7578/ - passed

[smlambert]

aarch64_mac - ignore the ClassLoadingTest_5m failure no other failures

[smlambert]

ppc64_aix -

Test_openjdk21_hs_sanity.openjdk_ppc64_aix ⚠️ UNSTABLE ⚠️
Rerun failed

java -version

openjdk version "21" 2023-09-19 LTS
OpenJDK Runtime Environment Temurin-21+35 (build 21+35-LTS)
OpenJDK 64-Bit Server VM Temurin-21+35 (build 21+35-LTS, mixed mode)

jdk_lang_1 => deep history 0/37 passed | possible issues
jdk_lang_0 => deep history 0/37 passed | possible issues
jdk_security4_0 => deep history 14/34 passed | possible issues
jdk_security4_1 => deep history 14/34 passed | possible issues
jdk_foreign_0 => deep history 8/36 passed | possible issues
jdk_foreign_1 => deep history 8/36 passed | possible issues

java/foreign/TestLargeSegmentCopy.java.TestLargeSegmentCopy
java/foreign/TestLargeSegmentCopy.java.TestLargeSegmentCopy
sun/security/krb5/auto/Cleaners.java.Cleaners
sun/security/krb5/auto/Cleaners.java.Cleaners
sun/security/krb5/auto/NoAddresses.java.NoAddresses
sun/security/krb5/auto/NoAddresses.java.NoAddresses
java/lang/String/concat/IntegerMinValue.java.IntegerMinValue
java/lang/String/concat/IntegerMinValue.java.IntegerMinValue
rerun on test-osuosl-aix72-ppc64-1 https://ci.adoptium.net/job/Grinder/7583/ -- sun/security/krb5/auto/NoAddresses passed.

Grinder_20230920145743.tap.zip

java/lang/String/concat/IntegerMinValue.java - https://bugs.openjdk.org/browse/JDK-8306007 , should be excluded or update to use the XLC17 compiler?.Action : should or could we update to XLC17? current xlc version : Version: 13.01.0003.0007
Both java/foreign/TestLargeSegmentCopy.java and sun/security/krb5/auto/Cleaners.java has been problemlisted and excluded in the nightly run. Not sure why it's running in the release run.

rerun https://ci.adoptium.net/job/Grinder/7595/
rerun https://ci.adoptium.net/job/Grinder/7597/

[smlambert]

x86-64_linux - ignore the ClassLoadingTest_5m failure no other failures

[smlambert]

x86-64_mac - ignore the ClassLoadingTest_5m failure and failed jobs are as following:

Test_openjdk21_hs_sanity.openjdk_x86-64_mac ⚠️ UNSTABLE ⚠️
Rerun failed

java -version

openjdk version "21" 2023-09-19 LTS
OpenJDK Runtime Environment Temurin-21+35 (build 21+35-LTS)
OpenJDK 64-Bit Server VM Temurin-21+35 (build 21+35-LTS, mixed mode, sharing)

jdk_util_0 => deep history 17/38 passed | possible issues
jdk_util_1 => deep history 17/38 passed | possible issues

Know issue #4751 rerun on mac12 https://ci.adoptium.net/job/Grinder/7577/ - passed as expected.

[smlambert]

x86-64_windows - ignore the ClassLoadingTest_5m failure and no other failures.

[Haroon-Khel]

ppc64le_linux ✅

Test_openjdk21_hs_extended.openjdk_ppc64le_linux ⚠️ UNSTABLE ⚠️
Rerun failed

java -version

openjdk version "21" 2023-09-19 LTS
OpenJDK Runtime Environment Temurin-21+35 (build 21+35-LTS)
OpenJDK 64-Bit Server VM Temurin-21+35 (build 21+35-LTS, mixed mode, sharing)

jdk_management_1 => deep history 0/1 passed | possible issues
jvm_compiler_1 => deep history 0/1 passed | possible issues
jdk_management_0 => deep history 0/1 passed | possible issues
jdk_tools_1 => deep history 0/1 passed | possible issues
jdk_tools_0 => deep history 0/1 passed | possible issues
jdk_jfr_0 => deep history 0/1 passed | possible issues
jdk_jfr_1 => deep history 0/1 passed | possible issues

Rerun https://ci.adoptium.net/job/Grinder/7580/console
jvm_compiler passed, the rest failed. Rerunning on many machines: grinders 7615 to 7623
https://ci.adoptium.net/job/Grinder/7616/console passed ✅
ppc64le_grinder_tap.zip

Test_openjdk21_hs_sanity.openjdk_ppc64le_linux ⚠️ UNSTABLE ⚠️
Rerun failed

java -version

openjdk version "21" 2023-09-19 LTS
OpenJDK Runtime Environment Temurin-21+35 (build 21+35-LTS)
OpenJDK 64-Bit Server VM Temurin-21+35 (build 21+35-LTS, mixed mode, sharing)

jdk_util_0 => deep history 29/34 passed | possible issues
jdk_util_1 => deep history 29/34 passed | possible issues

Rerun https://ci.adoptium.net/job/Grinder/7581/console ✅
Grinder_tap.zip

Test_openjdk21_hs_sanity.system_ppc64le_linux ⚠️ UNSTABLE ⚠️
Rerun failed

java -version

openjdk version "21" 2023-09-19 LTS
OpenJDK Runtime Environment Temurin-21+35 (build 21+35-LTS)
OpenJDK 64-Bit Server VM Temurin-21+35 (build 21+35-LTS, mixed mode, sharing)

ClassLoadingTest_5m_1 => deep history 1/33 passed | possible issues
ClassLoadingTest_5m_0 => deep history 1/33 passed | possible issues

Rerun with updated system test branch https://ci.adoptium.net/job/Grinder/7582/console
Can ignore classloading test failures

[Haroon-Khel]

s390x_linux ✅

Test_openjdk21_hs_sanity.system_s390x_linux ⚠️ UNSTABLE ⚠️
Rerun failed

java -version

openjdk version "21" 2023-09-19 LTS
OpenJDK Runtime Environment Temurin-21+35 (build 21+35-LTS)
OpenJDK 64-Bit Server VM Temurin-21+35 (build 21+35-LTS, mixed mode, sharing)

TestJlmRemoteClassAuth_0 => deep history 32/35 passed | possible issues
TestJlmRemoteClassNoAuth_0 => deep history 33/35 passed | possible issues
TestJlmRemoteClassAuth_1 => deep history 33/35 passed | possible issues
TestJlmRemoteClassNoAuth_1 => deep history 33/35 passed | possible issues
TestJlmRemoteMemoryAuth_0 => deep history 33/35 passed | possible issues
TestJlmRemoteMemoryAuth_1 => deep history 33/35 passed | possible issues
TestJlmRemoteNotifierProxyAuth_1 => deep history 33/35 passed | possible issues
TestJlmRemoteMemoryNoAuth_0 => deep history 33/35 passed | possible issues
TestJlmRemoteMemoryNoAuth_1 => deep history 33/35 passed | possible issues
TestJlmRemoteNotifierProxyAuth_0 => deep history 33/35 passed | possible issues
TestJlmRemoteThreadAuth_1 => deep history 32/35 passed | possible issues
TestJlmRemoteThreadAuth_0 => deep history 33/35 passed | possible issues
TestJlmRemoteThreadNoAuth_0 => deep history 33/35 passed | possible issues
ClassLoadingTest_5m_1 => deep history 2/35 passed | possible issues
TestJlmRemoteThreadNoAuth_1 => deep history 33/35 passed | possible issues
ClassLoadingTest_5m_0 => deep history 2/35 passed | possible issues
NioLoadTest_5m_0 => deep history 33/35 passed | possible issues
NioLoadTest_5m_1 => deep history 33/35 passed | possible issues

Ignoring the classloading test failures, running https://ci.adoptium.net/job/Grinder/7585/console ✅

Waiting for extended.openjdk to finish running https://ci.adoptium.net/job/Test_openjdk21_hs_extended.openjdk_s390x_linux/17/ ✅

[Haroon-Khel]

arm_linux ✅ (2 outstanding test failures are known upstream issues, excluding here #4785)

Test_openjdk21_hs_extended.functional_arm_linux ⚠️ UNSTABLE ⚠️ ✅
Rerun failed

java -version

openjdk version "21" 2023-09-19 LTS
OpenJDK Runtime Environment Temurin-21+35 (build 21+35-LTS)
OpenJDK Server VM Temurin-21+35 (build 21+35-LTS, mixed mode, sharing)

CryptoTests_0 => deep history 0/32 passed | possible issues

Rerunning https://ci.adoptium.net/job/Grinder/7586/console
Still fails due to timeout, open issue #4336
Comment in linked issue suggests its intermittent. Rerunning on all arm_linux machines for 10/10 iterations: grinders 7625 to 7628
Seeing intermittent passes: https://ci.adoptium.net/job/Grinder/7628/tapTestReport/ has 5 passes of CryptoTests_0
Cryptotests_grinder.zip

Test_openjdk21_hs_sanity.openjdk_arm_linux ⚠️ UNSTABLE ⚠️
Rerun failed

java -version

openjdk version "21" 2023-09-19 LTS
OpenJDK Runtime Environment Temurin-21+35 (build 21+35-LTS)
OpenJDK Server VM Temurin-21+35 (build 21+35-LTS, mixed mode, sharing)

jdk_lang_2 => deep history 4/37 passed | possible issues

Rerunning https://ci.adoptium.net/job/Grinder/7593/console

java/lang/ScopedValue/StressStackOverflow.java

java.lang.RuntimeException: java.lang.RuntimeException: java.lang.IllegalStateException: Subtask not completed or did not complete successfully
	at StressStackOverflow.lambda$run$1(StressStackOverflow.java:208)
	at java.base/jdk.internal.vm.ScopedValueContainer.runWithoutScope(ScopedValueContainer.java:112)
	at java.base/jdk.internal.vm.ScopedValueContainer.run(ScopedValueContainer.java:98)
	at java.base/java.lang.ScopedValue$Carrier.run(ScopedValue.java:510)

Deep history shows that it started failing in May
Known failure on x64 linux https://bugs.openjdk.org/browse/JDK-8313260. Passes on jdk22, needs a backport

Test_openjdk21_hs_extended.openjdk_arm_linux ⚠️ UNSTABLE ⚠️
Rerun failed

java -version

openjdk version "21" 2023-09-19 LTS
OpenJDK Runtime Environment Temurin-21+35 (build 21+35-LTS)
OpenJDK Server VM Temurin-21+35 (build 21+35-LTS, mixed mode, sharing)

jdk_jfr_2 => deep history 0/1 passed | possible issues
jdk_io_2 => deep history 0/1 passed | possible issues

Rerunning https://ci.adoptium.net/job/Grinder/7592/console

java/io/File/GetXSpace.java

FAILED: '/ usable space': 97777729536 > 1139773440
Warning: us > s.free()
FAILED: '/ usable vs. free space': 97777729536 > 3305226240
6 tests: 6 failure(s); first: java.lang.RuntimeException: '/ total space': 125693825024 != 1139773440
java.lang.RuntimeException: 2 test(s) failed
	at GetXSpace.main(GetXSpace.java:485)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at com.sun.javatest.regtest.agent.MainWrapper$MainTask.run(MainWrapper.java:138)
	at java.base/java.lang.Thread.run(Thread.java:1583)

Rerunning on multiple machines, grinders 7599 to 7602
Failed in all grinders 10/10 times
Known linux failure https://bugs.openjdk.org/browse/JDK-8251017

Test_openjdk21_hs_sanity.system_arm_linux ⚠️ UNSTABLE ⚠️
Rerun failed

java -version

openjdk version "21" 2023-09-19 LTS
OpenJDK Runtime Environment Temurin-21+35 (build 21+35-LTS)
OpenJDK Server VM Temurin-21+35 (build 21+35-LTS, mixed mode, sharing)

ClassLoadingTest_5m_2 => deep history 2/34 passed | possible issues

Classloading failures can be ignored

[Haroon-Khel]

x86-64_alpine-linux ✅

Test_openjdk21_hs_sanity.openjdk_x86-64_alpine-linux ⚠️ UNSTABLE ⚠️
Rerun failed

java -version

openjdk version "21" 2023-09-19 LTS
OpenJDK Runtime Environment Temurin-21+35 (build 21+35-LTS)
OpenJDK 64-Bit Server VM Temurin-21+35 (build 21+35-LTS, mixed mode, sharing)

jdk_security4_0 => deep history 29/33 passed | possible issues
jdk_security4_1 => deep history 29/33 passed | possible issues

Rerunning https://ci.adoptium.net/job/Grinder/7589/console ✅
grinder_tap.zip

Test_openjdk21_hs_extended.openjdk_x86-64_alpine-linux ⚠️ UNSTABLE ⚠️
Rerun failed

java -version

openjdk version "21" 2023-09-19 LTS
OpenJDK Runtime Environment Temurin-21+35 (build 21+35-LTS)
OpenJDK 64-Bit Server VM Temurin-21+35 (build 21+35-LTS, mixed mode, sharing)

jdk_net_1 => deep history 0/1 passed | possible issues
jdk_build_0 => deep history 0/1 passed | possible issues
jdk_build_1 => deep history 0/1 passed | possible issues

Rerunning https://ci.adoptium.net/job/Grinder/7590/console

jdk_build tests pass, jdk_net tests fail. Needs test image

Rerunning with test image https://ci.adoptium.net/job/Grinder/7598/console
Still one test failure:

java/net/httpclient/HttpClientLocalAddrTest.java
Program `/home/jenkins/workspace/Grinder/openjdkbinary/j2sdk-image/bin/java' timed out (timeout set to 960000ms, elapsed time including timeout handling was 960278ms).   

Rerunning on all available machines 10/10 iterations, grinders 7633 to 7636
https://ci.adoptium.net/job/Grinder/7636/console passed ✅
x64alpine_grinder_tap.zip

Test_openjdk21_hs_sanity.system_x86-64_alpine-linux ⚠️ UNSTABLE ⚠️
Rerun failed

java -version

openjdk version "21" 2023-09-19 LTS
OpenJDK Runtime Environment Temurin-21+35 (build 21+35-LTS)
OpenJDK 64-Bit Server VM Temurin-21+35 (build 21+35-LTS, mixed mode, sharing)

ClassLoadingTest_5m_0 => deep history 2/36 passed | possible issues
ClassLoadingTest_5m_1 => deep history 2/36 passed | possible issues

Classloading tests can be ignored