Identify which tests seem unstable in docker containers

[sxa]

This is partially for my own notes, but need to be looked at, and may also be covered elsewhere. Looks like the DDR stuff (not too surprising) will need some work

• In OpenJ9 extended.functional: testDDR* cmdLineTester and jit_hw_2 fail
• Similar in sanity.functional with cmdLineTester tests
• Intermittent failures in JDK8/J9 sanity.openjdk
• JDK11/J9 extended.system possibly fails on packet host -2 and not -1 - unrelated to docker?
• JDK11/J9 sanity.system TestIBMJlmFRemoteMemoryAuth_0 fails
• JDK15/J9 extended.system HCRLateAttachWorkload_previewEnabled_0 fails

Other's (on initial look - not too deep!) seem ok

Memo to self - how to check for RAM/CPU limits in a container:

• CPU: wc -l /sys/fs/cgroup/cpu,cpuacct/cgroup.procs (Not accurate)
• RAM: expr cat /sys/fs/cgroup/memory/memory.limit_in_bytes / 1024 / 1024 / 1024 (Or divide by 1073741824)
• Show stats: while true; do clear && uptime && docker stats --no-stream; sleep 60; done

[sxa]

NOTE - runs on the Fedora docker image testing after patching and rebooting the server:

• JDK11/J9 sanity.functional is running clean
• JDK11/J9 extended.functional (Re-run due to wrong artefact)
• JDK15/J9 sanity.functional (Re-run due to wrong artefact)
• JDK11/J9 sanity.system (Re-run)
• JDK15/J9 extended.system and same on the host system

[sxa]

Also trying on a couple of X64 docker images (Fedora 33 and Ubuntu 20.04)

• JDK8/J9 sanity.openjdk PASSED
• JDK11/HS sanity.functional PASSED
• JDK8/J9xl extended.functional
• JDK11/J9 sanity.funcational

[sxa]

NUMA interrogation is failing in Docker

[EDIT: Issue shows up with just numactl -s in the container. A resolution is to use --cap=sys_nice which gives the container access to the CPU scheduling options - se docker docs for details]

[sxa]

core dump generation is also failing (I've tried starting the container with various options that might help but to no avail ... so far) ... potentially same as described in adoptium/run-aqa#59

[EDIT: The (host) systems on which core files were not being produced had |/usr/share/apport/apport %p %s %c %d %P %E in /proc/sys/kernel/core_pattern - changing it to core resolves it (but we'll need to make that persistent) - raised https://github.com/adoptium/infrastructure/issues/1817]

[sxa]

Also not specific to docker, but we have seen instances if this when LANG is not set to en_US.UTF-8. It occurs only on OpenJ9 sanity.openjdk on JDK11 and above (not seen on 8 so far)

21:41:41  ACTION: main -- Failed. Execution failed: `main' threw exception: java.util.IllformedLocaleException: Ill-formed language: c.u [at index 0]
21:41:41  REASON: User specified action: run main/othervm -Duser.language.display=ja -Duser.language.format=zh LocaleCategory 
21:41:41  TIME:   8.802 seconds
21:41:41  messages:

This will be progressed via adoptium/run-aqa#59

[sophia-guo]

Grinder on testc-packet-fedora33-amd-2 and got

ERROR: Error cloning remote repo 'origin'
hudson.plugins.git.GitException: Command "git fetch --tags --force --progress -- https://github.com/AdoptOpenJDK/openjdk-tests.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: fatal: unable to access 'https://github.com/AdoptOpenJDK/openjdk-tests.git/': OpenSSL SSL_connect: Connection reset by peer in connection to github.com:443 

https://ci.adoptopenjdk.net/view/work-in-progress/job/grinder_sandbox/203/console

Suppose testc-packet-fedora33-amd-2 is one docker container?

[sxa]

Suppose testc-packet-fedora33-amd-2 is one docker container?

Yes - it's a docker container.

Hmmm that's a bit odd ... It's also nothing to do with the test if it's failing that early in the process. I've re-run it as 205 and it completed without any fatal failures so hopefully that won't occur, but if you see any further instances let me know so we can see if it happens regularly.

[smlambert]

From https://adoptopenjdk.slack.com/archives/C5219G28G/p1612761729068300, we should check whether the timeouthandler added to openj9 openjdk test runs is able to write a System dump in dockerized environment.

[knn-k]

I wonder if eclipse-openj9/openj9#12038 is another example of failure in docker environments or not.
"AssertionError: Free Physical Memory size cannot be greater than total Physical Memory Size."

[sxa]

I wonder if eclipse-openj9/openj9#12038 is another example of failure in docker environments or not.
"AssertionError: Free Physical Memory size cannot be greater than total Physical Memory Size."

Hmmm interesting thought. Certainly possibly but this is the first I've heard of it. Some of those containers we have are called in terms of CPU and RAM which could explain why you wouldn't necessarily be able to replicate locally without doing the same.

[jerboaa]

sanity.openjdk on JDK 8 (Hotspot) seems to randomly fail for these tests:

java/util/Arrays/TimSortStackSize2.java.TimSortStackSize2
java.lang.OutOfMemoryError: Java heap space
	at TimSortStackSize2.createArray(TimSortStackSize2.java:164)
	at TimSortStackSize2.doTest(TimSortStackSize2.java:59)
	at TimSortStackSize2.main(TimSortStackSize2.java:43)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127)
	at java.lang.Thread.run(Thread.java:748)

java/util/ResourceBundle/Bug4168625Test.java.Bug4168625Test 
14:10:19  ACTION: main -- Error. Agent communication error: java.io.EOFException; check console log for any additional details

java/lang/invoke/LFCaching/LFSingleThreadCachingTest.java.LFSingleThreadCachingTest 
Unexpected exit from test [exit code: 137]

See:
https://ci.adoptopenjdk.net/view/Test_upstream/job/Test_openjdk8_hs_sanity.openjdk_x86-64_linux_upstream/75/

Especially LFSingleThreadCachingTest.java looks like an OOM kill. Would be nice to overlay that failure with the kernel OOM kill logs.