[BUG] DEB systemd test failed as swap syscall not being blocked

[peterzhuamazon]

Describe the bug

[BUG] DEB systemd test failed as swap syscall not being blocked

> Task :qa:systemd-test:integTest

SystemdIntegTests > testSystemCallFilter FAILED
    java.lang.AssertionError: Swap system call should be blocked
        at __randomizedtesting.SeedInfo.seed([D94D25E53E38C947:850EC0A576478592]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.assertTrue(Assert.java:42)
        at org.opensearch.systemdinteg.SystemdIntegTests.testSystemCallFilter(SystemdIntegTests.java:148)

Tests with failures:
 - org.opensearch.systemdinteg.SystemdIntegTests.testSystemCallFilter

> Task :qa:systemd-test:integTest FAILED
60 actionable tasks: 1 executed, 59 up-to-date

Thanks.

Related component

Other

To Reproduce

See above

Expected behavior

See above

Additional Details

See above

[peterzhuamazon]

Hi @kumargu @RajatGupta02 could you take a look on this?

This is particularly showing up on DEB only, not on rpm.

Thanks.

[kumargu]

It looks like swap might not be enabled on all DEBs which fails the assertions. could we check the existence of swap before executing swapon -a

The existence of swap devices can be checked by reading contents of /proc/swap

  private boolean isSwapEnabled() throws IOException {
        String content = Files.readString(Paths.get("/proc/swaps"));
       // check if there are more than one entry 
        return content.lines().skip(1).findFirst().isPresent();
    }

[peterzhuamazon]

DEB seems very easy to stop the cluster after systemctl start:

java.lang.UnsupportedOperationException: seccomp(BOGUS_OPERATION): Operation not permitted
        at org.opensearch.bootstrap.SystemCallFilter.linuxImpl(SystemCallFilter.java:311) ~[opensearch-3.0.0-beta1.jar:3.0.0-beta1]
        at org.opensearch.bootstrap.SystemCallFilter.init(SystemCallFilter.java:666) ~[opensearch-3.0.0-beta1.jar:3.0.0-beta1]
        at org.opensearch.bootstrap.JNANatives.tryInstallSystemCallFilter(JNANatives.java:281) [opensearch-3.0.0-beta1.jar:3.0.0-beta1]
        at org.opensearch.bootstrap.Natives.tryInstallSystemCallFilter(Natives.java:128) [opensearch-3.0.0-beta1.jar:3.0.0-beta1]
        at org.opensearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:130) [opensearch-3.0.0-beta1.jar:3.0.0-beta1]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:192) [opensearch-3.0.0-beta1.jar:3.0.0-beta1]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:405) [opensearch-3.0.0-beta1.jar:3.0.0-beta1]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:168) [opensearch-3.0.0-beta1.jar:3.0.0-beta1]
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:159) [opensearch-3.0.0-beta1.jar:3.0.0-beta1]
        at org.opensearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:110) [opensearch-3.0.0-beta1.jar:3.0.0-beta1]
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) [opensearch-cli-3.0.0-beta1.jar:3.0.0-beta1]
        at org.opensearch.cli.Command.main(Command.java:101) [opensearch-cli-3.0.0-beta1.jar:3.0.0-beta1]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:125) [opensearch-3.0.0-beta1.jar:3.0.0-beta1]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:91) [opensearch-3.0.0-beta1.jar:3.0.0-beta1]

[peterzhuamazon]

opensearchcluster1.log

[peterzhuamazon]

Three steps:

1. Skip the swap check on systemd test for the time being. @RajatGupta02
   i. Disable SystemCallFilter test temporarily #18115
   ii. Remove testSystemCallFilter temporarily #18123
2. Add access for /dev/shm for the time being @RajatGupta02
   i. Add shared memory access #18114
3. Make sure deb is using the /etc/default folder instead of /etc/sysconfig for env @peterzhuamazon
   i. Make sure deb is correctly adding env file in OpenSearch opensearch-build#5479

Thanks.

[peterzhuamazon]

With #18114, both deb/rpm will fail with this error:

Apr 29 04:54:29 1e3522934a87 systemd[1]: Starting OpenSearch...
Apr 29 04:54:29 1e3522934a87 systemd[1]: opensearch.service: Main process exited, code=exited, status=226/NAMESPACE
Apr 29 04:54:29 1e3522934a87 systemd[1]: opensearch.service: Failed with result 'exit-code'.
Apr 29 04:54:29 1e3522934a87 systemd[1]: Failed to start OpenSearch.

-- Unit opensearch.service has begun starting up.
Apr 29 04:54:29 1e3522934a87 systemd[1357]: opensearch.service: Failed to set up mount namespacing: No such file or directory
Apr 29 04:54:29 1e3522934a87 systemd[1357]: opensearch.service: Failed at step NAMESPACE spawning /usr/share/opensearch/bin/systemd-entrypoint: No such file or directory
-- Subject: Process /usr/share/opensearch/bin/systemd-entrypoint could not be executed
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- The process /usr/share/opensearch/bin/systemd-entrypoint could not be executed and failed.
--
-- The error number returned by this process is 2.
Apr 29 04:54:29 1e3522934a87 systemd[1]: opensearch.service: Main process exited, code=exited, status=226/NAMESPACE
Apr 29 04:54:29 1e3522934a87 systemd[1]: opensearch.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- The unit opensearch.service has entered the 'failed' state with result 'exit-code'.
Apr 29 04:54:29 1e3522934a87 systemd[1]: Failed to start OpenSearch.
-- Subject: Unit opensearch.service has failed
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- Unit opensearch.service has failed.
--
-- The result is failed.

The thing is, /usr/share/opensearch/bin/systemd-entrypoint is on disk and would only fail if ReadWritePaths=/dev/shm/performanceanalyzer present in systemd service file.

This location is not existing by the start of the cluster and will only be created when PA is invoke, which could be the cause as the folder is not even there.

Thanks.