Tests are failing on top of tree(443cfca) for x86_64.

[Pratikt2312]

Hi Team,
I am working on validating OpenSearch on top of tree for x86_64 architecture.
Encountered an error while running the test suite with gradlew check command.
Tests are failing with the following error:

Tests with failures:
 - org.opensearch.index.remote.RemoteSegmentTransferTrackerTests.testStatsObjectCreationViaStream

12141 tests completed, 1 failed, 29 skipped

> Task :server:test FAILED

FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':test:fixtures:azure-fixture:composeBuild'.
> Exit-code 125 when calling /usr/bin/docker, stdout:

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
==============================================================================

2: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':server:test'.
> There were failing tests. See the report at: file:///home/user3/OpenSearch/server/build/reports/tests/test/index.html
==============================================================================

BUILD FAILED in 29m 7s
1649 actionable tasks: 1640 executed, 1 from cache, 8 up-to-date

Also encountered some flaky test failures as follows:

1: Task failed with an exception.
* What went wrong:
Execution failed for task ':test:fixtures:azure-fixture:composeBuild'.
> Exit-code 125 when calling /usr/bin/docker, stdout:

2 Tests with failures:
 - org.opensearch.client.RestClientSingleHostIntegTests.testManyAsyncRequests

89 tests completed, 1 failed, 1 skipped

> Task :client:rest:test FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':client:rest:test'.

3 Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':modules:transport-netty4:test'.

Steps to reproduce the behavior:
1 Install JDK 11,Docker & Docker Compose
2 Clone the repository
3 ./gradlew
4 ./gradlew check

Expected behavior
Tests should be passed successfully.

Plugins
Standard

Host/Environment (please complete the following information):

• Red Hat Enterprise Linux 9.0

Additional context
Also tried same behaviour on ubuntu-20.04 (x86_64 arch.) but facing similar issues.

Any suggestions and pointers regarding the same will be helpful.

[pratiktonage]

Hi @kotwanikunal, Any thoughts on this?

[dblock]

@pratiktonage @Pratikt2312 look at the logs for the actual test failures and copy-paste the failure/stack here?

Also check whether these are flaky by re-running the individual tests (in which case open an issue for every flake test if it doesn't exist).

[kotwanikunal]

Adding to what @dblock said - can you try pasting the logs as well as the instance of REPRODUCE with within the logs. That will help with a retryable seed which will help figure out the issue further.

[pratiktonage]

Hi @kotwanikunal @dblock ,
Thank you for your quick response, I have individually executed the tests and attached the logs for the non-flaky tests failures as suggested by you.
x86_error_logs.txt

[dblock]

Looks like at least the one you mentioned above isn't a flake, but produces a substantially different number/expectation (expected:<2.147483647E9> but was:<2.3402974945E10>). Given that you're trying to make OpenSearch work on another platform, you'll have to dig into why this is the case. Either the expectation is wrong, or it's a bug.