[BUG][Flaky Test] testPredictWithAutoDeployAndTTL_RemoteModel method shows flaky behavior

[dhrubo-os]

What is the bug?

Recently in our workflow testPredictWithAutoDeployAndTTL_RemoteModel method fails often which shows flaky behavior

RestMLRemoteInferenceIT > testPredictWithAutoDeployAndTTL_RemoteModel FAILED
    org.opensearch.client.ResponseException: method [POST], host [http://[::1]:40001], URI [/_plugins/_ml/models/y004_ZQBPbtF1ZLi0y-S/_predict], status line [HTTP/1.1 429 Too Many Requests]
    {"error":{"root_cause":[{"type":"circuit_breaking_exception","reason":"Memory Circuit Breaker is open, please check your resources!","bytes_wanted":0,"bytes_limit":0,"durability":"TRANSIENT"}],"type":"circuit_breaking_exception","reason":"Memory Circuit Breaker is open, please check your resources!","bytes_wanted":0,"bytes_limit":0,"durability":"TRANSIENT"},"status":429}
        at __randomizedtesting.SeedInfo.seed([CDC1E41B1C85E40B:785DD600402448C1]:0)
        at app//org.opensearch.client.RestClient.convertResponse(RestClient.java:479)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:371)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:346)
        at app//org.opensearch.ml.utils.TestHelper.makeRequest(TestHelper.java:186)
        at app//org.opensearch.ml.utils.TestHelper.makeRequest(TestHelper.java:159)
        at app//org.opensearch.ml.utils.TestHelper.makeRequest(TestHelper.java:148)
        at app//org.opensearch.ml.rest.RestMLRemoteInferenceIT.predictRemoteModel(RestMLRemoteInferenceIT.java:1109)
        at app//org.opensearch.ml.rest.RestMLRemoteInferenceIT.testPredictWithAutoDeployAndTTL_RemoteModel(RestMLRemoteInferenceIT.java:232)

2> REPRODUCE WITH: ./gradlew ':opensearch-ml-plugin:integTest' --tests "org.opensearch.ml.rest.RestMLRemoteInferenceIT.testPredictWithAutoDeployAndTTL_RemoteModel" -Dtests.seed=CDC1E41B1C85E40B -Dtests.security.manager=false -Dtests.locale=ga-IE -Dtests.timezone=Australia/Queensland -Druntime.java=11
  2> org.opensearch.client.ResponseException: method [POST], host [http://[::1]:40001], URI [/_plugins/_ml/models/y004_ZQBPbtF1ZLi0y-S/_predict], status line [HTTP/1.1 429 Too Many Requests]
    {"error":{"root_cause":[{"type":"circuit_breaking_exception","reason":"Memory Circuit Breaker is open, please check your resources!","bytes_wanted":0,"bytes_limit":0,"durability":"TRANSIENT"}],"type":"circuit_breaking_exception","reason":"Memory Circuit Breaker is open, please check your resources!","bytes_wanted":0,"bytes_limit":0,"durability":"TRANSIENT"},"status":429}
        at __randomizedtesting.SeedInfo.seed([CDC1E41B1C85E40B:785DD600402448C1]:0)
        at app//org.opensearch.client.RestClient.convertResponse(RestClient.java:479)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:371)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:346)
        at app//org.opensearch.ml.utils.TestHelper.makeRequest(TestHelper.java:186)
        at app//org.opensearch.ml.utils.TestHelper.makeRequest(TestHelper.java:159)
        at app//org.opensearch.ml.utils.TestHelper.makeRequest(TestHelper.java:148)
        at app//org.opensearch.ml.rest.RestMLRemoteInferenceIT.predictRemoteModel(RestMLRemoteInferenceIT.java:1109)
        at app//org.opensearch.ml.rest.RestMLRemoteInferenceIT.testPredictWithAutoDeployAndTTL_RemoteModel(RestMLRemoteInferenceIT.java:232)
  2> NOTE: leaving temporary files on disk at: /__w/ml-commons/ml-commons/plugin/build/testrun/integTest/temp/org.opensearch.ml.rest.RestMLRemoteInferenceIT_CDC1E41B1C85E40B-001

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

What is the expected behavior?
A clear and concise description of what you expected to happen.

What is your host/environment?

• OS: [e.g. iOS]
• Version [e.g. 22]
• Plugins

Do you have any screenshots?
If applicable, add screenshots to help explain your problem.

Do you have any additional context?
Add any other context about the problem.

[pyek-bot]

Potential fix

• increasing JVM size
• set circuit breaker setting to 100% for IT

[andrross]

Catch All Triage - 1 2 3

[jngz-es]

@Zhangxunmt , I see your pr merged, can we close this issue?