[AUTOCUT] Gradle Check Failure on push to main

[github-actions]

Gradle check has failed on push of your commit 990ddc3.
Please examine the workflow log https://build.ci.opensearch.org/job/gradle-check/40751/.
Is the failure a flaky test unrelated to your change?

[shiv0408]

Strangely the setting we are putting before the assertBusy is run to modify the clean up task was not applied even after the 500ms interval for assertBusy was completed and the finally when we disable the task only that setting got applied.

[2024-06-11T17:53:34,801][INFO ][o.o.c.s.ClusterSettings  ] [node_t0] updating [cluster.remote_store.state.cleanup_interval] from [5m] to [-1]
[2024-06-11T17:53:34,801][INFO ][o.o.g.r.RemoteClusterStateCleanupManager] [node_t0] updated remote state cleanup interval to -1

OpenSearch/server/src/internalClusterTest/java/org/opensearch/gateway/remote/RemoteClusterStateCleanupManagerIT.java

Lines 112 to 140 in aa329d3

	// set cleanup interval to 100 ms to make the test faster
	ClusterUpdateSettingsResponse response = client().admin()
	.cluster()
	.prepareUpdateSettings()
	.setPersistentSettings(Settings.builder().put(REMOTE_CLUSTER_STATE_CLEANUP_INTERVAL_SETTING.getKey(), "100ms"))
	.get();
	assertTrue(response.isAcknowledged());
	assertBusy(() -> {
	int manifestFiles = repository.blobStore().blobContainer(manifestContainerPath).listBlobsByPrefix("manifest").size();
	logger.info("number of current manifest file: {}", manifestFiles);
	// we can't guarantee that we have same number of manifest as Retained manifest in our repo as there can be other queued task
	// other than replica count change which can upload new manifest files, that's why we check that number of manifests is between
	// Retained manifests and Retained manifests + 2 * Skip cleanup state changes (each cluster state update uploads 2 manifests)
	assertTrue(
	"Current number of manifest files: " + manifestFiles,
	manifestFiles >= RETAINED_MANIFESTS && manifestFiles < RETAINED_MANIFESTS + 2 * SKIP_CLEANUP_STATE_CHANGES
	);
	}, 500, TimeUnit.MILLISECONDS);
	// disable the clean up to avoid race condition during shutdown
	response = client().admin()
	.cluster()
	.prepareUpdateSettings()
	.setPersistentSettings(Settings.builder().put(REMOTE_CLUSTER_STATE_CLEANUP_INTERVAL_SETTING.getKey(), "-1"))
	.get();
	assertTrue(response.isAcknowledged());

Will raise a change to verify that the setting is getting applied before we check if files are deleted.