Cannot set custom retries handler for Elasticsearch RestClient

[oridool]

When using httpclient directly, I have the option to set my own retries handler. Example:
https://www.programcreek.com/java-api-examples/?api=org.apache.http.client.HttpRequestRetryHandler

However, when configuring the RestClient with spring-data-elasticseaarch, I only have access to HttpAsyncClientBuilder through TerminalClientConfigurationBuilder#withHttpClientConfigurer, which doesn't allow me to do so (it is probably required that Elasticsearch client would upgrade internal httpclient to 5.x, while now it is using 4.1.4).

I'm also familiar with the ability to override ElasticsearchRestTemplate's execute method :

    private ElasticsearchRestTemplate createElasticsearchRestTemplateForElasticsearchOperations(ElasticsearchConverter elasticsearchConverter, RestHighLevelClient elasticsearchClient) {
        return new ElasticsearchRestTemplate(elasticsearchClient, elasticsearchConverter) {
            @Override
            public <T> T execute(ClientCallback<T> callback) {
                int retryCount = 0;
                T t = null;
                // do some logic
                return t;
            }
        };

But this doesn't help in all cases, because most APIs are not using execute.

Is there any way to configure my own retries strategy/handler?
If not, can it be added as an enhancement ?

Thank you.

[sothawo]

Spring Data Elasticsearch can only offer the configuration possibilities that Elasticsearch's RestClientBuilder provides.

because most APIs are not using execute.

which API functions of ElasticsearchRestTemplate do not use the execute method and use the client directly?

[oridool]

@sothawo ,
When I'm calling a RestHighLevelClient api such as:
client.indices().getAlias() / client.deleteByQuery() / client.search()

I don't see it is going through the ElasticsearchRestTemplate#execute method (I also put a breakpoint to make sure).
It does go through the CloseableHttpAsyncClient#execute method directly.

But maybe I'm doing something wrong. Here is my bean configuration:

@Configuration
public class ElasticSearchRestClientConfig extends AbstractElasticsearchConfiguration {

    // ...some other configuration here...

    @Override
    public ElasticsearchOperations elasticsearchOperations(ElasticsearchConverter elasticsearchConverter,
                                                           RestHighLevelClient elasticsearchClient) {
        return createElasticsearchRestTemplateForElasticsearchOperations(elasticsearchConverter, elasticsearchClient);
    }

    private ElasticsearchRestTemplate createElasticsearchRestTemplateForElasticsearchOperations(ElasticsearchConverter elasticsearchConverter, RestHighLevelClient elasticsearchClient) {
        return new ElasticsearchRestTemplate(elasticsearchClient, elasticsearchConverter) {
            @Override
            public <T> T execute(ClientCallback<T> callback) {
                int retryCount = 0;
                T t = null;
                // ...some logic here...
                return t;
            }
        };
    }

}

[sothawo]

You are calling methods on the RestHighLevelClient variable client. Of course these are not passing through the execute method of the ElasticsearchRestTemplate. ElasticsearchRestTemplateis using the RestHighLevelClient and not the other way round.

With Spring Data Elasticsearch you use the methods of the ElasticsearchOperations interface or the IndexOperations interface like this - you should never need to call methods on the RestHighLevelClient:

  ElasticsearchOperations operations; // injected into your class

  Query query = ...;  // create a query
  operations.search(query, Entity.class)

  operations.indexOps(Entity.class).getAliases(...)

As for the configuration with some error handling: The following code is taken from the org.springframework.data.elasticsearch.junit.jupiter.ElasticsearchRestTemplateConfiguration class:

@Override
	public ElasticsearchOperations elasticsearchOperations(ElasticsearchConverter elasticsearchConverter,
			RestHighLevelClient elasticsearchClient) {

		ElasticsearchRestTemplate template = new ElasticsearchRestTemplate(elasticsearchClient, elasticsearchConverter) {
			@Override
			public <T> T execute(ClientCallback<T> callback) {
				try {
					return super.execute(callback);
				} catch (DataAccessResourceFailureException e) {
					try {
						Thread.sleep(1_000);
					} catch (InterruptedException ignored) {}
					return super.execute(callback);
				}
			}
		};
		template.setRefreshPolicy(refreshPolicy());

		return template;
	}

Here we have a custom ElasticsearchRestTemplate that, if a DataAccessResourceFailureException occurs, sleeps for a second and then retries. You can build your own retry solution in a simiilar way.

[oridool]

@sothawo ,
Thanks for the clarification. Indeed, for many use cases we can use the ElasticsearchOperations to execute Elasticsearch APIs.
I'm also overriding the execute method as demonstrated above and it works.

However, there are some use cases where we need to use a lower level API or have more control (parameters) over the APIs provided by ElasticsearchOperations and therefore we use the rest client directly.
For example, if we want to call _cat/indices . Calling the regular getIndices API through ElasticsearchOperations or the restClient directly is not possible performance-wise when having thousands of indices and each has multiple aliases because the returned response size is too big (hundreds of Megabytes).

Hence, I still think that if there is an option to enhance spring-data-elasticsearch and to be able to configure the low level client with a HttpRequestRetriesHandler, it would be very beneficial in such cases.
WDYT ?

FYI, I also created another issue for upgrading the internal AsyncHttpClient to 5.x :
elastic/elasticsearch#66136

[sothawo]

As I wrote before, Spring Data Elasticsearch can only configure what Elasticsearch's RestClientBuilder allows to configure. And that does not include a retry handler.