[BUG] Java high-level REST client bulk() is not respecting the bulkRequest.requireAlias(true) method call

[olfuerniss]

Describe the bug

The Java high-level REST client bulk(BulkRequest bulkRequest, RequestOptions options) is not respecting the bulkRequest.requireAlias(true) method call. It automatically creates an index with the name of the alias which should not happen when the require alias flag is set.

Related component

Clients

To Reproduce

Following are 4 examples of how the require alias flag can be set and should be respected.

Fail 1: Auto creates the index testindex-1

private RestHighLevelClient osClient;

@Test  
public void t0010_bulkWithRequireAliasUsingGlobalIndex() {  
    String indexAliasName = "testindex-1";  
    try {  
        BulkRequest bulkRequest = new BulkRequest(indexAliasName);  
        bulkRequest.requireAlias(true);  
        bulkRequest.add(new IndexRequest().id("1").source("{ \"name\": \"Biden\" }", XContentType.JSON));  
        bulkRequest.add(new IndexRequest().id("2").source("{ \"name\": \"Trump\" }", XContentType.JSON));  
  
        BulkResponse bulkResponse = osClient.bulk(bulkRequest, RequestOptions.DEFAULT);  
  
        assertFalse(existsIndex(indexAliasName), "Should not auto-create the index.");  
        assertTrue(bulkResponse.hasFailures(), "Bulk response must have failures.");  
    } catch (Exception ex) {  
        Assert.fail("Must not throw an exception");  
    }  
}

Fail 2: Auto creates the index testindex-2

private RestHighLevelClient osClient;

@Test  
public void t0020_bulkWithRequireAliasUsingIndexOnEachRequest() {  
    String indexAliasName = "testindex-2";  
    try {  
        BulkRequest bulkRequest = new BulkRequest();  
        bulkRequest.requireAlias(true);  
        bulkRequest.add(new IndexRequest().index(indexAliasName).id("1").source("{ \"name\": \"Biden\" }", XContentType.JSON));  
        bulkRequest.add(new IndexRequest().index(indexAliasName).id("2").source("{ \"name\": \"Trump\" }", XContentType.JSON));  
  
        BulkResponse bulkResponse = osClient.bulk(bulkRequest, RequestOptions.DEFAULT);  
  
        assertFalse(existsIndex(indexAliasName), "Should not auto-create the index.");  
        assertTrue(bulkResponse.hasFailures(), "Bulk response must have failures.");  
    } catch (Exception ex) {  
        Assert.fail("Must not throw an exception");  
    }  
}

Fail 3: Auto creates the index testindex-3

private RestHighLevelClient osClient;

@Test  
public void t0030_bulkWithRequireAliasOnEachRequestUsingGlobalIndex() {  
    String indexAliasName = "testindex-3";  
    try {  
        BulkRequest bulkRequest = new BulkRequest(indexAliasName);  
        bulkRequest.add(new IndexRequest().id("1").setRequireAlias(true).source("{ \"name\": \"Biden\" }", XContentType.JSON));  
        bulkRequest.add(new IndexRequest().id("2").setRequireAlias(true).source("{ \"name\": \"Trump\" }", XContentType.JSON));  
  
        BulkResponse bulkResponse = osClient.bulk(bulkRequest, RequestOptions.DEFAULT);  
  
        assertFalse(existsIndex(indexAliasName), "Should not auto-create the index.");  
        assertTrue(bulkResponse.hasFailures(), "Bulk response must have failures.");  
    } catch (Exception ex) {  
        Assert.fail("Must not throw an exception");  
    }  
}

Fail 4: Auto creates the index testindex-4

private RestHighLevelClient osClient;

@Test  
public void t0040_bulkWithRequireAliasOnEachRequest() {  
    String indexAliasName = "testindex-4";  
    try {  
        BulkRequest bulkRequest = new BulkRequest();  
        bulkRequest.add(new IndexRequest().index(indexAliasName).id("1").setRequireAlias(true).source("{ \"name\": \"Biden\" }", XContentType.JSON));  
        bulkRequest.add(new IndexRequest().index(indexAliasName).id("2").setRequireAlias(true).source("{ \"name\": \"Trump\" }", XContentType.JSON));  
  
        BulkResponse bulkResponse = osClient.bulk(bulkRequest, RequestOptions.DEFAULT);  
  
        assertFalse(existsIndex(indexAliasName), "Should not auto-create the index.");  
        assertTrue(bulkResponse.hasFailures(), "Bulk response must have failures.");  
    } catch (Exception ex) {  
        Assert.fail("Must not throw an exception");  
    }  
}

Expected behavior

At the moment the 'require alias' flag does not get transferred from the high-level REST client to OpenSearch and OpenSearch automatically creates the index with the name of the alias. This should not happen. It should fail in case there is no index with the alias.

Logged call from the first example (Fail 1):

[TRACE] 2024-03-28 07:34:52.579 [main] tracer - curl -iX POST 'http://localhost:32787/_bulk?timeout=1m' -d '{"index":{"_index":"testindex-1","_id":"1"}}
{"name":"Biden"}
{"index":{"_index":"testindex-1","_id":"2"}}
{"name":"Trump"}
'
# HTTP/1.1 200 OK
# content-type: application/json; charset=UTF-8
# content-length: 373
#
# {"took":192,"errors":false,"items":[{"index":{"_index":"testindex-1","_id":"1","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1,"status":201}},{"index":{"_index":"testindex-1","_id":"2","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":1,"_primary_term":1,"status":201}}]}

Additional Details

Belongs to OpenSearch 2.12.0.

Btw. require alias works fine for single IndexRequest calls, thanks to this fix: #1528

private RestHighLevelClient osClient;

@Test  
public void t0060_singleIndexRequest() {  
    String indexAliasName = "testindex-5";  
    try {  
        IndexRequest indexRequest = new IndexRequest(indexAliasName)  
                .id("1")  
                .setRequireAlias(true)  
                .source("{ \"name\": \"Biden\" }", XContentType.JSON);  
  
        IndexResponse indexResponse = osClient.index(indexRequest, RequestOptions.DEFAULT);  
        Assert.fail("Must throw an exception in the line above.");  
    } catch (OpenSearchStatusException ex) {  
        assertEquals(RestStatus.NOT_FOUND, ex.status(), "Index must not found (= must not automatically created)");  
    } catch (Exception ex) {  
        Assert.fail("Must not throw this kind of exception");  
    }  
}

or when the bulk request gets called by hand

private RestHighLevelClient osClient;

@Test  
public void t0050_byHandBulkWithRequireAlias() {  
    String indexAliasName = "testindex-6";  
    try {  
        String jsonString = """  
			{ "index" : { "_id" : "1" } }                    
			{ "name": "Biden" }
			{ "index" : { "_id" : "2" } }
			{ "name": "Trump" }
		""";  
  
        Request request = new Request("PUT", "/" + indexAliasName + "/_bulk");  
        request.addParameter("require_alias", "true");  
        request.setJsonEntity(jsonString);  
  
        Response bulkResponse = osClient.getLowLevelClient().performRequest(request);  
  
        assertFalse(existsIndex(indexAliasName), "Should not auto-create the index.");  
		String responseString = EntityUtils.toString(bulkResponse.getEntity());  
		assertTrue(responseString.contains("[require_alias]"));
	} catch (Exception ex) {  
        Assert.fail("Must not throw an exception");  
    }  
}

Example content from 'responseString'

{"took":2,"errors":true,"items":[{"index":{"_index":"testindex-6","_id":"1","status":404,"error":{"type":"index_not_found_exception","reason":"no such index [testindex-6] and [require_alias] request flag is [true] and [testindex-6] is not an alias","index":"testindex-6","index_uuid":"_na_"}}},{"index":{"_index":"testindex-6","_id":"2","status":404,"error":{"type":"index_not_found_exception","reason":"no such index [testindex-6] and [require_alias] request flag is [true] and [testindex-6] is not an alias","index":"testindex-6","index_uuid":"_na_"}}}]}

[dblock]

Want to help? Turn this into a (failing) unit test?

[olfuerniss]

Will not create a pull request for just the test. But To the developer implementing this fix, you can execute the integration test by:

./gradlew \
	:client:rest-high-level:integTest \
	--tests "org.opensearch.client.CrudIT.testBulkWithRequireAlias"

when the following test case method (extend/modify to your liking) is added to org.opensearch.client.CrudIT (could be a good place for it):

    public void testBulkWithRequireAlias() throws IOException {
        {
            String indexAliasName = "testindex-1";

            BulkRequest bulkRequest = new BulkRequest(indexAliasName);
            bulkRequest.requireAlias(true);
            bulkRequest.add(new IndexRequest().id("1").source("{ \"name\": \"Biden\" }", XContentType.JSON));
            bulkRequest.add(new IndexRequest().id("2").source("{ \"name\": \"Trump\" }", XContentType.JSON));

            BulkResponse bulkResponse = execute(bulkRequest, highLevelClient()::bulk, highLevelClient()::bulkAsync, RequestOptions.DEFAULT);

            assertFalse("Should not auto-create the '" + indexAliasName + "' index.", indexExists(indexAliasName));
            assertTrue("Bulk response must have failures.", bulkResponse.hasFailures());
        }
        {
            String indexAliasName = "testindex-2";

            BulkRequest bulkRequest = new BulkRequest();
            bulkRequest.requireAlias(true);
            bulkRequest.add(new IndexRequest().index(indexAliasName).id("1").source("{ \"name\": \"Biden\" }", XContentType.JSON));
            bulkRequest.add(new IndexRequest().index(indexAliasName).id("2").source("{ \"name\": \"Trump\" }", XContentType.JSON));

            BulkResponse bulkResponse = execute(bulkRequest, highLevelClient()::bulk, highLevelClient()::bulkAsync, RequestOptions.DEFAULT);

            assertFalse("Should not auto-create the '" + indexAliasName + "' index.", indexExists(indexAliasName));
            assertTrue("Bulk response must have failures.", bulkResponse.hasFailures());
        }
        {
            String indexAliasName = "testindex-3";

            BulkRequest bulkRequest = new BulkRequest(indexAliasName);
            bulkRequest.add(new IndexRequest().id("1").setRequireAlias(true).source("{ \"name\": \"Biden\" }", XContentType.JSON));
            bulkRequest.add(new IndexRequest().id("2").setRequireAlias(true).source("{ \"name\": \"Trump\" }", XContentType.JSON));

            BulkResponse bulkResponse = execute(bulkRequest, highLevelClient()::bulk, highLevelClient()::bulkAsync, RequestOptions.DEFAULT);

            assertFalse("Should not auto-create the '" + indexAliasName + "' index.", indexExists(indexAliasName));
            assertTrue("Bulk response must have failures.", bulkResponse.hasFailures());
        }
        {
            String indexAliasName = "testindex-4";

            BulkRequest bulkRequest = new BulkRequest();
            bulkRequest.add(new IndexRequest().index(indexAliasName).id("1").setRequireAlias(true).source("{ \"name\": \"Biden\" }", XContentType.JSON));
            bulkRequest.add(new IndexRequest().index(indexAliasName).id("2").setRequireAlias(true).source("{ \"name\": \"Trump\" }", XContentType.JSON));

            BulkResponse bulkResponse = execute(bulkRequest, highLevelClient()::bulk, highLevelClient()::bulkAsync, RequestOptions.DEFAULT);

            assertFalse("Should not auto-create the '" + indexAliasName + "' index.", indexExists(indexAliasName));
            assertTrue("Bulk response must have failures.", bulkResponse.hasFailures());
        }
    }

[peternied]

[Triage - attendees 1 2 3 4 5 6]
@olfuerniss Thanks for creating this bug - with great detail as well. What do you think about creating a pull request to address this issue?

[parv0201]

@peternied I would like to work on this issue. Can I raise pull request for this?

[peternied]

@parv0201 Thank you, I'll assign this issue to you.