[BUG] 400 Bad Request: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"

[HugoKuo]

The filebeat shows this error after upgrading the OpenSearch to 2.0.0

2022-05-31T23:38:55.864Z ERROR [elasticsearch] elasticsearch/client.go:224 failed to perform any bulk index operations: 400 Bad Request: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"}],"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"},"status":400}

To Reproduce
Upgrade the OpenSearch from 1.3 to 2.0

• OS: CentOS 7.9
• Version : OpenSearch 2.0.0
• Filebeat 7.10.2-1 and 7.12.1-1

Are these related?

#2979 (comment)
#3131

[Wadlazky]

Same here.
OpenSearch from 1.3.2 to 2.0.0
Filebeat 7.12.1 and 7.17.3

[dreamer-89]

Thanks @HugoKuo and @Wadlazky for sharing this issue.

Looks like the Filebeat uses docType in bulk request metadata. As types are removed from OpenSearch 2.0, the request fails during Bulk request parsing, which is expected.

I suggest upgrading to latest OpenSearch clients for compatibility as legacy (ES) clients are not supported with OpenSearch 2.0+

[dreamer-89]

On closer look, _type field is defined with omitEmpty tag in BulkMeta struct, which means if _type is nil then _type should be absent in converted json bulk request.

@HugoKuo Can you please share one beat event which is been used for ingestion to OpenSearch.

[nknize]

@dreamer-89 is right. The error is tripped from the BulkRequestParser.

The new errorOnType parameter was added to help alleviate these bwc errors but since we're relying on upstream Beats here it looks like this wasn't caught before release. The default for errorOnType was set to true when it was merged to main and backported to 2.0 but we never followed up to set this to false for client bwc because there is no REST versioning mechanism in OpenSearch to dynamically switch based on client request. I think a simple short term fix is to unblock this issue is to default errorOnType to false in a 2.0 patch release and follow up w/ a REST API versioning mechanism in 2.2 (or 2.1 if it can make the release date).

Before releasing this as a 2.0.1 patch we need to check that this is the only incompatibility and there aren't more lingering that haven't yet been reported.