[BUG] Breaking changes with opensearch.http.override_main_response_version=true

[dblock]

There are a number of breaking changes in 2.0 RESTful APIs. I suspect (still need to confirm) that breaks opensearch.http.override_main_response_version=true (https://opensearch.org/docs/latest/clients/agents-and-ingestion-tools/index/).

I think we have two options.

a) Remove opensearch.http.override_main_response_version=true.
b) Ensure backwards compatibility.

This is tangentially related to opensearch-project/opensearch-clients#17.

[dblock]

@shwetathareja Could you help find a repro case where this is true (using override_main_response_version works in 1.x and is broken in 2.0)?

[dblock]

@mch2 @saratvemulapalli I believe we were originally planning to do (a) for 2.0, is that accurate? do we no longer want to?

[peternied]

While 2.0.0 introduces breaking changes to the rest interface, the clients abstraction the rest calls. Is that abstraction enough that a 1.x client is still logically comparable via the abstraction layer?

Assuming it is, we add support for OpenSearch 2.X in a new OpenSearch client 1.X release on inspection of the api version, it would be valuable to allow rolling backends forward while front ends only need the minor update.

[anasalkouz]

We have similar case on Inclusive naming and the plan to make sure you can do rolling upgrade without any issues. I would avoid doing the upgrade on server and client on the same time.

[dlvenable]

OpenSearch 2.0 removed the classic mapping templates. Clients may be expecting these are available if OpenSearch is advertising itself as a 7.10.2 cluster.

I pushed for the addition of this property (override_main_response_version) initially to help users migrate existing ingestion tools such as Logstash. Often these tools would use the version to decide how to create templates. Hopefully they were not using mapping types. But, if they are, then OpenSearch 2.0 would fail.

[saratvemulapalli]

@mch2 @saratvemulapalli I believe we were originally planning to do (a) for 2.0, is that accurate? do we no longer want to?

This is how I understand it.
We no longer want to have this change in 2.0.
We originally put in the override because technically Elasticsearch 7.10 is the same as OpenSearch 1.0 (i.e we didnt change any of our Rest API signatures).
For clients which work with 7.10.2 did not work with OpenSearch because of new version checks added and this change helps customers use existing clients and make it work.

Now us being at 2.0, this override is not needed anymore. Because the clients on older major (i.e 1.x) cannot talk to OpenSearch 2.0 as we have many breaking changes in our Rest APIs.
Also note, this change only effects Rest API responses and not transport (i.e communication between nodes).

[nknize]

Because the clients on older major (i.e 1.x) cannot talk to OpenSearch 2.0 as we have many breaking changes in our Rest APIs

Not entirely correct. Since OpenSearch Core 1.0.0 GA is nothing more than refactored, reversioned Legacy 7.10.x OpenSearch 2.0 is wire compatible back to legacy 7.10.0. We need to retain spoofing to ensure compatibility w/ 7.10.0 nodes. OpenSearch 3.0 will remove any/all compatibility w/ legacy versions so spoofing can be removed at that time.

[nknize]

Hopefully they were not using mapping types. But, if they are, then OpenSearch 2.0 would fail.

Correct. Types were deprecated in legacy 7.x (Nov 1, 2018); ergo they were removed since clients had over three years to stop depending on types.

[shdubsh]

We use a couple supporting tools that still depend on the 7.x ES client libraries. We look forward to disabling the 7.10.2 spoofing as soon as possible to remove our reliance on this workaround.

I based our plans on the answer I received at the community meeting on 2021-10-19: "Do we expect the ES version compatibility flag to be removed in 2.0?" Kyle's answer was to expect the version compatibility flag to be re-evaluated prior to 3.0. If removed, it would be removed no earlier than 3.0.

If this is no longer possible and it becomes necessary for us to remain on 1.x until we no longer rely on the workaround, will 1.x continue to get security patches for while?