[BUG] Setting `compatibility.override_main_response_version` not supported in OpenSearch 3.0

[atalakey4work]

Describe the bug

During the upgrade, OpenSearch fails to start because of compatibility.override_main_response_version: true config with the below error even though it was not mentioned in the breaking changes:

org.opensearch.bootstrap.StartupException: SettingsException[unknown setting [compatibility.override_main_response_version] please check that any required plugins are installed, or check the breaking changes documentation for removed settings]
	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:172) ~[opensearch-3.0.0.jar:3.0.0]
	at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:159) ~[opensearch-3.0.0.jar:3.0.0]
	at org.opensearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:110) ~[opensearch-3.0.0.jar:3.0.0]
	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-3.0.0.jar:3.0.0]
	at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-3.0.0.jar:3.0.0]
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:125) ~[opensearch-3.0.0.jar:3.0.0]
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:91) ~[opensearch-3.0.0.jar:3.0.0]
Caused by: org.opensearch.common.settings.SettingsException: unknown setting [compatibility.override_main_response_version] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
	at org.opensearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:606) ~[opensearch-3.0.0.jar:3.0.0]
	at org.opensearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:547) ~[opensearch-3.0.0.jar:3.0.0]
	at org.opensearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:517) ~[opensearch-3.0.0.jar:3.0.0]
	at org.opensearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:487) ~[opensearch-3.0.0.jar:3.0.0]
	at org.opensearch.common.settings.SettingsModule.<init>(SettingsModule.java:178) ~[opensearch-3.0.0.jar:3.0.0]
	at org.opensearch.node.Node.<init>(Node.java:647) ~[opensearch-3.0.0.jar:3.0.0]
	at org.opensearch.node.Node.<init>(Node.java:460) ~[opensearch-3.0.0.jar:3.0.0]
	at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:243) ~[opensearch-3.0.0.jar:3.0.0]
	at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:243) ~[opensearch-3.0.0.jar:3.0.0]
	at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:405) ~[opensearch-3.0.0.jar:3.0.0]
	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:168) ~[opensearch-3.0.0.jar:3.0.0]
	... 6 more
uncaught exception in thread [main]
SettingsException[unknown setting [compatibility.override_main_response_version] please check that any required plugins are installed, or check the breaking changes documentation for removed settings]
	at org.opensearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:606)
	at org.opensearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:547)
	at org.opensearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:517)
	at org.opensearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:487)
	at org.opensearch.common.settings.SettingsModule.<init>(SettingsModule.java:178)
	at org.opensearch.node.Node.<init>(Node.java:647)
	at org.opensearch.node.Node.<init>(Node.java:460)
	at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:243)
	at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:243)
	at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:405)
	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:168)
	at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:159)
	at org.opensearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:110)
	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
	at org.opensearch.cli.Command.main(Command.java:101)
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:125)
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:91)

Related component

Other

To Reproduce

1. Add compatibility.override_main_response_version: true to opensearch.yml
2. Upgrade to v3.0.0

Expected behavior

OpenSearch should start up without any issues. If this option was depreciated, kindly let me know if there is another way to implement this workaround:

https://docs.opensearch.org/docs/latest/tools/#agents-and-ingestion-tools

Additional Details

Host/Environment (please complete the following information):

• OS: Kubernets

Additional context
I am using the official helm chart: https://github.com/opensearch-project/helm-charts/tree/main/charts/opensearch

[andrross]

It appears the history here is that this parameter was deprecated in 1.x, removed in 2.0, but then added back in 2.0.1 patch release. From #3530:

The setting was originally added so users of legacy clients would not have to upgrade their client to begin using OpenSearch 1.x.
With 2.0 the REST API is further fragmented from legacy clients such that users are #3023 (comment) to the latest OpenSearch client.
To reduce this friction OpenSearch is working on a #3035 mechanism to support API compatibility with at least the legacy 7.x line; but this will not be ready until 2.2 at the earliest.

The immediate problem is that there is no OpenSearch version of Beats and Logstash and the upstream maintainers of these projects have no plans to support OpenSearch.
This has already lead to #3484 in beats which is alleviated by #3505 and temporarily #3524.
It is also leading to opensearch-project/security#1961 as these processors have explicit version checks for compatibility only with Elasticsearch.

To alleviate this pain, OpenSearch adds back the opensearch.http.override_main_response_version setting so users of Beats and logstash can continue to function by spoofing the OpenSearch cluster version to 7.10.2.

We missed listing this as a breaking change because the removal actually happened pre-2.0 and adding back happened via a patch. I think the bigger question is which of these ingestion tools are supported in OpenSearch 3.0. @atalakey4work which ingest tools are you using? Are you using the Logstash OpenSearch output plugin?

[atalakey4work]

@andrross we are using Filebeat OSS.

I recommend removing it from 3.0 docs given that it is removed.

[andrross]

Thanks @atalakey4work, we'll fix the documentation. Are you able to get your setup working using the OpenSearch output plugin?

[krisfreedain]

Catch All Triage - 1 2

[andrross]

Closing this issue as we don't support this parameter in OpenSearch 3.0. The documentation has been updated. See #18396 for more discussion on this topic.

[mattieserver]

If you go https://docs.opensearch.org/latest/tools/#agents-and-ingestion-tools it still show the option and how to configure it.