[Feature Request] Add an index setting to ignore field mapping for documents with fields beyond the index.mapping.depth.limit

[james-hercock-vgw]

Is your feature request related to a problem? Please describe

Currently, the index setting "index.mapping.depth.limit" allows us to specify a maximum depth limit for mapping.

However this limit only causes OpenSearch to reject documents beyond the limit, with the error "The depth of the field has exceeded the allowed limit of [X]. This limit can be set by changing the [index.mapping.depth.limit] index level setting."

While helpful at keeping the OpenSearch cluster performance in check, this limit is difficult to work with if there is data beyond the depth limit that needs to be ingested.

Describe the solution you'd like

Add a setting such as "index.mapping.depth.ignore_fields_beyond_limit: boolean" that allows us to ignore fields beyond the depth limit while still ingesting the document into the index.

Related component

Indexing

Describe alternatives you've considered

No response

Additional context

Originally brought up in the forum here

[peternied]

[Triage - attendees 1 2 3 4 5 6]
@james-hercock-vgw Thanks for creating this issue; however, it isn't being accepted due to invalid client requests should be modified to make them valid, relaxing input parsing requirements creates documents with unknown values.

[gaobinlong]

As @james-hercock-vgw said, he wants to ignore the fields beyond the depth limit while still ingesting the document into the index, I think the behavior just like set dynamic: false to the mapping, new detected fields will not be added to the mapping, and will not be indexed, not searchable but still be stored in _source, this behavior can only be controlled in OpenSearch server side, the client side cannot achieve that, so I think this feature request makes sense.

[peternied]

@gaobinlong I feel like we've had this discussion going both directions - stricter input validation vs more flexible input validation - I don't know that we've got guidance as a project on what path to talk, could you write up an RFC, or maybe come up with some criteria you have in mind for how we shift in one direction vs another?

For example; I think I'd be much more comfortable with relaxing input validation if there was an explicit query parameter &ignoreFieldsPastMaxDepth=true vs an index setting. I'd be curious what you / the broader set of maintainers think and if we can codify our stance.

[gaobinlong]

@gaobinlong I feel like we've had this discussion going both directions - stricter input validation vs more flexible input validation - I don't know that we've got guidance as a project on what path to talk, could you write up an RFC, or maybe come up with some criteria you have in mind for how we shift in one direction vs another?

For example; I think I'd be much more comfortable with relaxing input validation if there was an explicit query parameter &ignoreFieldsPastMaxDepth=true vs an index setting. I'd be curious what you / the broader set of maintainers think and if we can codify our stance.

Sure, I'll write an RFC containing more details about that.