Increase `maxMergeAtOnce` parameter in OpenSearch 3.0

[msfroh]

Is your feature request related to a problem? Please describe

There's a Lucene issue merged post-10.0 (apache/lucene#266) to increase the default maxMergeAtOnce parameter.

It (probably) can't go into Lucene 10 because it would be a surprising breaking change that can only come with a major version bump, so it's currently listed as a change that will go out with Lucene 11 (whenever that happens).

Describe the solution you'd like

Since we haven't released OpenSearch 3.0 yet, we still have the opportunity to pull this into our next major version.

If it's a better default then it shouldn't need to wait until OpenSearch 4.0.

Related component

Indexing:Performance

Describe alternatives you've considered

Wait until it becomes the default in Lucene 11.

Additional context

No response

[andrross]

Catch All Triage - 1, 2, 3, 4

[linuxpi]

Listing down some findings after looking around a bit:

• Lucene initially increased the default maxMergeAtOnce from 10 to 30, and allowed a single merge to pick up more segments. The total combined size of total segments to be merged is still capped by floorSegmentBytes. Related PR LUCENE-10073: Reduce merging overhead of NRT by using a greater mergeFactor on tiny segments. apache/lucene#266
• This change was backported for Lucene 10 minor version release, but default maxMergeAtOnce still remained 10(unchanged) to avoid surprises to users as mentioned in the issue description as well.
• Post this, in an another change Lucene removed the maxMergeAtOnce paramter completely. Now they are just relying on floorSegmentBytes as a cap on total combined segments size to be merged in a single merge. So essentially they are saying any number of segments can be merged at once as long as the combined size is less than floorSegmentBytes Remove maxMergeAtOnce option from TieredMergePolicy. apache/lucene#14165
• There is another parameter segsPerTier which also helps limit the no of segments in a single commit. So essentially Lucene picks up as many segments as available until both of the segsPerTier and floorSegmentBytes limits are exceeded.

Changing the default value of maxMergedAtOnce to 30 for 3.0 might be cutting it too close now as we are nearing the RC cut. Instead we can introduce a cluster level settings to allow users to configure a cluster level value for maxMergeAtOnce which applies to all indices if index level maxMergeAtOnce setting is not set for an index. The new cluster level setting we add for maxMergeAtOnce would still have default value of 10.

This will allow users to increase maxMergeAtOnce at cluster level to leverage better performance during merges for all indices using a single setting, while also being safe to not change the default behavior for users happy with status quo.

What do you think @msfroh @sachinpkale ?

[msfroh]

I would prefer to just expose the Lucene behavior (i.e remove maxMergeAtOnce) unless we can identify a compelling reason not to. The defaults in Lucene are generally quite reasonable for a broad set of use-cases.

In this case, removing the parameter from Lucene 10 would be an API-breaking change, which they don't want to do within a major version. If we can front-load it on OpenSearch 3.0 before release, we can get a head start on the Lucene 11 behavior.

[sachinpkale]

I would prefer to just expose the Lucene behavior (i.e remove maxMergeAtOnce) unless we can identify a compelling reason not to

But this would mean the index create/update APIs in OpenSearch will break as today we allow to set maxMergeAtOnce at index level. This setting is not marked as deprecated yet, not sure if we can remove it in 3.0

[linuxpi]

This setting is not marked as deprecated yet, not sure if we can remove it in 3.0

I think we should deprecate first before completely removing, otherwise it might be a surprise to users!

Also, just found out we are also changing the floorSegmentSize in #17699 , increasing the floorSegmentSize and maxMergeAtOnce together might have even higher impact, although both inline with lucene's plans for Lucene 11

We can increase the maxMergeAtOnce to 30 as default for Opensearch 3.0, but should provide some control to users to change it back to 10 if they see any kind of regression for their workload(cluster setting or any other way which lets them change it at cluster level.)

I believe a similar control would also help for floorSegmentSize!

[jainankitk]

But this would mean the index create/update APIs in OpenSearch will break as today we allow to set maxMergeAtOnce at index level. This setting is not marked as deprecated yet, not sure if we can remove it in 3.0

@sachinpkale - I see that the TieredMergePolicy is annotated as @lucene.experimental . Ideally we should have exposed the maxMergeAtOnce setting as experimental in Opensearch.

@msfroh / @andrross - I am wondering if this applies more generally to any setting / code change in Opensearch that depends on Lucene setting / code?

[andrross]

I am wondering if this applies more generally to any setting / code change in Opensearch that depends on Lucene setting / code?

@jainankitk To my knowledge we don't actually have any mechanism to declare a setting as experimental, other than documentation. This particular setting appears to be undocumented, unless I just can't find it. If this continues to be a problem we could consider adding an experimental property to settings that would log a warning, similar to how the deprecated property works.

For this particular setting, we should probably just mark it as deprecated now and change the default.