[Campaign] Ensure 1.x and 2.x branches (main should be 3.0)

[dblock]

What kind of business use case are you trying to solve? What are your requirements?

Currently plugins follow a branching strategy where they work on main for the next development iteration, effectively working on 2 versions at the same time. For example, at the time of writing plugins are working on 2.0 (main) and 1.3.2 (1.3). In contrast, OpenSearch works on 3 releases at the same time. Currently on 3.0 (main), 2.0 (2.x), and 1.3.2 (1.3).

What is the problem? What is preventing you from meeting the requirements?

• There's no OpenSearch 3.0 with all plugins continuously integrating, causing late integration, and bugs discovered during integration instead continuously.
• Features and fixes (e.g. CVEs) from main branches may need to be selectively backported into multiple 1.x and 2.x releases. Without an intermediate 1.x or 2.x branch there are as many merge conflicts as branches.
• Without a 2.x branch, developers don't have an easy way to express "backport to any next 2.x release" and must be aware of the state of the next 2.x (e.g. does it accept new features?).
• Workflows are different between OpenSearch core and plugins, which is confusing.
• Authors of breaking changes in OpenSearch for the next major release don't have a place to fix downstream plugins, even if they wanted to.
• https://github.com/opensearch-project/.github/blob/main/RELEASING.md#plugin-branching is incorrect.

What are you proposing? What do you suggest we do to solve the problem or improve the existing situation?

Follow OpenSearch core branching, described in #35. Create 1.x and 2.x branches, do not create 2.0 as a branch of main, instead create main -> 2.x -> 2.0. Maintain working CI for 3 releases at any given time.

• Implement OpenSearch core branching strategy alerting-dashboards-plugin#290
• Implement OpenSearch core branching strategy alerting#493
• Implement OpenSearch core branching strategy anomaly-detection-dashboards-plugin#290
• Implement OpenSearch core branching strategy anomaly-detection#614
• Implement OpenSearch core branching strategy asynchronous-search#159
• Implement OpenSearch core branching strategy common-utils#202
• Implement OpenSearch core branching strategy cross-cluster-replication#451
• Implement OpenSearch core branching strategy reporting#396
• Implement OpenSearch core branching strategy dashboards-visualizations#96
• Implement OpenSearch core branching strategy geospatial#94
• Implement OpenSearch core branching strategy index-management-dashboards-plugin#216
• Implement OpenSearch core branching strategy index-management#416
• Implement OpenSearch core branching strategy job-scheduler#205
• Implement OpenSearch core branching strategy k-NN#451
• Implement OpenSearch core branching strategy ml-commons#366
• Implement OpenSearch core branching strategy notifications#481
• Implement OpenSearch core branching strategy observability#869
• Implement OpenSearch core branching strategy performance-analyzer#241
• Implement OpenSearch core branching strategy performance-analyzer-rca#199
• Implement OpenSearch core branching strategy security-dashboards-plugin#1034
• Implement OpenSearch core branching strategy security#1951
• Implement OpenSearch core branching strategy sql#692
• [BUG] Split multiple plugins into their own repositories dashboards-maps#95

Exit Criteria

• main is building 3.0
• 2.x is building next 2.x, which is currently 2.4.0
• 2.3 is building next 2.3, which is 2.3.1
• manifests in https://github.com/opensearch-project/opensearch-build/tree/main/manifests include all 3

[peternied]

@dblock One day ago security choose to delete its 2.x branches because there is no active 3.x feature development, creating a 2.x when the need arises.

The aim of deleting this branch was to was to

• Control when 3.x breaking changes are scheduled to be worked on
• Reduced the amount of backporting management, considerations such questioning did a change go to the correct branches?
• Reduced the number of pull request branch targets to check for in a pull request to one location

While 3.0 will be released in the future and its important, these 'papercuts' impact a smaller project. If we had better mechanisms or potentially a better starting posture we would more quickly adopt the 2.x branch

[dblock]

@peternied Do you not feel that having a working 3.0 build of the distribution is a valuable goal?

[peternied]

@dblock The maintainers of security had a discussion yesterday where this topic was raised. While there are pain points as mentioned, ultimately we are in agreement we should move in this direction. By running through the process and connecting data we can generate more topics for the retro(s) if we have recommendations or the process isn't tenable.

[cliu123]

I like the idea of building 3 different versions in parallel to let OpenSearch breaking changes fail plugins early.
Security plugin tried building OpenSearch 3.0 on main branch. But yesterday some breaking changes in OpenSearch core 3.0 breaks the BWC, so the security plugin main branch CI failed and blocked the release 2.1.0. With building OpenSearch 3.0 on plugin main branch, all PRs in plugins repos need to be merged to main first -> 2.x branch-> 2.1 branch. Breaking changes in OpenSearch 3.0 makes plugin main branch fragile and potentially block all PRs until failures are resolved. So security plugin had to downgrade from 3.0 to 2.1 to unblock the main branch CI and 2.1.0 release. We need a better mechanism to prevent this moving forward.

[navneet1v]

@dblock One day ago security choose to delete its 2.x branches because there is no active 3.x feature development, creating a 2.x when the need arises.

The aim of deleting this branch was to was to

• Control when 3.x breaking changes are scheduled to be worked on
• Reduced the amount of backporting management, considerations such questioning did a change go to the correct branches?
• Reduced the number of pull request branch targets to check for in a pull request to one location

While 3.0 will be released in the future and its important, these 'papercuts' impact a smaller project. If we had better mechanisms or potentially a better starting posture we would more quickly adopt the 2.x branch

For K-NN and geo-spatial we also moved to the approach where we will cut 3.0 branches only when we know that there are breaking changes being introduced. This keeps the back-porting to minimum and requires less maintenance.

@dblock the idea suggested in this proposal aligns with the OpenSearch Core but will that be an overkill for plugins that don't have regular changes for different versions?

Also, at what level we are confident that this is the right strategy going forward?

[dblock]

@dblock the idea suggested in this proposal aligns with the OpenSearch Core but will that be an overkill for plugins that don't have regular changes for different versions?

Let's say there are no changes in the plugin ever at all. How do you propose we build a 3.0, 2.2 and 2.1 distribution that includes the plugin today?

[dblock]

For K-NN and geo-spatial we also moved to the approach where we will cut 3.0 branches only when we know that there are breaking changes being introduced. This keeps the back-porting to minimum and requires less maintenance.

Core introduces breaking changes into main (now 3.0) all the time. What branch of k-nn should it include in the 3.0 daily build?

[navneet1v]

@dblock the idea suggested in this proposal aligns with the OpenSearch Core but will that be an overkill for plugins that don't have regular changes for different versions?

Let's say there are no changes in the plugin ever at all. How do you propose we build a 3.0, 2.2 and 2.1 distribution that includes the plugin today?

The idea was simple, before the time of release cut the branch, as there were no changes required the cutting a branch will be simple.

The problem as our team saw with multiple branches was back-porting only and second it brings the plugins and core 1:1 from a branching standpoint which has its own pros and cons, apart from that I am very much aligned with proposal is being provided.

[navneet1v]

For K-NN and geo-spatial we also moved to the approach where we will cut 3.0 branches only when we know that there are breaking changes being introduced. This keeps the back-porting to minimum and requires less maintenance.

Core introduces breaking changes into main (now 3.0) all the time. What branch of k-nn should it include in the 3.0 daily build?

The main branch of K-NN and if that breaks some how then reaching out to the maintainers for fixing it and fix could be cutting a specific branch. The idea is just to delay when we need to cut a branch, for plugins with not a lot of changes.

Even with branches in K-NN we still need to go ahead and fix the breaking changes. It mainly about back-porting a change from main to 2.x branches with every PR.

[downsrob]

The main pain point which I have experienced with breaking changes is that a breaking change will be merged into 3.0 on core, and then the plugins will run CI later on during some PR and experience the breaking change. It can be a time consuming process to track down the breaking change in core versus other dependencies and then longer to find out how to fix this breaking change in the plugins. Additionally, we have seen breaking changes go in, plugins make changes, then core modifies the breaking change and then plugins need to revert. On backporting, I am not hugely concerned, though it is more burden. For example with the non inclusive language breaking changes. When I raise a PR for 3.0 using anything related to master then I need to modify my PR for backports. Not a big deal for the non inclusive language changes but there are breaking changes which could be more burdensome, and if I am trying to get a fix into a plugin repo on code freeze date then I don't want a small change to take a day of backporting or have CI on the main branch be compromised because of some tangent breaking change which blocks my PR from getting merged and backported.

Something to help with the slow feedback loop to help plugins quickly know what kind of changes to make is if core could have another github action which builds all of the plugins. Then we can open a github issue for failing plugins explaining the breaking change and required changes. Running all tests is flaky and slow so we can skip that for now, but the build alone can catch many issues.