Use single godot-cpp branch for Godot 4.1+ and version it independently from Godot

[dsnopek]

Describe the project you are working on

GDExtension and godot-cpp.

Describe the problem or limitation you are having in your project

Right now, we have 3 active branches of godot-cpp:

• 4.1
• 4.2
• master (for 4.3, but soon it'll be for 4.4, and we'll add a new 4.3 branch)

We try to cherry-pick as much as we can from godot-cpp master to all earlier branches as far back as 4.1 (we don't cherry-pick to 4.0 and make as few changes there as we can, since it's incompatible with Godot 4.1+).

We intend to keep doing this, because GDExtensions targeting a particular Godot version (starting with Godot 4.1) will work for that version and any later version. This creates an incentive for GDExtension developers to target the earliest possible version, in order to give the widest compatibility.

However, this cherry-picking creates a couple problems:

1. It's time-consuming to cherry-pick. At the moment it's not too bad - the last wave of cherry-picks from master to 4.1 and 4.2 took me about 90 minutes. However, as more versions are released, this could grow if we continue to cherry-pick all the way back to 4.1. However, we could work around this by only supporting the 2 previous minor versions, or something like that, in order to keep this constant.
2. Cherry-picking can lead to unpredictable changes for developers. Sometimes changes (including important bug fixes) will change behavior in subtle ways for developers. Since developers may be using, for example, the 4.1 branch over the long-term for compatibility reasons, we want to bring these bug fixes to them. However, since the changes to the godot-cpp 4.1 branch don't get a version number (except when a Godot release happens), we don't have a good way to provide a CHANGELOG. Even if we attempted to list the changes with the Godot synchronized release, it's a little awkward in that we're making the same godot-cpp changes to 4.1, 4.2, etc at the same time, but we don't have any way to connect them to each other, since godot-cpp is versioned as Godot is.

The first problem isn't that big of a deal, however, the second one is increasingly creating headaches for developers.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

I propose that we support all Godot 4.1+ versions using a single branch of godot-cpp, and then version godot-cpp independently of Godot itself.

We could still include the latest extension_api.json files for each minor version of Godot, but developers would indicate to the buildsystem which Godot version they wished to be compatible with (rather than selecting a particular branch of godot-cpp), which would select which one to use. (And they can still point to a custom extension_api.json, of course.)

We could always keep the latest gdextension_interface.h in the repo, but we'd use #ifdefs (based on the version pulled from the extension_api.json) to use the correct functions for the correct Godot compatibility.

And this would allow us to make a CHANGELOG for each released version of godot-cpp, which lists all the changes and how they affect developers, regardless of what Godot version they are targeting for their extension.

I'm not sure what the version number for this "universal" godot-cpp should start on, perhaps 4.4.0? That way versions 4.3 and below still refer to the old way, and we reserve version 5.0 to potentially mean something different for Godot 5.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

Described in the previous section :-)

If this enhancement will not be used often, can it be worked around with a few lines of script?

n/a

Is there a reason why this should be core and not an add-on in the asset library?

n/a

[Naros]

Regarding cherry-picking and introducing subtle behavior changes, I believe that can be solved by the developer setting their git submodule reference to a specific sha or tag rather than a generalized branch name. This would allow the Godot team to freely cherry-pick as needed to former branches while allowing developer builds to remain unaffected by the cherry-picks.

As for the CHANGELOG, I think every project does this slightly differently depending on the workflow that works best for them. What I can describe is how I do this professionally in the projects I am a committer for.

1. The main / master branch has a CHANGELOG that maintains all changes applied to that branch during the development cycle (alpha, beta, cr, and the first x.y.0.stable) releases.
2. There is a CHANGELOG on each of the minor branches (4.1, 4.2, 4.3) that mirrors the contents of main / master up to the point where the x.y branch is created when x.y.0.stable is officially cut. From this point forward, this CHANGELOG will be amended with any cherry-picks to this x.y branch.

The layout of this CHANGELOG is what I think is really the most critical, with a layout like the following:

## Version
**DATE** - **Link to GitHub/Jira list of issues tagged for this release**

### New features

* Feature A
* Feature B

### Breaking Changes

* List of each breaking change and what the impact is

### Fixes

* List of fixes

Now, could this CHANGELOG be represented similarly, given the tools that the Godot development team uses between GitHub and its project management? Absolutely.

I think what's most important is not where this CHANGELOG is maintained but rather the distinction between new features and regression fixes and any change that could introduce a breaking change that needs consideration by the developer.

For example, using the Bitfield to unsigned int discussion from the last GDExtension meeting, if there was a note in the CHANGELOG that outlined that PR 1234 introduced a compatibility change that GDExtension authors are responsible for treating bitfields as uint64_t and that negative values shouldn't be used, that gives a solid indication of what the impact of that change would be. As it stands now, it requires not only a review of the commit history but, in some cases, going to the PR to digest the discussion and change to understand the potential impact.

If it's easier, from the Godot team's point of view, to version godot-cpp separately and maintain things in a singular branch, I don't see any problem as a consumer of godot-cpp in theory, but we'd need to see what that workflow really would look like I think.

[dsnopek]

Thanks for the feedback!

2. There is a CHANGELOG on each of the minor branches (4.1, 4.2, 4.3) that mirrors the contents of main / master up to the point where the x.y branch is created when x.y.0.stable is officially cut.

The problem with maintaining a CHANGELOG in the current branches without changing how godot-cpp is versioned, is that we aren't in control of when a new version is released (because they are synchronized with Godot releases), and in the case of Godot 4.1, a new version may never be released.

I suppose the CHANGELOG is still valuable, in that you can do a git diff between the old hash and the new hash, and see what was added to the CHANGELOG.

But it doesn't allow developers to have a nice symbolic version number to refer to, and it doesn't allow maintainers to potentially distinguish between major/minor/patch releases of godot-cpp (and have, for example, no breaking changes in patch releases), like we could do if godot-cpp had its own independent version number.

[Naros]

The problem with maintaining a CHANGELOG in the current branches without changing how godot-cpp is versioned, is that we aren't in control of when a new version is released (because they are synchronized with Godot releases), and in the case of Godot 4.1, a new version may never be released.

This makes sense for regressions and fixes that are only scoped to godot-cpp, but I think that's primarily it.

Perhaps it's simply a product of what our plug-in does and how seamless we intend to be with the Godot editor, but we are forced to upgrade with the editor to take advantage of newly exposed APIs that we need, and the backport of those APIs, if applied to older branches at all, is even significantly slower than the development cycle.

Unfortunately whether godot-cpp uses a separate version or not, that won't solve that specific problem :(
I don't think there is a silver bullet for this particular point, but one can dream ☁️

[Bromeon]

We could still include the latest extension_api.json files for each minor version of Godot, but developers would indicate to the buildsystem which Godot version they wished to be compatible with (rather than selecting a particular branch of godot-cpp), which would select which one to use. (And they can still point to a custom extension_api.json, of course.)

This is precisely how we do it in godot-rust: people can set an "API level" (minimum compatible version), or choose a custom Godot version. This is done with a build flag (called "feature" in Cargo) when specifying dependencies. We also have a single unified codebase, with conditional compilation depending on the version. This does sometimes add quite a bit of complexity in the code.

I wrote more about it here: https://godot-rust.github.io/book/toolchain/godot-version.html

---

One thing I'd like to hear your opinion about: we currently support patch-level versions because extension_api.json can change between 4.2 and 4.2.1, for example, and the new features wouldn't be available otherwise. It adds quite a bit of overhead in our case though, so I'm wondering what you think of this practice? Or do you think limiting to minor versions is enough?

[allenwp]

I'm not sure what the version number for this "universal" godot-cpp should start on, perhaps 4.4.0? That way versions 4.3 and below still refer to the old way, and we reserve version 5.0 to potentially mean something different for Godot 5.

I expect that this versioning scheme will only make sense to new users if the history of the project is known, which is not ideal.

Another option is to use dates like 2024-07.0, 2024-07.1, 2024-08.0, etc. This would make it very clear that the version number is independent of Godot version and actually encapsulates multiple Godot versions. I don’t love this idea that I’m proposing, but maybe it’s good enough or prompts another better idea…

[Edit: actually, this idea of using a date for versioning is growing on me and I think I like the idea.]

[dsnopek]

@Bromeon:

This is precisely how we do it in godot-rust: people can set an "API level" (minimum compatible version), or choose a custom Godot version.

I think this may have been in my mind precisely because I saw that godot-rust does this when debugging a recent issue that had a Rust MRP. :-)

One thing I'd like to hear your opinion about: we currently support patch-level versions because extension_api.json can change between 4.2 and 4.2.1, for example, and the new features wouldn't be available otherwise. It adds quite a bit of overhead in our case though, so I'm wondering what you think of this practice? Or do you think limiting to minor versions is enough?

I personally think limiting to minor versions is enough. In 99% of cases, updating to a new patch version is completely safe, and so pushing folks to the latest patch version should be OK. And in the few cases where developers really want to support a particular older patch version, they can provide their own extension_api.json. That's my opinion, at least :-)

[dsnopek]

@allenwp:

I expect that this versioning scheme will only make sense to new users if the history of the project is known, which is not ideal.

I actually think it'll be old users that have the biggest adjustment, since we've been saying for so long that the godot-cpp version is synchronized with Godot. But, if you've never used Godot or godot-cpp before, I don't know that you'd assume the versions were synchronized.

For new users, the latest godot-cpp may be, for example, 4.4.2, and so when you decide you want to use, that's the version you grab. And as a new version comes out, if it's 4.4.3 vs 4.5.0, I think users will make pretty good assumptions about what that means.

If any thing, I think our current synchronized version numbers is the harder for new folks, because they have to learn early on not to just download the latest version, and instead use the right branch for their version of Godot.

Another option is to use dates like 2024-07.0, 2024-07.1, 2024-08.0, etc.

Hm, I'm not crazy about the idea of having all release versions in the same "stream".

While I'm proposing a single master branch to support all Godot versions, I'd really like to have the option to use minor vs patch versions to help signal to developers when we introduce behavior changes.

The main problem I'm hoping to solve is communicating what has changed to developers when they update godot-cpp. And, I guess, we could use a single stream of versions, like dates, and then just rely on the change logs. But releasing 4.4.3 vs 4.5.0 could also indicate to developers that they really need to pay attention to the change log in particular releases.

[Bromeon]

I personally think limiting to minor versions is enough. In 99% of cases, updating to a new patch version is completely safe, and so pushing folks to the latest patch version should be OK. And in the few cases where developers really want to support a particular older patch version, they can provide their own extension_api.json. That's my opinion, at least :-)

(emphasis mine)

But pushing to the lastest patch version means we reduce the compatibility range, and force people to upgrade Godot version even though the previous one is still working. If there's then a regression in 4.3.2, they cannot go back to 4.3.1 or 4.3. This can create quite some chore during an ongoing project...

My thought here was rather to stay with 4.3, which means users can use Godot 4.3, 4.3.1 or 4.3.2. However, they cannot use features only introduced in 4.3.2 (from C++/Rust at least). I'm not sure how often important extension_api.json features are added in patch versions 🤔

[vnen]

I do think it's a good idea overall though I'm not sure about how the versioning scheme should be. Hard to think of something that isn't confusing.

The only question I have (as I asked in the last meeting) is whether the old versions of godot-cpp will still be maintained. If we introduce a breaking change, some people might not be interested in updating since that may lead to a lot of work and in a established project you usually avoid that, but they could still benefit from bug fixes.

[dsnopek]

The only question I have (as I asked in the last meeting) is whether the old versions of godot-cpp will still be maintained. If we introduce a breaking change, some people might not be interested in updating since that may lead to a lot of work and in a established project you usually avoid that, but they could still benefit from bug fixes.

This is a great question!

We'll need to decide what exactly a "breaking change" is: Do we want to use a super strict definition (and hence generate more major/minor versions and branches), or a looser one (which means fewer branches, but more potential annoyance for developers)? And connected with that: How strictly do we want to follow SemVer?

I think if we take a looser approach on both (similar to how Godot does versioning) we wouldn't have very many branches - only for the biggest compatibility breaks. And in that case, we could promise to maintain the current stable minor version and the previous one? That would lead to a similar cherry-picking burden to what we have currently.

I could certainly see an argument for taking a stricter approach to versioning, given that godot-cpp is a library, unlike Godot (although Godot does have some similarity to a library as well).

However, I think my inclination would be to go for the looser approach. I'd be very curious to hear what others think?

[akien-mga]

I fully support this proposal, I think the management of branches following Godot versions on godot-cpp is becoming tricky, and hard to navigate as a user.

For example for godot-git-plugin we recently had an issue that was solved by updating the godot-cpp version to the 4.2 branch, to get a bugfix which was cherry-picked there but not to 4.1 (because we decided to drop support for 4.1): godotengine/godot-git-plugin#246

It would be much clearer if this was an update to a godot-cpp 1.4.1 or whatnot that fixes a known and documented issues in versions <= 1.4.0, regardless of which Godot API is targeted as a baseline.

---

Regarding versioning, I'm also in favor of a "loose SemVer" like Godot uses, where the major version implies an "epoch" change which breaks compatibility significantly, to the point of requiring adapting the code of projects using godot-cpp (usually in non backwards-compatible ways).
Minor version bumps would be done for lesser compat breaking changes, dropping support for very old Godot versions (if needed), or to signify significant new features (e.g. the addition of the documentation generation could have been a minor version bump).
And patch version bumps for maintenance/bug fixes.

That's just my suggestion though, I'm happy with the maintainers deciding to follow SemVer to the letter, or any other system. The context of godot-cpp is slightly different to Godot's so the reasons that made us "loosen" SemVer may not apply. On the other hand it's a versioning scheme that's familiar to Godot users, so to me it makes sense if they can reason similarly about godot-cpp versions.

As for what version to start with, I wouldn't necessarily make it a continuation of the current godot-x.y.z-stable version. We could start at 1.0 and go from there following the versioning scheme we'll adopt. Or for maximum historical value, we could retroactively tag past milestones in the godot-cpp development following that scheme, starting with 1.0 for the first release which we're happy to consider stable (e.g. around Godot 4.1 time after we broke compat - or that could be 2.0 if we decide to start 1.0 with the 4.0 release).