Opencast
Opencast full rewrite in 2025? #6423
Replies: 2 comments
-
|
Eh yo, sure did get my attention. OK back to normal-speak. Since I cannot participate in today's meeting, here are some random thoughts: My biggest concern is an organizational one. Assume the money is enough to finish a full rewrite, AFAIK this money is only spent (1) by Uni Osnabrück and (2) for elan. This means there is a high level of control from these two parties for the direction of the whole project. I also think the incentives are misaligned for an open-source project. If all orgs are to contribute to this effort and only one is paid to do so, I see some not participating. And yes, I'm saying this as CEO of shio solutions GmbH, so take it as you like. But we have had no progress on this since OC DACH (bottleneck)! Finally, say after the rewrite is done, elan could be in a position with Opencast-NG experts while others have little knowledge about the new codebase. And if orgs are unwilling to move on to Opencast-NG, we end up in a Python 2 vs Python 3 situation. OK, yes, it may be a very pessimistic scenario... But I do see this as a very real concern. So leaving this aside for now and looking at the product itself. I would approach this question in a different way. Instead of answering if we want incremental improvements, a full rewrite or something in between, I would suggest spending not an insignificant amount of time on the drawing board and specifying what we want the system to look like. Starting with a high level and then going down until we can make a call for the next direction. Finally, my gut feeling is that the approach will be to split up Opencast into smaller pieces and modernize each component at a time. If you want, this has already started with the UIs we have. The risk is way smaller since money can dry up, and we only have the current component that is not finished. We can also ship already finished components helping the community to adopt the rewrite. And who knows, maybe at the end, we will have a full rewrite. |
Beta Was this translation helpful? Give feedback.
-
|
We met today with a large part of the core dev community to discuss this topic. I'm trying to summarize this here, please comment if I forgot something or summarizing incorrectly. The general consensus seems to be a clear "don't rewrite". While basically everyone acknowledged that Opencast hast problems, and that many of the things I list as motivation are true (at least to a degree), everyone speaking up in the meeting were against a rewrite. (I These were mostly, as I remember: Lars, Greg, Katrin, Sascha). The main arguments I remember were:
On a meta level, we all recognized that it would be better to have a clear goal (where we want Opencast to end up) in order to discuss the best path there. I personally also think that a mismatch in what any one envisions as the goal results in different views on this topic: as someone thinking OC should be quite different and more minimal, I of course see a rewrite as more favorable than someone who thinks the goal is "Opencast as it is now, with a few things fixed". There were many concrete suggestions how to instead approach "incrementally improving Opencast". Most notably for me: first throwing out stuff we might not need, to make refactoring other things easier. For reference, here is the full protocol we took during the meeting. Note that this was created by multiple people and is very incomplete and not structured. So it doesn't fairly reflect the full discussion. DetailsOC Rewrite (2025-02-18)
Hope/idea: Cons: Things to think about:
Personal takes: Ideas:
Call to action: Personally, while many arguments made sense to me, I wasn't actually convinced by this meeting that the "incremental improvement" route is the better one. I'm also still not sure rewrite is better, to be clear. But it is clear that the Opencast community is against a rewrite, so that's the route we are taking. And it will certainly result in a better Opencast! And I'm glad we had this discussion. Since the consensus seemed very clear to me, I will close this GH discussion thread. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
This was still posted on the mailing list (without replies yet). I'm just copying the text here for migration.
Sup,
I sure know how to get your attention with a subject line 🙂
In a meeting yesterday, the Spring Security update came up briefly. And it got me thinking: is it actually time to rewrite?
I know, rewriting Opencast is a meme. One that has been around for longer than I've been part of this community. But let's actually have a proper conversation about this for once. Where everyone can bring their arguments, experiences and assessments to the table in order for us to make an informed decision at the end.
But wait! Despite what most of you were probably suspecting, I am not bringing this up to replace Java. This is truly not my point! So let's please completely set aside the question of programming language, to narrow the scope of this discussion.
Also note: while I'm leaning toward "yes, rewrite" right now, I don't want to convince you – I merely want to start a discussion and hear from all of you.
With that meta stuff out of the way, let me lay out my view of the situation.
Reasons for
Coming back to Spring Security: the outdated version has been a problem for many years and no one managed to update it yet, not even Greg! The amount of time sunk into maintenance work like this is enormous. The state of the codebase significantly slows down the implementation of new features and other changes. Opencast often requires arcane knowledge to work with and code in: I have lost count of how many times, in a meeting, a part of Opencast was mentioned, only for many experienced devs to chuckle and say "no one understands that, I wouldn't touch it with a ten foot pole".
Random crashes, unexplained behavior and inconsistent data are common. There is so much unneeded complexity, so many code smells, edge cases and bugs. I cannot remember when I last tried an Opencast feature that works flawlessly, or when I last looked at Opencast code that looked reasonably fine to me.
As you know, Opencast recently received a large fund in order to modernize, refactor and generally improve it. We had a couple of meetings about this and agreed on some good improvements already, but the coding work has not been started yet.
I am more and more convinced that the "incremental improvements" route is actually slower than a full rewrite. That's the main reason I'm writing this mail. Yes, rewrites take a ton of time, but the velocity is way faster. Imagining trying to implement the discussed "data model" changes gives me shivers. I don't see how this can be merged in less than half a year, and this is just a first step.
A fresh start means we can learn from our mistakes, get rid of unnecessary complexity, part with historical baggage, and build a more robust and maintainable product in the end. This funding is the perfect opportunity: if not now, it will never happen.
Learning from mistakes and backward-compatibility
Of course, this requires us to actually learn from our mistakes. Just starting a rewrite aimlessly will get us nowhere. We need to really understand the pain points of devs, admins and users of Opencast. And find out what ultimately causes them and how we can improve. We don't want to just rewrite the code, but also improve on Opencast's data model and semantics.
But we cannot throw out everything either. We want current users of Opencast to eventually switch to the rewritten version, so a certain amount of backwards compatibility is required. We probably should keep the ingest API for capture agents, for example. This requires careful consideration, but in the end, there needs to be a reasonable migration path for admins of installations and for developers of third-party integrations!
Reasons against
My biggest worry is: I am still too naive and just wrong about a rewrite being faster than incremental improvements. This is what I'm most interested in hearing your assessment of and opinion on!
Even if the rewrite is faster, it will take a while, during which we have two parallel codebases. That means some feature might need to be developed twice and/or people are discouraged from putting any work into the old codebase since it will be thrown away at some point. We had a similar problem with the admin UI before.
I also want to touch on a topic not commonly talked about: a rewrite means that experienced members of the community have to re-learn several aspects and some of their knowledge will become useless. It's easy to dismiss this as irrational, but Opencast is worthless without its community and disregarding "human problems" like that doesn't get us anywhere. I personally think that a rewrite is possible without alienating or splintering parts of the community, but I understand if others feel differently about this.
Finally, a rewrite is scary: if we run out of money before reaching a usable state, we have essentially nothing. With incremental improvements, we can stop at "any time" and have still gained something. We'd be leaving the safe and familiar shores.
Thank you for taking the time to read this. I am truly curious to hear what you think.
Have a nice Christmas and see you in 2025!
Lukas
Beta Was this translation helpful? Give feedback.
All reactions