[Proposal] Improving the Autoware Repository Structure for New Contributors

[mitsudome-r]

Background

Currently, the main Autoware repository (https://github.com/autowarefoundation/autoware) appears empty, as it does not contain source code but instead relies on external repositories.

This structure has historical and architectural reasons, but for new contributors and industry partners, this leads to significant confusion:

• The repository appears inactive or incomplete at first glance.
• New users cannot identify where the actual code is located.
• Entry points are inconsistent across different repositories.
• Developers expect source code in the primary repo (following conventional OSS practices).
• Onboarding time increases, especially for companies evaluating Autoware.

These issues reduce Autoware’s approachability and hinder ecosystem growth. This proposal aims to:

• Reduce onboarding friction for new users and companies.
• Improve clarity of the repository structure through documentation and design.
• Align user expectations with the architectural layout (multi-repository design).
• Strengthen the perception of Autoware as a mature, well-organized OSS project.

Proposals

Option 1: Merge Autoware Core into the Autoware Meta-Repository

Move all source code from autoware_core into src folder in the autoware repository. Autoware Universe will be treated as extension to the Autoware repository and can be maintained via vcs .repos files.

Pros

• Simplest UX as people could just run colcon build out of the box to build minimal set of Autoware
• Eliminates “empty repo” issue entirely
• Single entry point for contributions and onboarding

Cons

• Major structural change → disrupts forks, CI, contribution workflows
• Requires migration planning & broad TSC consensus
• Reduced modularity

Option 2: Replace VCS with Git Submodule

Replace vcs with git submoulde. The meta-repo becomes a user-friendly bundle.

Pros

• Maintains modularity and separation of responsibilities
• Low-risk way to improve newcomer experience
• git submodule is well known than vcs tools

Cons

• Requires periodic sync or submodule updates
• Slight maintenance overhead vs. keeping the status quo
• CI integration becomes somewhat more complex (as we have different .repos files for different purpose)

Option 3: Add a src/ Directory with a README Explaining Where the Code Is

Leave the meta-repo empty, but create a src/ folder containing only documentation explaining the link to the code
Pros

• low cost
• Immediately solves the “empty repo” problem
• No impact on CI or contributor workflows

Cons

• Users cannot build Autoware directly from the meta-repo
• Less powerful improvements compared to other options
• Still multiple repositories → some complexity remains

Option 4: Keep Current State

Keep using vsc tools with autoware.repos files

Summary

Option	UX Improvement	Migration Cost	Maintains Modularity	Newcomer Friendliness	Risk
Merge Core into Autoware	Very High	Very High	Low	Very High	High
Submodule/Subtree	High	Medium-Low	Medium-High	High	Medium
README under src/	Medium	Very Low	High	Medium	Very Low
Keep current	Low	None	High	Low	Low

[xmfcx]

I highly suggest adding a README.md under src/ and improve the documentation where necessary.

The other options all bring unnecessary complexity and/or chaos.

Also let's move the .repos files under repos/ directory and do more tidying for the main repository.

❌ Merging autoware_core into autoware repository

We have already did so many migrations and created so much confusion in our history. GitHub autoware.ai -> GitLab Autoware.Auto -> GitHub autoware.core + autoware.universe -> autoware_core + autoware_universe.

Right now the roles of the repositories are clear, modular and we have precedent https://github.com/ros2/ros2
And the role of the main repository is to maintain dependencies for all the sub repositories, allow people to pick their modules.
For example, autoware_msgs is a core repository but it is separated, we don't talk about merging it with the main repository.

Also versioning autoware_core and autoware meta repository will be confusing too.

❌ Submodules

This makes us lose too much flexibility. One example is, "Submodules are designed to pin an exact commit." so we won't have nightly anymore. Making development harder.

vcs import < autoware.repos
vcs import < simulator.repos
vcs import < tools.repos

Right now we (and the autoware integrators) have modularity with what repos to pick. With git submodules, this wont be possible.

Also it will become complicated when people fork both this meta-repo and some sub-repos. If someone tested if where a submodule is modified to point to some other repo, please let us know.

[paulsohn]

One example is, "Submodules are designed to pin an exact commit." so we won't have nightly anymore. Making development harder.

@xmfcx I think you can do git submodule update --remote whenever you want, to retrieve the latest HEAD.
In .gitmodules, you can even specify which branch to follow using submodule.<SUBMODULE_NAME>.branch.
The nightly is then obtained in your local by syncing all versions using git submodule update --remote.
If you want to pin a version because of the consistency issue, then add submodule.<SUBMODULE_NAME>.update = none. This prevents the above command checks out the latest branch HEAD unless you explicitly add --checkout.

[paulsohn]

Gemini told me that

vcs import < autoware.repos
vcs import < simulator.repos
vcs import < tools.repos

can be done similar using submodule: place all repositories in autoware.repos into src/autoware/, repositories in simulator.repos into src/simulator/ and so on. You can fork Autoware non-recursively, and then git submodule update --init src/autoware/ to use only repositories previously from autoware.repos.

I'm unsure about this yet, so I will do some experiment soon.

[evshary]

I prefer option 3 because options 1 and 2 decrease modularity and increase complexity.
The detailed reason why I don't like it is just as Fatih mentioned, so I don't repeat it again.

As for option 3, I think we can not only tell newcomers where the code is, but also tell them how to get the code.
I suppose the newcomers are not familiar with vcs and struggle to find a way to get the code.
It would be great to mention this in the README. My two cents.

[felixf4xu]

yes, list the https://autowarefoundation.github.io/autoware-documentation/main/installation/autoware/source-installation/ in the README to be the first step.

[paulsohn]

Definitely the short-term goal should be option 3, but I don't get why submodule should be hated this much.

I don't think submodule ever compromises our development flexibility, but even if it did, I would still prioritize observability over flexibility as we're dealing with safety-critical software.
For instance, with vcs status we can't tell whether the entire super-repository is dirty or not by one-liner. It will output so many lines of HEAD detached at <SHA>, unless you filtered it using grep. It doesn't even work on the project base, but only in src.
Also, other third-party version management tools, such as git graph (VS code extension) supports git submodule but obviously not vcstool.

[xmfcx]

For submodules, I would like to see a reference implementation of this as a separate repository and see if it can support all the use cases we have with vcs.

Since our .repos files are well defined, it should be possible to be done by anyone.

Using vcs, I can keep the sub-repositories intact while jumping up and down commits in the main repository. And only update the vcs import when I want to. But when submodules are involved, it will make it more bulky to change branches/commits etc.

I haven't used submodules for a long time so this is why I'm recommending to create a mirror of the main autoware repoistory using submodules so we can try and see how it compares objectively.

[paulsohn]

@xmfcx
Here's my example of my intended submodule management: https://github.com/paulsohn/toy-advanced-submodule .
I've been testing how it works, can I do partial cloning or perform nightly update etc.

I would now like to move on to applying this mechanism into my forked autoware; if you feel some of the vcstool features are not yet covered, please let me know which.

[paulsohn]

For submodules, I would like to see a reference implementation of this as a separate repository and see if it can support all the use cases we have with vcs.

@xmfcx cc @mitsudome-r
I have worked on my said reference implementation in #6611 . If we ever consider the submodule option, then I hope it could be a good starting point.

[xmfcx]

• refactor: move *.repos under repositories/ #6646 adds a README.md under src/ to help alleviate this issue.
• feat: add repositories, versioning and release process pages autoware-documentation#761 supports it with the main documentation.

[mojomex]

@mitsudome-r Thank you for this proposal!

The need for this change

This is a sorely needed improvement: my first experience with Autoware four years ago was quite rough and I did not at all enjoy using it: the main branch didn't build most days.

While this has gotten better, the -nightly.repos approach is against software engineering best practices:

• The main branch of a project shall always build successfully and be in a working state.
• A specific commit abcdef1 of a project should produce the same build result today, and in a year from now - it should be reproducible.

I would strongly favor your option 2:

Git submodules are a well-understood and battle-tested mechanism for meta-repos. The very fact that they always point to a fixed Git revision ensures that an update to a dependency (e.g. autoware_universe does not break or change the meta-repo autoware).

About submodules

@xmfcx

❌ Submodules
This makes us lose too much flexibility. One example is, "Submodules are designed to pin an exact commit." so we won't have nightly anymore. Making development harder.

I don't think this is true. When using submodules, users, including us developers, can be certain that whenever they git pull Autoware, the state they get has been tested, or at least built in CI with the fixed dependency versions.

Currently, this check is burdened on users: users pull Autoware and try to build, (hopefully) creating an issue when there are problems.

Or worse, people get cynical and fix the issues silently only on their machine, or use old versions of Autoware that are known-good. Most people around me are of the cynical kind and are using old Autoware for months on end.

The nightly functionality can be automated by CI in the autoware repo: a workflow similar to dependabot that bumps all dependency versions to the newest commit, and checks if the project builds and tests pass.

That CI would run either

• every time a dependency (e.g. autoware_universe) changes
• daily
• when users trigger it manually

Its changes would only get merged if the build succeeds with the updated dependencies.

Note that this does not restrict developers:

• if you need changes the second they are made, you can locally update the submodules instantly using git submodule update --remote
• otherwise, any time you pull autoware, it will have dependency versions at most a day old, thanks to the CI above
• as a big bonus, in the latter case, the user is guaranteed that the repo builds

Conclusion

There are no numbers on this, but having talked to people about this, I suspect that the current issues cost us dearly in terms of new contributors and industry adoption. At one trade show, a person commented that Autoware doesn't seem to be very good - they downloaded it and it didn't build.

Submodules are a de-facto standard in the rest of the industry - even our competition is using them!

[mitsudome-r]

If my understanding is correct, your request to actually have a nightly build version of autoware besides the stable version and dev version.

At the moment, we treat autoware.repos to be the stable release version and autoware-nightly.repos to be the dev version. If your proposal is to make autoware.repos to be the nightly version, I'm personally fine with it. However, we should set the default branch to be the tagged version in that case.

[mitsudome-r]

We could add this as a discussion topic in Software WG in the next call.

[mojomex]

@mitsudome-r Thank you for your feedback!

I'll certainly attend next call and think some more about it in the meantime.

[paulsohn]

Could you go into what the migration cost is?

I think I've been overestimating the migration cost. We can have the transition period with zero downtime.
There are still lots of CI tasks to do if we break things step-by-step, while that doesn't prevent us from proceeding submodule migration.

1. Basic CI supports
   a. Pre-commit checks to prevent inconsistency between legacy .repos and submodule status, and we should also check .gitmodules whether the SHA is on the specified branch if branch = <branch> is specified, and either update = none or branch = <branch> exists.
   b. For above, agreement on directory conventions (as I mentioned in feat(proposal draft, submodule): manage repositories as submodules instead of vcstool. #6611) : which directory should we map to which repos? A simple convention is included in the same PR, where simply src/<xxx>/path/to/repo corresponds to repositories."/path/to/repo" in <xxx>.repos. Maybe after repos files are dropped we will have a chance to revise directory structure once, but restructuring existing submodules might be painful to end users if not handled carefully.
   c. While I succeeded to to mimic all features in vcs2l using submodules (see my toy-advanced-submodule repository), some features are indirect, e.g. you need git submodule status to check which tag is on the submodule head. On PRs we might want to check submodule status and submodule summary quickly.
   d. (TIER IV specific) branch manager will be broken for a while if we apply this to TIER IV repositories.

2. webauto support
   a. We should take vcs2l control back from webauto configuration.
   b. While not much necessary, webauto can have a submodule based checkout. An example is at the end of this comment.

3. Instead of having nightly (or unstable) as repos, we need to configure unstable run to update daily.
   a. We need an unstable branch that automatically checkouts submodules to branch head, and (if we regard main = nightly) create/update a PR targeted to main when updated.
   b. Scheduled run should be configured to use that nightly branch instead of main.

4. autoware_core, autoware_universe etc should not require submodule migration. It can still use vcs2l for Github Actions.

autoware/.webauto-ci.yaml

Lines 14 to 19 in 8e74cdb

	source:
	vcstool:
	repositories_files:
	- repositories/autoware.repos
	- repositories/simulator.repos
	import_path: src

      source:
-       vcstool:
-         repositories_files:
-           - repositories/autoware.repos
-           - repositories/simulator.repos
-         import_path: src
+      submodule_pathspecs:
+        - src/autoware
+        - src/simulator

[mojomex]

Thank you! Seems doable. I'd propose the following transition steps:

1. Implement CI for frequent, checked autoware.repos updates

Have CI automation deal with updating dependency versions in autoware.repos. The key is that it runs proper builds/tests and only after CI succeeds are the changes committed to main.

This would be needed for submodules as well, and gets us most of the way to reproducible builds.

2. Shadow operation of submodules

While still keeping the .repos as you proposed, we can introduce submodules as an alternative mechanism. Once CI, Web.Auto and our workflows work with submodules, we can deprecate .repos and encourage users to use submodules.

3. Sunset .repos files

After a deprecation period, remove all .repos files and force users to use submodules.

Crucially, even if people want to stick to -nightly, that is doable with the branch/update properties of submodules as you showed.

[technolojin]

I prefer monorepo BTW 😉

Anyway, I also experienced the first trial of

1. almost empty repo
2. find submodule and fail
3. read the official guild and find *.repos
4. pull and build (why so many repos?)
5. build fail

I think this is not only the multi-repo strategy but also versioning.
Multi-vcs configs may satisfy flexibility desires, but that is double sided sword.