Zephyr Modules: document changes & improvements in the workflow

[ioannisg]

I 'd like to open, already, a PR to draft the proposed updates/improvements to the workflow around Zephyr Modules.
Based on the feedback from the Process Forum meetings. Captured in https://docs.google.com/presentation/d/1okOQShAr3GZF7VOfRCGa9mw_L7Hyis-oH1kGUpOA66c/edit?userstoinvite=jettrink%40google.com&ts=5f1eef33&actionButton=1#slide=id.p

Keep as draft until the Process Forum has finalized the proposal.

Moving this to non-draft, since the discussions are concluded, and the Forum's feedback is reflected here.
Comments are still welcome.
The proposal requires TSC approval.

[aescolar]

@ioannisg : There is a fundamental disagreement here, so re-requesting a review won't change mine (#27185 (review)). I still think it should be possible for a module to track directly the upstream repository main branch without the need for human intervention + This proposal forgets about modules which are not meant to be compiled with the Zephyr code base (for ex. tools).
If you do not want to address these concerns you can dismiss my review.

[ioannisg]

@ioannisg : There is a fundamental disagreement here, so re-requesting a review won't change mine (#27185 (review)). I still think it should be possible for a module to track directly the upstream repository main branch without the need for human intervention + This proposal forgets about modules which are not meant to be compiled with the Zephyr code base (for ex. tools).
If you do not want to address these concerns you can dismiss my review.

@aescolar I re-requested review by all participants here, since I addressed a huge list of comments (including many of yours) and updated the proposal text. You might want to ACK the other feedback I resolved :)

Regarding your comment, I understand the disagreement, but I don't know how to respond to that, other than inviting all frequent project forum participants (@galak @carlescufi @daor-oti @nashif @jettr @MaureenHelm) to attend to this matter (and possibly iterate on the disagreement once more). I don't decide myself whether to address or to dismiss review feedback.

FYI, the whole proposal will be brought in TSC for approval, once it is finalized.

[nvlsianpu]

@ioannisg : There is a fundamental disagreement here, so re-requesting a review won't change mine (#27185 (review)). I still think it should be possible for a module to track directly the upstream repository main branch without the need for human intervention.

@aescolar If track directly the upstream repository mean to use exact copy of the upstream on zephyr-rtos/ - I can agree.
Looks like this manual makes thing more complex than it was before, just because forces standardization to two synchronization methods. Both of this methods miss automatic GIT main benefit - versions relations control.

[aescolar]

@aescolar If track directly the upstream repository mean to use exact copy of the upstream on zephyr-rtos/ - I can agree.

@nvlsianpu yep. Just like this one: https://github.com/zephyrproject-rtos/edtt/ , which is an automated mirror* of https://github.com/EDTTool/EDTT (* only adds commits, doesn't force push)

[henrikbrixandersen]

I tend to agree with @aescolar on allowing zephyr-rtos/ forks to track upstream one-to-one, especially if we somehow could move zephyr/module.yaml and related files into the zephyr/modules repo/subdir.

This would allow for - at least some modules - to be maintained in a much less rigid fashion than what is currently proposed in this PR, while at the same time allowing downstream Zephyr users to update those modules in their manifest on their own pace (e.g. to track upstream fixes) since no Zephyr-specific code/commits are needed in the module.

[tejlmand]

I tend to agree with @aescolar on allowing zephyr-rtos/ forks to track upstream one-to-one, especially if we somehow could move zephyr/module.yaml and related files into the zephyr/modules repo/subdir.

I agree too. I think that if life can be made much easier for those maintaining modules, by extending zephyr_module.py, then that should be a goal. And the pure mirror approach should be described.

[aescolar]

Note that even without any changes in zephyr_module.py, already today some west modules are direct replicas. Either because the upstream repo accepted a zephyr/ folder, or because it does not need one at all (say a tool which is not compiled with Zephyr).

The change in zephyr_module.py would make this situation more likely.

[ioannisg]

Note that even without any changes in zephyr_module.py, already today some west modules are direct replicas. Either because the upstream repo accepted a zephyr/ folder, or because it does not need one at all (say a tool which is not compiled with Zephyr).

The change in zephyr_module.py would make this situation more likely.

@aescolar @tejlmand @nvlsianpu @henrikbrixandersen appreciate your feedback. This issue was discussed in the last Forum session. The decision is not to deal with this use-case at all, in the current proposal. The reason is that essentially, this policy is to be applied to non-module external projects. Normally, modules will require downstream commits and need to include a module.yml file in the code base; I've added this in the module definition which is re-worded in this PR.

That said, https://github.com/zephyrproject-rtos/edtt/ will currently not classify asa Zephyr module, but rather as an external project, so for now, this proposal does not restrict the update policies in this particular repository. We will deal with this in the future.

[aescolar]

@aescolar @tejlmand @nvlsianpu @henrikbrixandersen appreciate your feedback. This issue was discussed in the last Forum session. The decision is not to deal with this use-case at all, in the current proposal. The reason is that essentially, this policy is to be applied to non-module external projects. Normally, modules will require downstream commits and need to include a module.yml file in the code base; I've added this in the module definition which is re-worded in this PR.

That said, https://github.com/zephyrproject-rtos/edtt/ will currently not classify asa Zephyr module, but rather as an external project, so for now, this proposal does not restrict the update policies in this particular repository. We will deal with this in the future.

Thanks @ioannisg. Indeed my immediate concern are the EDTT and the nrf_hw_models as those are the ones I maintain.
And although my request for allowing direct replicas is general, I would personally have less trouble with this new procedure, if those 2 modules are considered excluded for whatever reason.
Anyhow:
The EDTT is indeed just a "west module", but not compiled with Zephyr
The nrf_hw models is both a "west module" and a "zephyr module" (meaning it is compiled in). But it is only used for CI/workstation testing, so that code does not end up in real targets.

[erwango]

There is nothing about policy around module specific CI (github actions for instance).
Is that allowed ? Any specific policy around these ?
I think this should be clarified, at least the existing status.

[ioannisg]

@carlescufi please, revisit your review, so we can get this PR merged :)

[carlescufi]

OK, fair enough