Using the template-id will have the random order.

[seankingyang]

Bug Description

After I use template-id, the order will be random, the test cases can't keep the order which is pre-defined in wireless-netplan-cert-automated and client-cert-desktop-24-04-automated.
The order is different in each time when I run the checkbox.

I create two branch to check this issue, wireless_use_regex and wireless_use_template_id.
And the output :
Regex_output.zip
Template_id_output.zip

checkbox job order will be different in each time:

$ diff wireless_auto_template_id_1.log wireless_auto_template_id_2.log | grep wireless                                                                               
< com.canonical.certification::wireless_sta_protocol
< com.canonical.certification::wireless/wireless_connection_open_ax_nm_wlp2s0
< com.canonical.certification::wireless/wireless_connection_wpa_ax_nm_wlp2s0
> com.canonical.certification::wireless/wireless_connection_open_bg_nm_wlp2s0
> com.canonical.certification::wireless_sta_protocol
> com.canonical.certification::wireless/wireless_connection_wpa_be_np_wlp2s0
< com.canonical.certification::wireless/wireless_connection_wpa3_be_nm_wlp2s0
< com.canonical.certification::wireless/wireless_connection_wpa_bg_nm_wlp2s0
< com.canonical.certification::wireless/wireless_connection_open_be_np_wlp2s0
< com.canonical.certification::wireless/nm_connection_restore_wlp2s0
< com.canonical.certification::wireless/wireless_connection_wpa_bg_np_wlp2s0
< com.canonical.certification::wireless/wireless_connection_wpa_n_nm_wlp2s0
< com.canonical.certification::wireless/wireless_connection_wpa_ax_np_wlp2s0
< com.canonical.certification::wireless/wireless_connection_wpa_ac_nm_wlp2s0
< com.canonical.certification::wireless/wireless_connection_open_be_nm_wlp2s0
> com.canonical.certification::wireless/wireless_connection_wpa_bg_np_wlp2s0
> com.canonical.certification::wireless/wireless_connection_wpa_ax_nm_wlp2s0
< com.canonical.certification::wireless/wireless_connection_wpa3_ax_nm_wlp2s0
< com.canonical.certification::wireless/wireless_connection_wpa3_ax_np_wlp2s0
< com.canonical.certification::wireless/wireless_connection_wpa_n_np_wlp2s0
< com.canonical.certification::wireless/wireless_scanning_wlp2s0
< com.canonical.certification::wireless/wireless_connection_wpa_be_np_wlp2s0
< com.canonical.certification::wireless/wireless_connection_open_ac_nm_wlp2s0
< com.canonical.certification::wireless/wireless_connection_open_n_nm_wlp2s0
> com.canonical.certification::wireless/wireless_connection_wpa_ac_nm_wlp2s0
< com.canonical.certification::wireless/wireless_connection_open_bg_nm_wlp2s0
> com.canonical.certification::wireless/wireless_connection_open_be_nm_wlp2s0
> com.canonical.certification::wireless/wireless_connection_wpa_n_nm_wlp2s0
> com.canonical.certification::wireless/wireless_connection_wpa_bg_nm_wlp2s0
> com.canonical.certification::wireless/wireless_connection_wpa3_ax_np_wlp2s0
> com.canonical.certification::wireless/wireless_connection_wpa3_be_nm_wlp2s0
> com.canonical.certification::wireless/wireless_connection_wpa3_ax_nm_wlp2s0
> com.canonical.certification::wireless/nm_connection_restore_wlp2s0
> com.canonical.certification::wireless/wireless_connection_open_n_nm_wlp2s0
> com.canonical.certification::wireless/wireless_connection_open_be_np_wlp2s0
> com.canonical.certification::wireless/wireless_connection_wpa_ax_np_wlp2s0
> com.canonical.certification::wireless/wireless_scanning_wlp2s0
> com.canonical.certification::wireless/wireless_connection_open_ac_nm_wlp2s0
> com.canonical.certification::wireless/wireless_connection_wpa_n_np_wlp2s0
> com.canonical.certification::wireless/wireless_connection_open_ax_nm_wlp2s0

And there is no any wireless test job difference if using the regex.

To Reproduce

1. install checkbox
2. git clone these two branch wireless_use_regex and wireless_use_template_id
3. sideload the base certification-client
4. checkbox-cli list-bootstrapped com.canonical.certification::client-cert-desktop-24-04-automated

Environment

• OS: 22, 24
• checkbox: deb 4.2.0.dev99

Relevant log output

No response

Additional context

No response

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/CHECKBOX-1603.

This message was autogenerated

[fernando79513]

The ordering of the regex is not an intended behavior. We can not ensure that the tests are going to be executed in the order they are defined in the pxu.

Even with the regex syntax, if one of the tests fails, the order after retrying will probably be different.

We could implement some alphabetical ordering that keeps the dependency structure of the tests, or force them to keep the order in which they are defined (also keeping the dependency structure) but that's not a feature at the moment.