Netplan support network manager 2404 PC Desktop Test case/plan (New)

[seankingyang]

Description

This PR only change/add the test case/plan to support the netplan test cases can be run in the 24.04 PC desktop test plan.
It need the #1301 PR script will make the script can support on the NetworkManager.
This PR need to wait the #1324 PR, or it will meet this issue:OEMQA-4582 NetworkManager test case impact in 24.04 Desktop

WARNING: This modifies com.canonical.certification::sru-server

Change the requires field in bellow test cases:

• {after-suspend-}wireless/wireless_connection_{open|WPA}_{bg|n|ac}_np_.*
• {after-suspend-}wireless/wireless_connection_{open|WPA|WPA3}_{ax|be}_np_.*

From:

requires:
 wireless_sta_protocol.{{ interface }}_be == 'supported'
 net_if_management.device == '{{ interface }}' and net_if_management.managed_by == 'networkd'

TO:

requires:
 wireless_sta_protocol.{{ interface }}_be == 'supported'
 (net_if_management.device == '{{ interface }}' and net_if_management.managed_by == 'networkd') or lsb.release >= "24"

Add a new test plan {after-suspend-}wireless-netplan-cert-automated to support the following test cases:

• {after-suspend-}wireless/wireless_connection_{open|WPA}_{bg|n|ac}_np_.*
• {after-suspend-}wireless/wireless_connection_{open|WPA|WPA3}_{ax|be}_np_.*

Resolved issues

Documentation

Tests

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 48.32%. Comparing base (6205aed) to head (66c9b21).
Report is 113 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1313   +/-   ##
=======================================
  Coverage   48.32%   48.32%           
=======================================
  Files         372      372           
  Lines       40091    40091           
  Branches     6759     6759           
=======================================
  Hits        19373    19373           
  Misses      20002    20002           
  Partials      716      716           

Flag	Coverage Δ
provider-base	24.81% <ø> (ø)
provider-certification-client	57.14% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[seankingyang]

Rebase

[fernando79513]

So the idea of this PR is that the netplan wireless tests are run in 24.04 onwards despite of the results of the net_if_management resource.
That's because the net_if_management resource does not work properly in 24.04?
Could it be possible to modify net_if_management.py to work in both cases and keep the syntax of the requires section as simple as possible?

[seankingyang]

net_if_management can be run in the 24.04.
Here is the output:

===========================[ Running Selected Jobs ]============================
==============[ Running job 1 / 1. Estimated time left: 0:00:02 ]===============
-----[ Identify what service is managing each physical network interface ]------
ID: com.canonical.certification::net_if_management
Category: com.canonical.certification::information_gathering
... 8< -------------------------------------------------------------------------
device: eno1
managed_by: NetworkManager
master_mode_managed_by: not-applicable

device: wlp2s0f0
managed_by: NetworkManager
master_mode_managed_by: NetworkManager

------------------------------------------------------------------------- >8 ---
Outcome: job passed

wireless/wireless_connection_wpa_XX_netplan_* were supposed to run when management_by == 'networkd'.
In 24.04, it suppose to be run in both management_by == 'networkd' and management_by == 'NetworkManger'. (Netplan officially used in 24.04, not officially used before 24.04.)

(net_if_management.device == '{{ interface }}' and net_if_management.managed_by == 'networkd') is for OS Version < 24.

Do you have any suggestion to change the way if not use complex syntax of requires section?
My ideal is put the conditional net_if_management.managed_by == 'networkd' or lsb.release >= "24" in the net_if_management.py, print netplan_supported: <True/False> in output:
so the format when OS < 24 will be like:

device: eno1
managed_by: networkd
master_mode_managed_by: not-applicable
netplan_supported: True

device: wlp2s0f0
managed_by: networkd
master_mode_managed_by: NetworkManager
netplan_supported: True

and

device: eno1
managed_by: NetworkManager
master_mode_managed_by: not-applicable
netplan_supported: False

device: wlp2s0f0
managed_by: NetworkManager
master_mode_managed_by: NetworkManager
netplan_supported: False

The for format when OS >=24 will be like:

device: eno1
managed_by: NetworkManager
master_mode_managed_by: not-applicable
netplan_supported: True

device: wlp2s0f0
managed_by: NetworkManager
master_mode_managed_by: NetworkManager
netplan_supported: True

[seankingyang]

After I use template-id, the order will be random, the test cases can't keep the order which is pre-defined in wireless-netplan-cert-automated and client-cert-desktop-24-04-automated.
The order is different in each time when I run the checkbox.

Please check the following attachment.

auto_certtestplan_with_template_id_1.log
auto_certtestplan_with_template_id_2.log

[fernando79513]

After I use template-id, the order will be random, the test cases can't keep the order which is pre-defined in wireless-netplan-cert-automated and client-cert-desktop-24-04-automated. The order is different in each time when I run the checkbox.

Is that random ordering an issue? The order in which tests are run in checkbox is not deterministic at the moment. We only allow depends or after to manage the ordering of tests.

The ordering of regexes is incidental, and we are still prefering the "template-id" syntax to create automated testplan documentation

We will follow in the future the issue #1529 if it helps QA testing.

[seankingyang]

Rebase

[fernando79513]

Hello @seankingyang, thanks for the changes!
Have you tested in some devices that these changes (mainly the removing of config save/restore) do not mess up with other tests by running the sru test-plan in some devices using 24.04 and older versions?
If that's the case, I'm happy with this changes.
If not, then I think we agreed during office hours to use the "regex" approach until we have a better way to manage the ordering of the tests.

[seankingyang]

Hi @fernando79513, i tested it and it worked well. Please check this submission https://certification.canonical.com/hardware/202411-35967/submission/403611/.
Remove the store/restore nm configuration case is very safe, cos these two case it only want to make the machine can reconnect back to original wifi ap. Normaly, we will not connect the wifi ap before testing, and the nm configuration is empty.

[fernando79513]

LGTM +1!