Run ethernet/ping_interface unless hidden manifest set (e.g. lab runs) (new)

[pieqq]

Description

Use newly introduced hidden manifest (see #1699) to condition the run of the ethernet/ping_interface template jobs. This way, in the lab where devices with multiple Ethernet interfaces usually only have one of them connected, these jobs can be skipped and the generic ethernet/ping-with-any-cable-interface job can be run instead (validating that the Ethernet feature works to connect to ping the gateway). To that effect, ethernet/ping-with-any-cable-interface is added to the ethernet-automated and after-suspend-ethernet-automated test plans, alongside the ethernet/ping_interface template.

Resolved issues

https://warthogs.atlassian.net/browse/RTW-401

Documentation

The hidden manifest has a hidden-reason field which helps for documentation.

Tests

Create the following launcher (my-launcher):

[launcher]
launcher_version = 1
app_id = com.canonical.certification:RTW401
stock_reports = text

[test plan]
unit = com.canonical.certification::ethernet-automated
forced = yes

[test selection]
forced = yes

[manifest]
com.canonical.certification::_ignore_disconnected_ethernet_interfaces = true

and execute it:

$ checkbox-cli my-launcher

Observe that the ping_{interface} template jobs are skipped, and the ping-with-any-cable-interface is run:

==============[ Running job 3 / 4. Estimated time left: 0:00:08 ]===============
-----------[ Can ping another machine over Ethernet port enp0s31f6 ]------------
ID: com.canonical.certification::ethernet/ping_enp0s31f6
Category: com.canonical.plainbox::ethernet
Job cannot be started because:
 - resource expression "manifest._ignore_disconnected_ethernet_interfaces == 'False'" evaluates to false
Outcome: job cannot be started
==============[ Running job 4 / 4. Estimated time left: 0:00:04 ]===============
-----------[ Can ping the gateway with any cable Ethernet interface ]-----------
ID: com.canonical.certification::ethernet/ping-with-any-cable-interface
Category: com.canonical.plainbox::ethernet
... 8< -------------------------------------------------------------------------
Looking for all cable interfaces...
FAIL: Couldn't find any suitable cable interface.
------------------------------------------------------------------------- >8 ---

Remove the [manifest] section from the launcher and repeat the operation. This time, both ping_{interface} template jobs and ping-with-any-cable-interface are run.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 49.28%. Comparing base (c52cf57) to head (0e62122).
Report is 148 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1709      +/-   ##
==========================================
+ Coverage   49.26%   49.28%   +0.01%     
==========================================
  Files         373      374       +1     
  Lines       40434    40445      +11     
  Branches     6830     6831       +1     
==========================================
+ Hits        19921    19932      +11     
  Misses      19785    19785              
  Partials      728      728              

Flag	Coverage Δ
provider-base	∅ <ø> (∅)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[mz2]

Since this is a recurrent source of SRU test hiccups, could you please...

• put in contribution guide a mention aimed at Canonical team members to update the lab machine manifests before merging a new test like this.
• rename the relevant repository of manifests to something that doesn't take a galaxy brain to remember it 😄 (https://github.com/canonical/ce-oem-dut-checkbox-configuration -- which is safe to mention in the public) and broadcast that. Renaming won't invalidate URLs, deploy keys etc immediately (or even very quickly I think), but they'll need to be done eventually.
• add a CI check in said repo to guard that expected manifest keys are included in all of the manifests included, so that people contributing new manifests know what to do.

[pieqq]

GitHub actions on both sides (here, and in the DUT configuration repo) have been created and tweaked so that:

• (Checkbox project) on every PR in Checkbox, a check for new hidden manifest entries is triggered
• (DUT Configs project) if there are new hidden manifest entries, the "check missing manifests" action is triggered on the DUT config project
• (DUT Configs project) if these new manifest entries have not been entered for manifest.json files belonging to devices in the lab, a commit status failure is sent back to the Checkbox commit, marking the PR in Checkbox as having "failed checks", with a link back to the DUT config action to see what's wrong
• (DUT Configs project) otherwise, a commit status success is sent back to the Checkbox commit

Initially, this check was failing on this PR, because 1012 devices in the lab were missing the new hidden manifest.

After adding them, the check item is green:

[mz2]

Great work, thanks for the big CI lift that this involved!