New match feature (new)

[Hook25]

Description

Re-running an entire test plan is very time consuming, sometimes we have to do it to only fix a handful of jobs and this stresses our lab and makes us waste time. One solution on the Checkbox side is to have a new configuration that allows us to run a few specific tests.

Similarly to how exclude= works, we should add a new section to the [test selection] configuration that does the opposite of exclude. Namely: Given a bootstrapped testplan run all tests that match the match or that the matching tests depend on

If this is the dependency graph, where the arrow means: depends on (namely H depends on G)

When the following match constraint is applied:

[test selection]
match=
  B
  E

The following jobs will be run:

• anything in Bootstrap
• B (direct match)
• A (B depends on A)
• E (direct match)
• D (E depends on D)
• C (D depends on C)

The following wont be run:

• F
• H
• G

Note: This feature must support creating a submission after running

Resolved issues

Fixes: CHECKBOX-1525

Documentation

Documentation added in the reference section along with other sections. No section was added to the tutorial as I feel this is a more "advanced" topic

Tests

New metabox tests and unit tests added

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 45.24%. Comparing base (4123afe) to head (cb882e3).
Report is 122 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1422      +/-   ##
==========================================
+ Coverage   45.19%   45.24%   +0.04%     
==========================================
  Files         367      367              
  Lines       39119    39125       +6     
  Branches     6613     6616       +3     
==========================================
+ Hits        17681    17701      +20     
+ Misses      20764    20751      -13     
+ Partials      674      673       -1     

Flag	Coverage Δ
checkbox-ng	67.69% <100.00%> (+0.10%)	⬆️
checkbox-support	59.71% <ø> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[pieqq]

Great change that will help us a lot, especially with the re-runs in the lab!

I tested it locally using the following launcher:

[test plan]
unit = com.canonical.certification::sru
forced = yes

[test selection]
match = com.canonical.certification::benchmarks/disk/hdparm-read_name

It launched the SRU test plan, but only selected the stuff in bootstrap and the Disk benchmark test to be run (benchmarks/disk/hdparm-read_name is a template id). I also modified the match section to check that it worked properly with job ids and regexes. All good!

Tiny things here and there (see inline) that you might want to modify, but otherwise this can land.

[pieqq]

Thanks!