Some status of cases are still be non-blocker even apply blocker to .* to test plan

[baconYao]

Bug Description

In PR#1206 Certification team made every test job in client-cert-iot-.* test plans as cert blocker according to a decision made a few years ago which we have been using to judge the bug importance since then (All checkbox jobs being executed for the IoT project is a cert blocker)

Therefore, QA team started implementing this rule in each project. However, we observed that the status is still be non-blocker of some cases who are from base provider.

• sideload submission on genio-1200 desktop image: https://certification.canonical.com/hardware/202307-31864/submission/395052/test-results/
• sideload submission on baytown: https://certification.canonical.com/hardware/202401-33430/submission/394986/test-results/

Here is the simple PR to implement this feature. What I did is like below (nest the cert test plan and have certification_status_overrides field be defined):

bootstrap_include:
    baoshan_device_soc
include:
nested_part:
    .
    .
    com.canonical.certification::client-cert-desktop-22-04-automated
    .
    .
certification_status_overrides:
    apply blocker to .*
exclude:
    .
    .
    .

To Reproduce

1. Have a test plan that nests the com.canonical.certification::client-cert-desktop-22-04-automated
2. Add certification_status_overrides field
3. Sideload this test plan
4. Checkbox remote runs this auto plan and choose the miscellaneous and info sets then executing it.

Environment

• OS: Classic Desktop / Server / Ubuntu Core
• Checkbox type: Snap

Relevant log output

No response

Additional context

No response

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/CHECKBOX-1577.

This message was autogenerated

[pieqq]

If everything is in the same namespace, the issue does not happen.

However, if the override is applied to a test plan in one of the provider (with one namespace), and that test plan is then nested in another provider (with another namespace), the override is not applied.

→ Issue confirmed.

[pieqq]

A workaround for you:

use

certification_status_overrides:
    apply blocker to .*::.*

instead of

certification_status_overrides:
    apply blocker to .*

The issue lies in the testplan unit where Checkbox adds the namespace to everything that does not have a namespace.

[baconYao]

Hi @pieqq
I've tried the method you provided but still has non-blocker cases.

certification_status_overrides:
    apply blocker to .*::.*

The Submission-A use apply blocker to .*::.*

The Submission-B use apply blocker to .*

Compare Submission-A and Submission-B, we can find the miscellanea/submission-resources case has been blocker but info/systemd-analyze and info/systemd-analyze-critical-chain cases are still non-blocker

[pieqq]

I think I found the culprit:

checkbox/checkbox-ng/plainbox/impl/unit/testplan.py

Lines 803 to 806 in c9201b3

	return sorted(
	(key, field_value_list)
	for key, field_value_list in override_map.items()
	)

For some reason, the list of override_map items is returned sorted. This is why the non-blocker status you see is still returned, even though it should be overridden by the certification_status_overrides section of the test plan.

By removing the sorted function, I get what you expect, so this seems to be the solution. However, I need to investigate why this list is returned sorted, and I need to write more tests to make sure I'm not breaking something else, somewhere else :)

[pieqq]

I spent quite some time on this yesterday and I cannot reproduce the issue anymore...

If I use the following override in the genio-baoshan-core-22-automated test plan:

certification_status_overrides:
    apply blocker to .*::.*

and run

checkbox-cli expand "com.canonical.qa.baoshan::genio-baoshan-core-22-automated" --format json | jq > /var/tmp/baoshan.json

All the jobs in the list have the certification_status set to blocker, as expected.

I've also tried with a very small and dumb test plan, namely:

id: my-testplan-override
_name: my testplan override
unit: test plan
include:
  com.canonical.certification::info/systemd-analyze certification_status=non-blocker
certification_status_overrides:
  apply blocker to .*::.*

and when I use the expand command as above, the job is, indeed, set to blocker as expected, even though it has a non-blocker inline override.

[baconYao]

Hi @pieqq,

After sideload testsing, I found these two info/systemd-analyze and info/systemd-analyze-critical-chain jobs are still non-blocker

• https://certification.canonical.com/hardware/202309-32029/submission/411667/test-results/

I attached the baoshan server auto plan like below,

id: genio-baoshan-server-22-04-automated
unit: test plan
_name: genio-baoshan - Automated only QA tests for Ubuntu Server 22.04
_description:
 Ubuntu Server 22.04 QA test plan for the baoshan genio hardware. This test plan contains
 all of the automated tests used to validate the baoshan device.
bootstrap_include:
    baoshan_device_soc
include:
    com.canonical.certification::install/apt-get-gets-updates
nested_part:
    baoshan-mipi-camera-automated
    baoshan-thermal-automated
    baoshan-boot-automated
    baoshan-ebbr-automated
    baoshan-peripheral-automated
    baoshan-power-management-automated
    baoshan-mir-classic-automated
    baoshan-info-automated
    baoshan-machine-automated
    com.canonical.contrib::ce-oem-automated
    com.canonical.certification::client-cert-iot-server-22-04-automated
    com.canonical.contrib::after-suspend-ce-oem-automated
    after-suspend-baoshan-machine-automated
    after-suspend-baoshan-boot-automated
    after-suspend-baoshan-power-management-automated
    after-suspend-baoshan-peripheral-automated
    after-suspend-baoshan-mir-classic-automated
    after-suspend-baoshan-mipi-camera-automated
    # Thermal simulate-overheating will shut system down
    # So keep it at the bottom of after-suspend
    after-suspend-baoshan-thermal-automated
certification_status_overrides:
    apply blocker to .*::.*
exclude:
    com.canonical.certification::networking/predictable_names
    com.canonical.certification::disk/read_performance_sd[ab]           # LP#2039902
    com.canonical.certification::disk/storage_device_sd[ab]             # LP#2039902
    com.canonical.certification::after-suspend-disk/read_performance_sd[ab]             # LP#2039902
    com.canonical.certification::after-suspend-disk/storage_device_sd[ab]               # LP#2039902