Skip to content

[checkbox-ce-oem] Revise the test plan structure of strict confinement test (Bugfix) #1419

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
stanley31huang merged 4 commits into from
Aug 20, 2024
Merged

[checkbox-ce-oem] Revise the test plan structure of strict confinement test (Bugfix) #1419

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
347af8e
Revise the test plan structure of strict confinement test
rickwu666666 Aug 15, 2024
1a0d81d
Fix the wrong connections condition
rickwu666666 Aug 15, 2024
f1a9ee0
Modify thermal job via regular expression
rickwu666666 Aug 15, 2024
e3a832d
Add manual/automated as postfix of test-plan for each feature
rickwu666666 Aug 15, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 9 additions & 5 deletions ...finement/mediacard-strict-confinement.pxu → ...its/strict-confinement/mediacard/jobs.pxu
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
plugin: user-interact
category_id: strict-confinement-mode
id: strict-confine/mediacard/sdhc-insert
id: strict-confinement/mediacard/sdhc-insert
flags: also-after-suspend
estimated_duration: 30.0
command:
Expand All @@ -9,10 +9,14 @@ imports:
from com.canonical.plainbox import manifest
from com.canonical.certification import snap
from com.canonical.certification import lsb
from com.canonical.certification import connections
requires:
manifest.has_card_reader == 'True'
lsb.distributor_id == 'Ubuntu Core'
snap.name == 'test-strict-confinement'
connections.slot == "snapd:mount-control" and connections.plug == "test-strict-confinement:mount-control"
connections.slot == "snapd:log-observe" and connections.plug == "test-strict-confinement:log-observe"
connections.slot == "snapd:removable-media" and connections.plug == "test-strict-confinement:removable-media"
user: root
_summary: Test that insertion of an SDHC card is detected
_description:
Expand All @@ -31,9 +35,9 @@ _verification:

plugin: shell
category_id: strict-confinement-mode
id: strict-confine/mediacard/sdhc-storage
id: strict-confinement/mediacard/sdhc-storage
estimated_duration: 30.0
depends: strict-confine/mediacard/sdhc-insert
depends: strict-confinement/mediacard/sdhc-insert
user: root
flags: preserve-cwd reset-locale also-after-suspend
command: test-strict-confinement.usb-read-write
Expand All @@ -44,10 +48,10 @@ _description:

plugin: user-interact
category_id: strict-confinement-mode
id: strict-confine/mediacard/sdhc-remove
id: strict-confinement/mediacard/sdhc-remove
flags: also-after-suspend
estimated_duration: 30.0
depends: strict-confine/mediacard/sdhc-insert
depends: strict-confinement/mediacard/sdhc-insert
command:
test-strict-confinement.run-watcher removal mediacard
user: root
Expand Down
22 changes: 22 additions & 0 deletions ...checkbox-ce-oem/checkbox-provider-ce-oem/units/strict-confinement/mediacard/test-plan.pxu
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
id: strict-confinement-mediacard-manual
unit: test plan
_name: Test mediacard in strict confinement mode.
_description:
Test mediacard in strict confinement mode.
Rely on test-strict-confinement SNAP to test.
include:
strict-confinement/mediacard/sdhc-insert
strict-confinement/mediacard/sdhc-storage
strict-confinement/mediacard/sdhc-remove


id: after-suspend-strict-confinement-mediacard-manual
unit: test plan
_name: After suspend test mediacard in strict confinement mode.
_description:
Afrer suspend test mediacard in strict confinement mode.
Rely on test-strict-confinement SNAP to test.
include:
after-suspend-strict-confinement/mediacard/sdhc-insert
after-suspend-strict-confinement/mediacard/sdhc-storage
after-suspend-strict-confinement/mediacard/sdhc-remove
4 changes: 4 additions & 0 deletions ...nt/powermanagement-strict-confinement.pxu → ...ict-confinement/power-management/jobs.pxu
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,11 @@ depends: com.canonical.certification::init-boot-loop-data
imports:
from com.canonical.certification import snap
from com.canonical.certification import lsb
from com.canonical.certification import connections
requires:
lsb.distributor_id == 'Ubuntu Core'
snap.name == 'test-strict-confinement'
connections.slot == "snapd:shutdown" and connections.plug == "test-strict-confinement:shutdown"
command:
rtcwake -d "${RTC_DEVICE_FILE:-rtc0}" -v -m no -s "${STRESS_BOOT_WAKEUP_DELAY:-120}"
test-strict-confinement.dbus-cold-boot
Expand Down Expand Up @@ -56,9 +58,11 @@ depends: com.canonical.certification::init-boot-loop-data
imports:
from com.canonical.certification import snap
from com.canonical.certification import lsb
from com.canonical.certification import connections
requires:
lsb.distributor_id == 'Ubuntu Core'
snap.name == 'test-strict-confinement'
connections.slot == "snapd:shutdown" and connections.plug == "test-strict-confinement:shutdown"
command:
test-strict-confinement.dbus-warm-boot
sleep 60
Expand Down
26 changes: 26 additions & 0 deletions ...x-ce-oem/checkbox-provider-ce-oem/units/strict-confinement/power-management/test-plan.pxu
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
id: strict-confinement-dbus-warm-boot-automated
unit: test plan
_name: Warm reboot test via dbus
_description:
Warm reboot test by using dbus command.
Rely on test-strict-confinement SNAP to test.
bootstrap_include:
mandatory_include:
com.canonical.certification::init-boot-loop-data
include:
strict-confinement/dbus-warm-boot
strict-confinement/dbus-warm-boot-test


id: strict-confinement-dbus-cold-boot-automated
unit: test plan
_name: Cold boot test via dbus
_description:
Cold boot test by using dbus command.
Rely on test-strict-confinement SNAP to test.
bootstrap_include:
mandatory_include:
com.canonical.certification::init-boot-loop-data
include:
strict-confinement/dbus-cold-boot
strict-confinement/dbus-cold-boot-test
10 changes: 6 additions & 4 deletions ...onfinement/thermal-strict-confinement.pxu → .../units/strict-confinement/sensor/jobs.pxu
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,20 +1,22 @@
unit: template
template-engine: jinja2
template-resource: thermal_zones
template-id: strict-confine/temperature-test
id: strict-confine/temperature_{{ name }}_{{ type }}
template-id: strict-confinement/temperature-test
id: strict-confinement/temperature_{{ name }}_{{ type }}
_summary: Check Thermal temperature of {{ name }} - {{ type }}
_description:
Test a thermal temperature for {{ name }} - {{ type }}.
category_id: thermal
category_id: strict-confinement-mode
plugin: shell
estimated_duration: 5m
flags: also-after-suspend
imports:
from com.canonical.certification import snap
from com.canonical.certification import lsb
from com.canonical.certification import connections
requires:
lsb.distributor_id == 'Ubuntu Core'
snap.name == 'test-strict-confinement'
connections.slot == "snapd:hardware-observe" and connections.plug == "test-strict-confinement:hardware-observe"
command:
test-strict-confinement.thermal-test monitor -n {{ name }} --extra-commands "dd if=/dev/zero of=/dev/null"
test-strict-confinement.thermal-test monitor -n {{ name }} --extra-commands "dd if=/dev/zero of=/dev/null"
21 changes: 21 additions & 0 deletions ...ib/checkbox-ce-oem/checkbox-provider-ce-oem/units/strict-confinement/sensor/test-plan.pxu
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
id: strict-confinement-thermal-automated
unit: test plan
_name: Test thermal sensor in strict confinement mode.
_desription:
Test thermal sensor in strict confinement mode.
Rely on test-strict-confinment SNAP to test.
bootstrap_include:
thermal_zones
include:
strict-confinement/temperature_.*

id: after-suspend-strict-confinement-thermal-automated
unit: test plan
_name: Test thermal sensor in strict confinement mode.
_desription:
Test thermal sensor in strict confinement mode.
Rely on test-strict-confinment SNAP to test.
bootstrap_include:
thermal_zones
include:
after-suspend-strict-confinement/temperature_.*
115 changes: 47 additions & 68 deletions ...ce-oem/checkbox-provider-ce-oem/units/strict-confinement/test-plan-strict-confinement.pxu
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,86 +1,65 @@
id: strict-confinement-dbus-warm-boot
id: strict-confinement-full
unit: test plan
_name: Warm reboot test via dbus
_name: Strict-confinement - Full manual + automated tests for Ubuntu Core in strict confinement environment
_description:
Warm reboot test by using dbus command.
Rely on test-strict-confinement SNAP to test.
bootstrap_include:
mandatory_include:
com.canonical.certification::init-boot-loop-data
Combined manual and automated test plans for Ubuntu Core in strict confinement environment.
include:
strict-confinement/dbus-warm-boot
strict-confinement/dbus-warm-boot-test
nested_part:
strict-confinement-manual
strict-confinement-automated
after-suspend-strict-confinement-manual
after-suspend-strict-confinement-automated


id: strict-confinement-dbus-cold-boot
id: strict-confinement-manual
unit: test plan
_name: Cold boot test via dbus
_name: Strict-confinement - Manual only QA tests for Ubuntu Core in strict confinement environment
_description:
Cold boot test by using dbus command.
Rely on test-strict-confinement SNAP to test.
bootstrap_include:
mandatory_include:
com.canonical.certification::init-boot-loop-data
Ubuntu Core QA test plan for strict confinement environment.
This test plan contains all of the tests that require manual
control of device hardware or some other user input to complete.
estimated_duration: 3600
include:
strict-confinement/dbus-cold-boot
strict-confinement/dbus-cold-boot-test

nested_part:
strict-confinement-mediacard-manual
exclude:

id: strict-confine-mediacard
id: strict-confinement-automated
unit: test plan
_name: Test mediacard in strict confinement mode.
_name: Strict-confinement - Automated only QA tests for Ubuntu Core in strict confinement environment
_description:
Test mediacard in strict confinement mode.
Rely on test-strict-confinement SNAP to test.
Ubuntu Core QA test plan for the strict confinement environment.
This test plan contains all of the automated tests used to validate
the Ubuntu Core in strict confinement environment.
include:
strict-confine/mediacard/sdhc-insert
strict-confine/mediacard/sdhc-storage
strict-confine/mediacard/sdhc-remove
nested_part:
strict-confinement-timedatectl-automated
strict-confinement-thermal-automated
strict-confinement-dbus-warm-boot-automated
strict-confinement-dbus-cold-boot-automated
exclude:


id: after-suspend-strict-confine-mediacard
id: after-suspend-strict-confinement-manual
unit: test plan
_name: After suspend test mediacard in strict confinement mode.
_name: Strict-confinement - After suspend Manual only QA tests for Ubuntu Core in strict confinement environment
_description:
Afrer suspend test mediacard in strict confinement mode.
Rely on test-strict-confinement SNAP to test.
Ubuntu Core QA test plan for strict confinement environment.
This test plan contains all of the tests that require manual
control of device hardware or some other user input to complete.
estimated_duration: 3600
include:
after-suspend-strict-confine/mediacard/sdhc-insert
after-suspend-strict-confine/mediacard/sdhc-storage
after-suspend-strict-confine/mediacard/sdhc-remove

nested_part:
after-suspend-strict-confinement-mediacard-manual
exclude:

id: strict-confine-thermal
id: after-suspend-strict-confinement-automated
unit: test plan
_name: Test thermal sensor in strict confinement mode.
_desription:
Test thermal sensor in strict confinement mode.
Rely on test-strict-confinment SNAP to test.
bootstrap_include:
thermal_zones
include:
strict-confine/temperature-test


id: strict-confine-timedatectl
unit: test plan
_name: Test timedatectl in strict confinement mode.
_desription:
Test timedatectl command in strcit confinement mode.
Rely on test-strict-confinment SNAP to test.
bootstrap_include:
include:
strict-confinement/timedatectl-timezone
strict-confinement/timedatectl-ntp


id: after-suspend-strict-confine-timedatectl
unit: test plan
_name: After suspend test timedatectl in strict confinement mode.
_desription:
After suspend test timedatectl command in strcit confinement mode.
Rely on test-strict-confinment SNAP to test.
bootstrap_include:
_name: Strict-confinement - After suspend Automated only QA tests for Ubuntu Core in strict confinement environment
_description:
Ubuntu Core QA test plan for the strict confinement environment.
This test plan contains all of the automated tests used to validate
the Ubuntu Core in strict confinement environment.
include:
after-suspend-strict-confinement/timedatectl-timezone
after-suspend-strict-confinement/timedatectl-ntp
nested_part:
after-suspend-strict-confinement-timedatectl-automated
after-suspend-strict-confinement-thermal-automated
exclude:
3 changes: 3 additions & 0 deletions ...nement/timedatectl-strict-confinement.pxu → ...s/strict-confinement/timedatectl/jobs.pxu
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@ imports:
requires:
lsb.distributor_id == 'Ubuntu Core'
snap.name == 'test-strict-confinement'
connections.slot == "snapd:timeserver-control" and connections.plug == "test-strict-confinement:timeserver-control"
connections.slot == "snapd:time-control" and connections.plug == "test-strict-confinement:time-control"
connections.slot == "snapd:timezone-control" and connections.plug == "test-strict-confinement:timezone-control"
command:
test-strict-confinement.timedatectl-timezone
Expand All @@ -40,5 +42,6 @@ requires:
snap.name == 'test-strict-confinement'
connections.slot == "snapd:timeserver-control" and connections.plug == "test-strict-confinement:timeserver-control"
connections.slot == "snapd:time-control" and connections.plug == "test-strict-confinement:time-control"
connections.slot == "snapd:timezone-control" and connections.plug == "test-strict-confinement:timezone-control"
command:
test-strict-confinement.timedatectl-ntp
22 changes: 22 additions & 0 deletions ...eckbox-ce-oem/checkbox-provider-ce-oem/units/strict-confinement/timedatectl/test-plan.pxu
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
id: strict-confinement-timedatectl-automated
unit: test plan
_name: Test timedatectl in strict confinement mode.
_desription:
Test timedatectl command in strcit confinement mode.
Rely on test-strict-confinment SNAP to test.
bootstrap_include:
include:
strict-confinement/timedatectl-timezone
strict-confinement/timedatectl-ntp


id: after-suspend-strict-confinement-timedatectl-automated
unit: test plan
_name: After suspend test timedatectl in strict confinement mode.
_desription:
After suspend test timedatectl command in strcit confinement mode.
Rely on test-strict-confinment SNAP to test.
bootstrap_include:
include:
after-suspend-strict-confinement/timedatectl-timezone
after-suspend-strict-confinement/timedatectl-ntp