Skip to content

Commit ee51ad5

Browse files
pieqqbinli
pieqq
authored and
binli
committed
Add snap refresh/revert jobs to SRU test plans and add manifest entries to control when to execute them (New) (canonical#883)
Fix CHECKBOX-718 * Add manifest entries to test gadget/kernel/snapd snaps The snap-refresh-*-from-[stable|base]-rev jobs are executed if the revision of the related snap (gadget, kernel, snapd) currently installed on the device is different from the revision in base version and/or stable channel. It is better to provide an option to decide whether or not these tests should be executed, per device. This is achieved using three manifest entries (one per snap type). If these entries are explicitely set to True in the manifest, they will be executed (if they match the condition described above, of course). Otherwise, they will be skipped. * Move snap refresh/revert jobs and test plans in the snapd section These jobs and test plans are not specific to Ubuntu Core and can be run on any device running snapd (classic, server, core and hybrid images). This commit will: - Move jobs and test plans from the the base provider's ubuntucore/ section to the snapd/ section - Adjust their names (rename prefixes from ubuntucore to snapd) - Adjust their categories (from ubuntucore to snapd) * Add snap-refresh-revert nested part to SRU and snappy-snap-automated Doing so ensure the snap refresh/revert tests are executed on every device running the SRU test plan (mostly classic images) as well as on devices running IoT-related test plans (mostly Ubuntu core images). * Add README.md for the base provider's snapd section * Move the snap refresh/revert manifest to the snapd section * Add some explanation in snapd manifest file
1 parent c752cef commit ee51ad5

7 files changed

Lines changed: 347 additions & 267 deletions

File tree

‎providers/base/units/snapd/README.md‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
# Snap Refresh/Revert Tests
2+
3+
## Rationale
4+
5+
Snapd offers the ability to refresh a given snap to a specific revision,
6+
and to revert to the previously installed one if needs be.
7+
8+
Moreover, Snapd has mechanisms in place to automatically refresh every
9+
snap to the latest revision available in their tracked channel. This is
10+
interesting for devices in the field, but also to automatically update a
11+
system that has just been setup, for instance.
12+
13+
In order to test these features, the `snapd` section of Checkbox base
14+
provider offers a test plan, `snap-refresh-revert`, that contains jobs doing
15+
the following:
16+
17+
1. Generate resource information for any gadget, kernel or snapd snap available
18+
on the system; then, if the currently installed revision is different from
19+
the targeted revision and if required (see "Manifest entries" section below):
20+
2. Refresh them again, but this time to their base revision (the revision
21+
that came pre-installed with the image)
22+
3. Revert them back to their original revision
23+
4. Refresh each of these snaps to the revision number from the stable channel
24+
5. Revert them back
25+
26+
After each of these steps, the device is rebooted, then Checkbox checks if
27+
the given snap has been updated to the expected revision before proceeding
28+
the next step.
29+
30+
## Manifest entries
31+
32+
For each of the snap types (gadget, kernel, snapd), a manifest entry controls
33+
whether or not the tests should be executed. If the manifest entries are
34+
not defined, Checkbox will skip every job in the test plan by default. To
35+
enable these tests, the following manifest entries can be set to `true`:
36+
37+
- `need_gadget_snap_update_test`
38+
- `need_kernel_snap_update_test`
39+
- `need_snapd_snap_update_test`
40+
41+
See the [Manifest Entry] section of Checkbox documentation for more
42+
information.
43+
44+
This test plan is aimed at being used during the Snap Update Verification
45+
(SUV) process or the Stable Release Update (SRU) process.
46+
47+
During SUV in particular, the snap that is installed before running the
48+
test plan is from the beta (or the candidate) channel, so it should always
49+
be different from the version in the stable channel or the base version
50+
present on the system.
51+
52+
[Manifest Entry]: https://checkbox.readthedocs.io/en/stable/reference/units/manifest-entry.html

‎providers/base/units/snapd/jobs.pxu‎

Lines changed: 207 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,207 @@
1+
id: snap_revision_info
2+
plugin: resource
3+
_summary:
4+
Gather name, tracking, and revision info of the kernel, snapd and gadget snaps
5+
estimated_duration: 3s
6+
category_id: snapd
7+
command:
8+
snap_update_test.py --resource
9+
10+
unit: template
11+
template-resource: snap_revision_info
12+
template-unit: job
13+
id: snapd/snap-refresh-{type}-{name}-to-stable-rev
14+
_summary: Refresh {name} snap to latest revision in stable channel
15+
_description:
16+
This test is currently for SUV process, the snap to be tested is on the beta
17+
channel. Test the availability to refresh to the older version on the stable
18+
channel. This test will be excuted only when the current installed revision
19+
(on beta channel) is different from the revsion in stable channel.
20+
plugin: shell
21+
estimated_duration: 1m
22+
category_id: snapd
23+
user: root
24+
imports:
25+
from com.canonical.certification import snap_revision_info
26+
from com.canonical.plainbox import manifest
27+
requires:
28+
(snap_revision_info.name == "{name}") and snap_revision_info.stable_rev != snap_revision_info.original_installed_rev
29+
manifest.need_{type}_snap_update_test == "True"
30+
command:
31+
path="$PLAINBOX_SESSION_SHARE/{name}_snap_revision_info"
32+
snap_update_test.py --refresh --revision {stable_rev} --info-path "$path" {name}
33+
34+
unit: template
35+
template-resource: snap_revision_info
36+
template-unit: job
37+
id: snapd/reboot-after-snap-refresh-{type}-{name}-to-stable-rev
38+
_summary: Reboot after {name} snap refresh to latest revision in stable channel
39+
plugin: shell
40+
flags: noreturn autorestart
41+
estimated_duration: 3m
42+
category_id: snapd
43+
user: root
44+
depends: snapd/snap-refresh-{type}-{name}-to-stable-rev
45+
command:
46+
echo "Waiting 90s for any snap operation to finish before rebooting..."
47+
sleep 90
48+
reboot
49+
50+
unit: template
51+
template-resource: snap_revision_info
52+
template-unit: job
53+
id: snapd/snap-verify-after-refresh-{type}-{name}-to-stable-rev
54+
_summary: Verify {name} snap revision after refreshing to latest revision in stable channel
55+
plugin: shell
56+
estimated_duration: 30s
57+
category_id: snapd
58+
user: root
59+
depends: snapd/reboot-after-snap-refresh-{type}-{name}-to-stable-rev
60+
command:
61+
path="$PLAINBOX_SESSION_SHARE/{name}_snap_revision_info"
62+
snap_update_test.py --verify-refresh --info-path "$path" {name}
63+
64+
unit: template
65+
template-resource: snap_revision_info
66+
template-unit: job
67+
id: snapd/snap-revert-{type}-{name}-from-stable-rev
68+
_summary: Revert {name} snap to original revision from stable channel
69+
plugin: shell
70+
estimated_duration: 3m
71+
category_id: snapd
72+
user: root
73+
depends: snapd/snap-verify-after-refresh-{type}-{name}-to-stable-rev
74+
command:
75+
path="$PLAINBOX_SESSION_SHARE/{name}_snap_revision_info"
76+
snap_update_test.py --revert --info-path "$path" {name}
77+
78+
unit: template
79+
template-resource: snap_revision_info
80+
template-unit: job
81+
id: snapd/reboot-after-snap-revert-{type}-{name}-from-stable-rev
82+
_summary: Reboot after {name} snap reverting to latest revision in stable channel
83+
plugin: shell
84+
flags: noreturn autorestart
85+
estimated_duration: 3m
86+
category_id: snapd
87+
user: root
88+
depends: snapd/snap-revert-{type}-{name}-from-stable-rev
89+
command:
90+
echo "Waiting 90s for any snap operation to finish before rebooting..."
91+
sleep 90
92+
reboot
93+
94+
unit: template
95+
template-resource: snap_revision_info
96+
template-unit: job
97+
id: snapd/snap-verify-after-revert-{type}-{name}-from-stable-rev
98+
_summary: Verify {name} snap revision after reverting from stable revision
99+
plugin: shell
100+
estimated_duration: 3s
101+
category_id: snapd
102+
user: root
103+
depends: snapd/reboot-after-snap-revert-{type}-{name}-from-stable-rev
104+
command:
105+
path="$PLAINBOX_SESSION_SHARE/{name}_snap_revision_info"
106+
snap_update_test.py --verify-revert --info-path "$path" {name}
107+
108+
unit: template
109+
template-resource: snap_revision_info
110+
template-unit: job
111+
id: snapd/snap-refresh-{type}-{name}-to-base-rev
112+
_summary: Refresh {name} snap to its base revision
113+
_description:
114+
This test is currently for SUV process, the snap to be tested is on the beta
115+
channel. Test the availability to refresh to the base revision (the revision
116+
that came with the image). This test will be excuted only when the current
117+
installed revision (on beta channel) is different from the base revision.
118+
This job (and the jobs depending on it) are useful when the device cannot
119+
be reprovisioned easily, but you still want to simulate a fresh install
120+
that would bump to the latest revision available.
121+
plugin: shell
122+
estimated_duration: 3m
123+
category_id: snapd
124+
user: root
125+
imports:
126+
from com.canonical.certification import snap_revision_info
127+
from com.canonical.plainbox import manifest
128+
requires:
129+
(snap_revision_info.name == "{name}") and snap_revision_info.base_rev != snap_revision_info.original_installed_rev
130+
manifest.need_{type}_snap_update_test == "True"
131+
command:
132+
path="$PLAINBOX_SESSION_SHARE/{name}_snap_revision_info"
133+
snap_update_test.py --refresh --revision {base_rev} --info-path "$path" {name}
134+
135+
unit: template
136+
template-resource: snap_revision_info
137+
template-unit: job
138+
id: snapd/reboot-after-snap-refresh-{type}-{name}-to-base-rev
139+
_summary: Reboot after {name} snap refresh to base revision
140+
plugin: shell
141+
flags: noreturn autorestart
142+
estimated_duration: 3m
143+
category_id: snapd
144+
user: root
145+
depends: snapd/snap-refresh-{type}-{name}-to-base-rev
146+
command:
147+
echo "Waiting 90s for any snap operation to finish before rebooting..."
148+
sleep 90
149+
reboot
150+
151+
unit: template
152+
template-resource: snap_revision_info
153+
template-unit: job
154+
id: snapd/snap-verify-after-refresh-{type}-{name}-to-base-rev
155+
_summary: Verify {name} snap revision after refreshing to base revision
156+
plugin: shell
157+
estimated_duration: 30s
158+
category_id: snapd
159+
user: root
160+
depends: snapd/reboot-after-snap-refresh-{type}-{name}-to-base-rev
161+
command:
162+
path="$PLAINBOX_SESSION_SHARE/{name}_snap_revision_info"
163+
snap_update_test.py --verify-refresh --info-path "$path" {name}
164+
165+
unit: template
166+
template-resource: snap_revision_info
167+
template-unit: job
168+
id: snapd/snap-revert-{type}-{name}-from-base-rev
169+
_summary: Revert {name} snap from base revision to original revision
170+
plugin: shell
171+
estimated_duration: 3m
172+
category_id: snapd
173+
user: root
174+
depends: snapd/snap-verify-after-refresh-{type}-{name}-to-base-rev
175+
command:
176+
path="$PLAINBOX_SESSION_SHARE/{name}_snap_revision_info"
177+
snap_update_test.py --revert --info-path "$path" {name}
178+
179+
unit: template
180+
template-resource: snap_revision_info
181+
template-unit: job
182+
id: snapd/reboot-after-snap-revert-{type}-{name}-from-base-rev
183+
_summary: Reboot after {name} snap revert to base revision
184+
plugin: shell
185+
flags: noreturn autorestart
186+
estimated_duration: 3m
187+
category_id: snapd
188+
user: root
189+
depends: snapd/snap-revert-{type}-{name}-from-base-rev
190+
command:
191+
echo "Waiting 90s for any snap operation to finish before rebooting..."
192+
sleep 90
193+
reboot
194+
195+
unit: template
196+
template-resource: snap_revision_info
197+
template-unit: job
198+
id: snapd/snap-verify-after-revert-{type}-{name}-from-base-rev
199+
_summary: Verify {name} snap revision after reverting from base revision
200+
plugin: shell
201+
estimated_duration: 3s
202+
category_id: snapd
203+
user: root
204+
depends: snapd/reboot-after-snap-revert-{type}-{name}-from-base-rev
205+
command:
206+
path="$PLAINBOX_SESSION_SHARE/{name}_snap_revision_info"
207+
snap_update_test.py --verify-revert --info-path "$path" {name}

‎providers/base/units/snapd/manifest.pxu‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
2+
# If the following manifest entries are set to true, SRU and Cert IoT test
3+
# plans will try to run snap refresh/revert to and from the base revision and
4+
# the stable revision. See README.md for more information.
5+
6+
unit: manifest entry
7+
id: need_kernel_snap_update_test
8+
_prompt: Do the following snaps types require snap refresh/revert testing?
9+
_name: Kernel
10+
value-type: bool
11+
12+
unit: manifest entry
13+
id: need_snapd_snap_update_test
14+
_prompt: Do the following snaps types require snap refresh/revert testing?
15+
_name: Snapd
16+
value-type: bool
17+
18+
unit: manifest entry
19+
id: need_gadget_snap_update_test
20+
_prompt: Do the following snaps types require snap refresh/revert testing?
21+
_name: Gadget
22+
value-type: bool

‎providers/base/units/snapd/test-plan.pxu‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -70,6 +70,8 @@ include:
7070
snappy/test-store-config-.*
7171
snappy/test-system-confinement
7272
snappy/test-snaps-confinement
73+
nested_part:
74+
snap-refresh-revert
7375
mandatory_include:
7476
snap
7577
bootstrap_include:
@@ -86,3 +88,66 @@ exclude:
8688
snappy/test-store.*
8789
nested_part:
8890
snappy-snap-automated
91+
92+
# See README.md file for more information about this test plan
93+
id: snap-refresh-revert
94+
unit: test plan
95+
_name: Snaps refresh and revert automated tests
96+
_description:
97+
For each of the kernel/snapd/gadget snap, this test plan will:
98+
- simulate fresh install snap refresh by first refreshing to the base revision
99+
of the kernel snap
100+
- revert to the version used (usually revision from beta channel if used in
101+
the snap-update-verification process)
102+
- refresh to the previous stable version
103+
- revert to version used
104+
This test plan is useful for devices that cannot be provisioned easily in the
105+
lab.
106+
bootstrap_include:
107+
snap_revision_info
108+
include:
109+
######################
110+
# Gadget-related tests
111+
######################
112+
snapd/snap-refresh-gadget-.*-base-rev
113+
snapd/reboot-after-snap-refresh-gadget-.*-base-rev
114+
snapd/snap-verify-after-refresh-gadget-.*-base-rev
115+
snapd/snap-revert-gadget-.*-base-rev
116+
snapd/reboot-after-snap-revert-gadget-.*-base-rev
117+
snapd/snap-verify-after-revert-gadget-.*-base-rev
118+
snapd/snap-refresh-gadget-.*-stable-rev
119+
snapd/reboot-after-snap-refresh-gadget-.*-stable-rev
120+
snapd/snap-verify-after-refresh-gadget-.*-stable-rev
121+
snapd/snap-revert-gadget-.*-stable-rev
122+
snapd/reboot-after-snap-revert-gadget-.*-stable-rev
123+
snapd/snap-verify-after-revert-gadget-.*-stable-rev
124+
#####################
125+
# Snapd-related tests
126+
#####################
127+
snapd/snap-refresh-snapd-.*-base-rev
128+
snapd/reboot-after-snap-refresh-snapd-.*-base-rev
129+
snapd/snap-verify-after-refresh-snapd-.*-base-rev
130+
snapd/snap-revert-snapd-.*-base-rev
131+
snapd/reboot-after-snap-revert-snapd-.*-base-rev
132+
snapd/snap-verify-after-revert-snapd-.*-base-rev
133+
snapd/snap-refresh-snapd-.*-stable-rev
134+
snapd/reboot-after-snap-refresh-snapd-.*-stable-rev
135+
snapd/snap-verify-after-refresh-snapd-.*-stable-rev
136+
snapd/snap-revert-snapd-.*-stable-rev
137+
snapd/reboot-after-snap-revert-snapd-.*-stable-rev
138+
snapd/snap-verify-after-revert-snapd-.*-stable-rev
139+
######################
140+
# Kernel-related tests
141+
######################
142+
snapd/snap-refresh-kernel-.*-base-rev
143+
snapd/reboot-after-snap-refresh-kernel-.*-base-rev
144+
snapd/snap-verify-after-refresh-kernel-.*-base-rev
145+
snapd/snap-revert-kernel-.*-base-rev
146+
snapd/reboot-after-snap-revert-kernel-.*-base-rev
147+
snapd/snap-verify-after-revert-kernel-.*-base-rev
148+
snapd/snap-refresh-kernel-.*-stable-rev
149+
snapd/reboot-after-snap-refresh-kernel-.*-stable-rev
150+
snapd/snap-verify-after-refresh-kernel-.*-stable-rev
151+
snapd/snap-revert-kernel-.*-stable-rev
152+
snapd/reboot-after-snap-revert-kernel-.*-stable-rev
153+
snapd/snap-verify-after-revert-kernel-.*-stable-rev

0 commit comments

Comments
 (0)