Conversation
The snap-refresh-*-from-[stable|base]-rev jobs are executed if the revision of the related snap (gadget, kernel, snapd) currently installed on the device is different from the revision in base version and/or stable channel. It is better to provide an option to decide whether or not these tests should be executed, per device. This is achieved using three manifest entries (one per snap type). If these entries are explicitely set to True in the manifest, they will be executed (if they match the condition described above, of course). Otherwise, they will be skipped.
These jobs and test plans are not specific to Ubuntu Core and can be run on any device running snapd (classic, server, core and hybrid images). This commit will: - Move jobs and test plans from the the base provider's ubuntucore/ section to the snapd/ section - Adjust their names (rename prefixes from ubuntucore to snapd) - Adjust their categories (from ubuntucore to snapd)
Doing so ensure the snap refresh/revert tests are executed on every device running the SRU test plan (mostly classic images) as well as on devices running IoT-related test plans (mostly Ubuntu core images).
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #883 +/- ##
=======================================
Coverage 35.70% 35.70%
=======================================
Files 303 303
Lines 34250 34250
Branches 5917 5917
=======================================
Hits 12230 12230
Misses 21458 21458
Partials 562 562
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
yphus
approved these changes
Dec 11, 2023
yphus
left a comment
yphus
left a comment
There was a problem hiding this comment.
Great work with this new set of jobs.
Manifests will allow fine tuning of the snaps requiring such test coverage.
patliuu
approved these changes
Dec 13, 2023
Contributor
patliuu
left a comment
patliuu
left a comment
There was a problem hiding this comment.
Thanks for the whole work on revamping these tests.
Manifest provides flexibility so it won't impact the pre-GM tests. 👍️
binli
pushed a commit
to binli/checkbox
that referenced
this pull request
Mar 22, 2024
…es to control when to execute them (New) (canonical#883) Fix CHECKBOX-718 * Add manifest entries to test gadget/kernel/snapd snaps The snap-refresh-*-from-[stable|base]-rev jobs are executed if the revision of the related snap (gadget, kernel, snapd) currently installed on the device is different from the revision in base version and/or stable channel. It is better to provide an option to decide whether or not these tests should be executed, per device. This is achieved using three manifest entries (one per snap type). If these entries are explicitely set to True in the manifest, they will be executed (if they match the condition described above, of course). Otherwise, they will be skipped. * Move snap refresh/revert jobs and test plans in the snapd section These jobs and test plans are not specific to Ubuntu Core and can be run on any device running snapd (classic, server, core and hybrid images). This commit will: - Move jobs and test plans from the the base provider's ubuntucore/ section to the snapd/ section - Adjust their names (rename prefixes from ubuntucore to snapd) - Adjust their categories (from ubuntucore to snapd) * Add snap-refresh-revert nested part to SRU and snappy-snap-automated Doing so ensure the snap refresh/revert tests are executed on every device running the SRU test plan (mostly classic images) as well as on devices running IoT-related test plans (mostly Ubuntu core images). * Add README.md for the base provider's snapd section * Move the snap refresh/revert manifest to the snapd section * Add some explanation in snapd manifest file
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Following #805, this PR pushes things further (see below for a rundown).
Note: Although this test plan is added to the SRU and IoT-related test plans, it does not have any effect by default because it requires a manifest entry to be declared in the
machine-manifest.jsonfile (or manually input in the Manifest Entry screen if running tests manually). See newly createdbase/units/snapd/README.mdfile for more info.Add manifest entries to test gadget/kernel/snapd snaps
The
snap-refresh-*-from-[stable|base]-revjobs are executed if therevision of the related snap (gadget, kernel, snapd) currently installed
on the device is different from the revision in base version and/or
stable channel.
It is better to provide an option to decide whether or not these tests
should be executed, per device.
This is achieved using three manifest entries (one per snap type).
If these entries are explicitly set to
truein the manifest, they willbe executed (if they match the condition described above, of course).
Otherwise, they will be skipped.
Move snap refresh/revert jobs and test plans in the
snapdsectionThese jobs and test plans are not specific to Ubuntu Core and can be run
on any device running snapd (classic, server, core and hybrid images).
To make this clearer, we:
section to the snapd/ section
Add snap-refresh-revert nested part to SRU and
snappy-snap-automatedtest plansDoing so ensure the snap refresh/revert tests are executed on every
device running the SRU test plan (mostly classic images) as well as on
devices running IoT-related test plans (mostly Ubuntu core images).
Add documentation about all of this in the base provider's
snapdsectionA
README.mdfile is added in base/units/snapd/ to provide more explanation about this test plan and the jobs it contains.Resolved issues
CHECKBOX-718
Documentation
See
base/units/snapd/README.md.Tests
Tested locally using a virtual environment on 22.04 desktop.
Resource
Since 22.04 desktop has
snapdinstalled but no gadget nor kernel snaps, the resource job only returns data forsnapd:✔️
Manifest entries
checkbox-clicom.canonical.certification::snap-refresh-revert)T(X) Noand pressTto launch the test run→ after a short while, the jobs re-run screen is show because all of the jobs have been skipped due to "failed dependencies". ✔️
Fto finish. In the text summary, you can see:which is expected since we selected "No" in the manifest screen. ✔️
Run the same steps, but select "Yes" in the manifest:
Inclusion in SRU and IoT test plans
The
snap-refresh-reverttest plan is found by Checkbox and contains the expected jobs:Output of the snap-refresh-revert test plan on a 22.04 desktop
These jobs are also present in the SRU test plan since the
snap-refresh-revertis now nested in it:Selective output of the SRU test plan
And the jobs are also available in every IoT related test plans, since they are nested in
snappy-snap-automatedwhich is present in every IoT test plans:Partial output from the client-cert-iot-ubuntucore-16 test plan
Partial output from the client-cert-iot-ubuntucore-22 test plan
Partial output from the client-cert-iot-server-22-04 test plan