diff --git a/docs/_images/checkbox-snappy-1-test-plan.png b/docs/_images/checkbox-snappy-1-test-plan.png deleted file mode 100644 index 85ecd29068..0000000000 Binary files a/docs/_images/checkbox-snappy-1-test-plan.png and /dev/null differ diff --git a/docs/_images/checkbox-snappy-2-resume-session.png b/docs/_images/checkbox-snappy-2-resume-session.png deleted file mode 100644 index 15472450ae..0000000000 Binary files a/docs/_images/checkbox-snappy-2-resume-session.png and /dev/null differ diff --git a/docs/_images/checkbox-snappy-3-select-jobs.png b/docs/_images/checkbox-snappy-3-select-jobs.png deleted file mode 100644 index d54d59cb3d..0000000000 Binary files a/docs/_images/checkbox-snappy-3-select-jobs.png and /dev/null differ diff --git a/docs/_images/checkbox-snappy-4-user-interact-job.png b/docs/_images/checkbox-snappy-4-user-interact-job.png deleted file mode 100644 index 594893c8b8..0000000000 Binary files a/docs/_images/checkbox-snappy-4-user-interact-job.png and /dev/null differ diff --git a/docs/_images/checkbox-snappy-5-rerun-jobs.png b/docs/_images/checkbox-snappy-5-rerun-jobs.png deleted file mode 100644 index f2816a3aa1..0000000000 Binary files a/docs/_images/checkbox-snappy-5-rerun-jobs.png and /dev/null differ diff --git a/docs/_images/checkbox-snappy-6-test-results.png b/docs/_images/checkbox-snappy-6-test-results.png deleted file mode 100644 index 61057faf47..0000000000 Binary files a/docs/_images/checkbox-snappy-6-test-results.png and /dev/null differ diff --git a/docs/how-to/index.rst b/docs/how-to/index.rst index f1175fca8d..c3fc9ae4a7 100644 --- a/docs/how-to/index.rst +++ b/docs/how-to/index.rst @@ -7,7 +7,6 @@ These how-to guides cover key operations and processes in Checkbox. :maxdepth: 1 :glob: - testing-ubuntu-core custom-app nested-test-plan side-loading diff --git a/docs/how-to/testing-ubuntu-core.rst b/docs/how-to/testing-ubuntu-core.rst deleted file mode 100644 index 869d9eb6e9..0000000000 --- a/docs/how-to/testing-ubuntu-core.rst +++ /dev/null @@ -1,115 +0,0 @@ -.. _testing-snappy: - -Running Checkbox on Ubuntu Core -=============================== - - -Introduction ------------- - -Checkbox is a hardware testing tool developed by Canonical for certifying -hardware with Ubuntu. Checkbox is free software and is available at -https://launchpad.net/checkbox-project. - -To support the release of devices running snappy Ubuntu Core, Canonical has -produced versions of Checkbox tailored specifically for these systems. - -This document aims to provide the reader with enough information to install and -run Checkbox on an Ubuntu Core system, and how to view/interpret/submit test -results. - - -Installation ------------- - -Installing Ubuntu Core -`````````````````````` - -You can try out Ubuntu Core on a range of devices or in a virtual machine. For -downloads and instructions see this `page `. - -Installing Checkbox Snap -```````````````````````` - -Now you are ready to install the Checkbox snap your Ubuntu Core device. It can -be found in the Ubuntu store and there are versions targeting each Ubuntu Core -series. For Series 20:: - - $ snap install checkbox-snappy --devmode --channel=20/stable - -For Ubuntu Core Series 18:: - - $ snap install checkbox-snappy --devmode --channel=18/stable - -For Ubuntu Core Series 16:: - - $ snap install checkbox-snappy --devmode --channel=16/stable - -Running Checkbox ----------------- - -Launch Checkbox using:: - - $ checkbox-snappy.test-runner - -.. image:: ../_images/checkbox-snappy-1-test-plan.png - -Checkbox keeps track of previous test runs, so if a session is not completed, -you’ll be asked to resume your previous run or create a new session: - -.. image:: ../_images/checkbox-snappy-2-resume-session.png - -The first selection screen will ask you to select a test plan to run: - -.. image:: ../_images/checkbox-snappy-1-test-plan.png - -Move the selection with the arrow keys, select with ``Space`` and confirm your -choice by pressing ``Enter``. The next screen will allow you to fine tune the -tests you want to run: - -.. image:: ../_images/checkbox-snappy-3-select-jobs.png - -Tests are grouped by categories. Expand/collapse with ``Enter``, select/unselect -with ``Space`` (also works on categories). Press ``S`` to select all and ``D`` to -deselect all the tests. Press ``H`` to display a help screen with more keyboard -shortcuts. - -Start the tests by pressing ``T``. - -Checkbox is a test runner able to process fully automated tests/commands and -tests requiring user interaction (whether to setup or plug something to the -device, e.g. USB insertion or to confirm that the device acts as expected, e.g. -a led blinks). - -Please refer to the Checkbox documentation to learn more about the supported -types of tests. - -A fully automated test will stream stdout/stderr to your terminal allowing you -to immediately look at the I/O logs (if the session is run interactively). -Attachments jobs are treated differently as they could generate lots of I/O. -Therefore their outputs are hidden by default. - -Interactive jobs will pause the test runner and detail the steps to complete -the test: - -.. image:: ../_images/checkbox-snappy-4-user-interact-job.png - - -Getting Results ---------------- - -When the test selection has been run, the first displayed screen will allow you -to re-run failed jobs: - -.. image:: ../_images/checkbox-snappy-5-rerun-jobs.png - -Commands to select the tests to rerun are the same used to select tests in the -first selection screen. Here you can re-run your selection with ``R`` or finish -the session by pressing ``F``. - -Checkbox will then print the test results in the terminal and save them in -different formats locally on the device (and print their respective filenames): - -.. image:: ../_images/checkbox-snappy-6-test-results.png - -The resulting reports can be pulled from the system via ``scp`` for instance. diff --git a/docs/reference/units/category.rst b/docs/reference/units/category.rst index ac93477fc5..1ee45a0316 100644 --- a/docs/reference/units/category.rst +++ b/docs/reference/units/category.rst @@ -1,3 +1,5 @@ +.. _category_unit: + ============= Category Unit ============= diff --git a/docs/tutorial/index.rst b/docs/tutorial/index.rst index 0f45fb5a81..de53f9526a 100644 --- a/docs/tutorial/index.rst +++ b/docs/tutorial/index.rst @@ -44,7 +44,7 @@ Once you have completed the core tutorial, the extended optional tutorial sections can be followed in any order - they don’t depend on each other. .. toctree:: - :maxdepth: 1 + :maxdepth: 1 writing-tests/test-case writing-tests/test-plan diff --git a/docs/tutorial/using-checkbox/installing-checkbox.rst b/docs/tutorial/using-checkbox/installing-checkbox.rst index cae9e19f49..0005b9ecf5 100644 --- a/docs/tutorial/using-checkbox/installing-checkbox.rst +++ b/docs/tutorial/using-checkbox/installing-checkbox.rst @@ -23,12 +23,27 @@ run the following:: [...] checkbox (22.04/stable) X.Y.Z from Canonical Certification Team (ce-certification-qa) installed +If you are on Ubuntu Core, then you can't install the classic frontend. +You should use the following strict frontend instead:: + + $ sudo snap install checkbox --channel=uc22/stable --devmode + [...] + checkbox (22.04/stable) X.Y.Z from Canonical Certification Team (ce-certification-qa) installed + .. note:: There are multiple frontends as you may discover by typing ``snap info checkbox``. If you are unsure about what ``frontend`` you should use, consider reading this page: :ref:`ref_which_snap`, but for the scope of this tutorial the one installed in this snipped is enough. +.. warning:: + Checkbox will automatically start a agent service on your device when you + install it. To follow this tutorial you will need this service but you + should consider stopping it whenever you don't need it as it allows full + unauthenticated root level remote control of your machine! To disable it run + ``snap stop --disable checkbox``. To start it once you need it run + ``snap start checkbox``. + Now that we have installed both we can launch Checkbox running: .. code-block:: none diff --git a/docs/tutorial/using-checkbox/remote.rst b/docs/tutorial/using-checkbox/remote.rst index 5fc34449b4..ed85410b99 100644 --- a/docs/tutorial/using-checkbox/remote.rst +++ b/docs/tutorial/using-checkbox/remote.rst @@ -20,33 +20,29 @@ Run the following command: .. code-block:: none - systemctl status snap.checkbox.agent.service + snap services checkbox You should see something like this: .. code-block:: none - ● snap.checkbox.agent.service - Service for snap application checkbox.service - Loaded: loaded (/etc/systemd/system/snap.checkbox.service.service; enabled; vendor preset: enabled) - Active: active (running) since Fri 2023-07-21 13:38:48 CST; 1h 29min ago - Main PID: 1411 (python3) - Tasks: 1 (limit: 19014) - Memory: 69.7M - CPU: 2.537s - CGroup: /system.slice/snap.checkbox.agent.service - └─1411 python3 /snap/checkbox22/current/bin/checkbox-cli run-agent - - Jul 21 13:38:48 coltrane systemd[1]: Started Service for snap application checkbox.service. - (...) + Service Startup Current Notes + checkbox.agent disabled inactive - When you install Checkbox on a device, a Systemd service is started to turn -this device into a Checkbox agent. - -For the sake of this tutorial, let's stop this service for the moment: +the device into a Checkbox agent. We have instructed you to disable this +service in :ref:`installing_checkbox` but in case you didn't do so, run the +following: .. code-block:: none - sudo systemctl stop snap.checkbox.agent.service + snap stop --disable checkbox + +.. warning:: + Remember that while the agent is running, the device is fully controllable + via the network at a root level without any authentication! This is the + purpose of Checkbox, but a bit too scary to keep this enabled on your + personal device. Now, open two terminal windows using ``Ctrl+Alt+T``. In the first one, start the Checkbox agent: diff --git a/docs/tutorial/writing-tests/test-case.rst b/docs/tutorial/writing-tests/test-case.rst index 0a088af32c..28d9f9fe53 100644 --- a/docs/tutorial/writing-tests/test-case.rst +++ b/docs/tutorial/writing-tests/test-case.rst @@ -1,4 +1,4 @@ -.. _test_case: +.. _adv_test_case: ================= Writing Test Jobs @@ -15,7 +15,7 @@ ideal for prototyping. To provision Checkbox from source do the following: .. code-block:: shell # first install python3 and python3-venv - $ sudo apt install python3 python3-venv python3-pip + $ sudo apt install python3 python3-virtualenv python3-pip # clone the Checkbox repository $ git clone https://github.com/canonical/checkbox.git # call the mk-venv script with the location of your virtualenv @@ -37,6 +37,12 @@ ideal for prototyping. To provision Checkbox from source do the following: Remember to activate the virtual environment! You can also create an alias in your ``~/.bashrc`` to enable it when you need it. +.. note:: + All of the commands in this tutorial are using the + ``com.canonical.certification`` namespace. If you want to continue the one you + have started before, remember to change the namespace + (i.e. ``2024.com.tutorial::tutorial``) in the commands as well! + Creating a new provider ======================= @@ -45,7 +51,7 @@ Checkbox organizes and manages all jobs, test plans and other test units in vari Let's create a new Checkbox provider by using the Checkbox sub-command ``startprovider``. -.. code-block:: shell +.. code-block:: none (checkbox_venv) $ checkbox-cli startprovider 2024.com.tutorial:tutorial @@ -105,6 +111,13 @@ Now to run our test we can use the ``run`` sub-command. Try the following: ☑ : A job that always passes +.. important:: + You should always run ``python3 manage.py validate`` before running your + tests. This ensures that your unit is valid and Checkbox will interpret it + correctly. When you don't do it, Checkbox may ignore some errors in your + unit and, for example, fail to load some jobs leaving you wondering why the + ``run`` command doesn't work! + First concrete test example =========================== @@ -151,12 +164,44 @@ like this: ==================================[ Results ]=================================== ☑ : Test that the internet is reachable +Similarly to ``summary`` and ``id``, consider also providing a ``category`` for +your tests. It makes the output easier to read and clearer (you can see that we +are getting an ``uncategorised`` category right now). Additionally it is used +in the Test Selection screen to group your tests, so when you have many of +them, it makes sifting through them that much easier. You can create your own +category (See: :ref:`category_unit`), but consider using the built-in ones. + +To get a list of them, try the following: + +.. code-block:: none + + (checkbox_venv) $ checkbox-cli list "category" | grep "com.canonical.plainbox" + .. note:: - Similarly to ``summary`` and ``id``, consider also providing a ``category`` - for your tests. It makes the output easier to read and clearer (you can see - that we are getting an ``uncategorised`` category right now). Additionally - it is used in the Test Selection screen to group your tests, so when you - have many of them, it makes sifting through them that much easier. + We grep for ``com.canonical.plainbox`` categories because those are part of + Checkbox. If you use a category id that is not builtin, remember that doing + so adds a new dependency between your provider and the one that defines the + category unit you are using! + +We can use the ``com.canonical.plainbox::networking`` category for our tests by +modifying the unit as follows: + +.. code-block:: none + + id: network_available + flags: simple + _summary: Test that the internet is reachable + category_id: com.canonical.plainbox::networking + command: + ping -c 1 1.1.1.1 + +As any other unit that is not defined in the same namespace you are using, when +referring to it you have to use the full name, including the namespace! + +.. note:: + We will omit the ``category_id`` from all units in this tutorial to make the + snippets shorter but you should always use it for your production unit. + Dependencies ============ @@ -337,7 +382,8 @@ Create a new job with the following content: "\nlink_info_kind: " + .linkinfo.info_kind + "\nlink_type: " + .link_type + "\n"' -We are using ``jq`` to parse the output of the ``ip`` command, which means we need to make sure ``jq`` is available. We need to declare this in +We are using ``jq`` to parse the output of the ``ip`` command, which means we +need to make sure ``jq`` is available. We need to declare this in the correct spot, otherwise this will not work in a reproducible manner. Let's add a packaging meta-data unit to our ``units/extended_tutorial.pxu`` file: @@ -349,7 +395,10 @@ a packaging meta-data unit to our ``units/extended_tutorial.pxu`` file: Depends: jq -If you now run the following command you will notice a validation error. +If you have ``developed`` the other providers that Checkbox comes with, by +running the following command you will notice a validation error. If you don't +see this error, don't worry, it means you don't have the base provider +``installed`` or ``developed`` yet. .. code-block:: none diff --git a/docs/tutorial/writing-tests/test-plan.rst b/docs/tutorial/writing-tests/test-plan.rst index fd8ccde326..eeca207ffe 100644 --- a/docs/tutorial/writing-tests/test-plan.rst +++ b/docs/tutorial/writing-tests/test-plan.rst @@ -1,4 +1,4 @@ -.. _test_plan: +.. _adv_test_plan: =================== Writing A Test Plan @@ -9,6 +9,12 @@ connection on your machine. We will do this by re-using tests that are already available in the ``tutorial provider`` and that you got to write yourself in the previous tutorial. +.. note:: + All of the commands in this tutorial are using the + ``com.canonical.certification`` namespace. If you want to continue the one you + have started before, remember to change the namespace + (i.e. ``2024.com.tutorial::tutorial``) in the commands as well! + Inclusions ========== @@ -19,14 +25,14 @@ new test plan in the same provider we created in the previous tutorial. .. note:: - We generally advise to keep test plans ant test jobs in separate files, but + We generally advise to keep test plans and test jobs in separate files, but this is not compulsory. You can find this definition in ``providers/tutorial/units/test-plan.pxu`` We now have a convenient container where to put all tests we previously developed, let's include them in a new ``test plan``: -.. code-block:: +.. code-block:: none unit: test plan id: tutorial-extended @@ -38,7 +44,7 @@ developed, let's include them in a new ``test plan``: To run the test plan we can use ``run`` as we previously did for individual tests: -.. code-block:: +.. code-block:: none (checkbox_venv) $ checkbox-cli run com.canonical.certification::tutorial-extended [...] @@ -47,6 +53,10 @@ tests: ☑ : Test that the internet is reachable ☑ : Test that the network speed is acceptable +.. important:: + Remember to run ``python3 manage.py validate`` before trying your changes. + You will catch many mistakes that way. + Note how, as we previously saw, Checkbox automatically pulled the resource job needed. This operation, as we previously mentioned, is not the safe way to go about dependency management, it is just an aid Checkbox gives you while @@ -55,7 +65,7 @@ the list where you may already have broken the thing you are trying to fetch info about! With that being said, let's fix it before we forget: -.. code-block:: +.. code-block:: none unit: test plan id: tutorial-extended @@ -71,7 +81,7 @@ more advanced querying? That is where ``expand`` comes to our rescue. Run the following and see the result: -.. code-block:: +.. code-block:: none (checkbox_venv) $ checkbox-cli expand -f json com.canonical.certification::tutorial-extended | jq @@ -116,7 +126,7 @@ category (uncommon) of a job in that specific test plan. Going back to the test plan we just defined let's add the following and see the effect in the ``expand`` output: -.. code-block:: +.. code-block:: none unit: test plan id: tutorial-extended @@ -131,7 +141,7 @@ effect in the ``expand`` output: Running ``expand`` we can see that the certification status changed: -.. code-block:: +.. code-block:: none (checkbox_venv) $ checkbox-cli expand -f json com.canonical.certification::tutorial-extended | jq 'map({id: .id, "certification-status": .["certification-status"]})' @@ -175,7 +185,7 @@ gathering jobs. Let's go back to our test plan and move the resource job ``network_iface_info`` in the ``bootstrap_include`` section: -.. code-block:: +.. code-block:: none unit: test plan id: tutorial-extended @@ -198,7 +208,7 @@ actually pulled the resource automatically (the one that uses it as in the Let's update the test plan including it: -.. code-block:: +.. code-block:: none unit: test plan id: tutorial-extended @@ -225,7 +235,7 @@ output: want to see all the jobs that would be executed on the current machine if we ran that test plan, we can use ``list-bootstrapped``: -.. code-block:: +.. code-block:: none # Note: your output will be slightly different, depending on how many ifaces you have! (checkbox_venv) $ checkbox-cli list-bootstrapped com.canonical.certification::tutorial-extended @@ -253,7 +263,7 @@ is being developed for certification purposes, one nested part is compulsory to include (or the submissions will be rejected): ``submission-cert-automated``. Let's include it in our test plan: -.. code-block:: +.. code-block:: none :emphasize-lines: 10-12 unit: test plan @@ -266,10 +276,17 @@ Let's include it in our test plan: network_available network_speed certification-status=blocker nested_part: - submission-cert-automated + com.canonical.certification::submission-cert-automated certification_status_overrides: apply blocker to network_available +.. note:: + In your provider, you have to specify the full namespace to get access to + ``submission-cert-automated``. Also, if you didn't install it before, you + have to install the base provider, as there is where this test plan is + defined. As you did for your, run ``python3 manage.py develop`` while having + the virtual env active. + Another very useful thing you can do with nested parts is to create aliases. For example, if you were to rename a test plan in a provider that is used by others, it may be useful for everyone if you provide a backward compatible @@ -278,7 +295,7 @@ started publishing our tutorial test plan giving it the id ``tutorial-extended-oldid``. This is how we would create the backward compatible alias: -.. code-block:: +.. code-block:: none unit: test plan id: tutorial-extended-oldid @@ -305,7 +322,7 @@ them. If this is the case then ``exclusions`` are the way to go. For example, the ``network_speed`` test that we have in our test plan may be expensive to run, we can create a new test plan with it excluded as follows: -.. code-block:: +.. code-block:: none unit: test plan id: tutorial-extended-no-speed @@ -319,7 +336,7 @@ expensive to run, we can create a new test plan with it excluded as follows: Now if we ``list-bootstrapped`` the test plan we will see that the test is missing: -.. code-block:: +.. code-block:: none (checkbox_venv) $ checkbox-cli list-bootstrapped com.canonical.certification::tutorial-extended-no-speed [...jobs from submission-cert-automated...] @@ -361,7 +378,7 @@ included, it is not affected by exclude. To get an example, let's go back to our new test plan and try to exclude the ``info/systemd-analyze-critical-chain`` test: -.. code-block:: +.. code-block:: none unit: test plan id: tutorial-extended-no-speed @@ -374,7 +391,7 @@ To get an example, let's go back to our new test plan and try to exclude the See how the output of ``list-bootstrapped`` is unaffected. -.. code-block:: +.. code-block:: none (checkbox_venv) $ checkbox-cli list-bootstrapped com.canonical.certification::tutorial-extended-no-speed [...] diff --git a/providers/tutorial/units/extended_tutorial.pxu b/providers/tutorial/units/extended_tutorial.pxu index 758196b317..49c2b113c3 100644 --- a/providers/tutorial/units/extended_tutorial.pxu +++ b/providers/tutorial/units/extended_tutorial.pxu @@ -1,5 +1,6 @@ id: network_iface_info _summary: Fetches information of all network intefaces +category_id: com.canonical.plainbox::networking plugin: resource command: ip -details -json link show | jq -r ' @@ -11,6 +12,7 @@ command: id: network_available flags: simple _summary: Test that the internet is reachable +category_id: com.canonical.plainbox::networking requires: (network_iface_info.link_info_kind == "" and network_iface_info.link_type == "ether") command: @@ -18,6 +20,7 @@ command: id: network_speed flags: simple +category_id: com.canonical.plainbox::networking _summary: Test that the network speed is acceptable depends: network_available command: @@ -27,6 +30,7 @@ command: id: network_speed_99 flags: simple _summary: Test that the network speed is acceptable +category_id: com.canonical.plainbox::networking environ: ACCEPTABLE_BYTES_PER_SECOND_SPEED command: @@ -38,6 +42,7 @@ unit: template template-resource: network_iface_info template-unit: job id: network_available_{interface} +category_id: com.canonical.plainbox::networking template-id: network_available_interface template-filter: (network_iface_info.link_type == "ether" and network_iface_info.link_info_kind == "") @@ -49,6 +54,7 @@ _summary: Test that the internet is reachable via {interface} flags: simple id: vfork_memory_share +category_id: com.canonical.plainbox::memory _summary: Check that vfork syscall shares the memory between parent and child flags: simple command: