Skip to content

Replaced DSS shell scripts with Python and added running DSS integration tests (New)#1724

Closed
motjuste wants to merge 54 commits intomainfrom
CHECKBOX-1693-refactor-dss-sh-tests-to-python
Closed

Replaced DSS shell scripts with Python and added running DSS integration tests (New)#1724
motjuste wants to merge 54 commits intomainfrom
CHECKBOX-1693-refactor-dss-sh-tests-to-python

Conversation

@motjuste
Copy link
Copy Markdown
Contributor

@motjuste motjuste commented Feb 14, 2025

Description

  • The main change is replacing all jobs' shell scripts for DSS provider with Python scripts, alongside the tests for the Python scripts.
  • A new GH workflow has been added to run tests against these Python scripts using the added tox.ini. The tests can also be run locally following recommendations added to the README.
  • The existing GH workflow to test different DSS and microk8s versions has been updated.
    • Update DSS snap channels to reflect what's in the store.
    • Use the more general queues in Testflinger based on device type, instead of an exact device, to have more parallel runs, helping against flakiness of devices or Testflinger itself.
  • 3 new jobs have been added that purge DSS, run integration tests from the DSS repo, and optionally run GPU integration tests from the DSS repo.
  • A few jobs, specifically related to verifying the Intel GPU plugin have been removed. These were not reliable tests (specifically, in counting GPUs, as they also counted NVIDIA GPUs). These tests were, arguably, not related to testing DSS itself.

Resolved issues

Documentation

No changes to Checkbox documentation.

Tests

NOTE: all links to Testflinger require VPN.

The testing GH workflow has been added in this PR. These were run using Testflinger, with the coverage report available at the bottom of the full report here.

A run of the GH workflow to test DSS with this provider was started here. Unfortunately, this workflow is not reliable to always succeed, failing during provisioning from Testflinger, failing due to bugs in DSS, etc.

A manual run was submitted to Testflinger testing DSS 1/stable (which has the latest changes) against microk8s 1.28/stable, and its report can be found here.

remove a test case of verifying dss namespace is deployed.
that will be done using kubectl.  it is covered by the
upcoming integration tests from dss as well.
renamed some notebooks to be explicitly checkbox related
now we rely on kubectl so that canonical-k8s can be tested in the future
@motjuste motjuste changed the title Replaced DSS .sh tests to Python and added running DSS integration tests (New) Replaced DSS shell scripts with Python and added running DSS integration tests (New) Feb 14, 2025
@motjuste motjuste marked this pull request as ready for review February 14, 2025 13:24
@motjuste motjuste requested a review from a team as a code owner February 14, 2025 13:24
@motjuste motjuste requested a review from pieqq February 14, 2025 13:24
@fernando79513
Copy link
Copy Markdown
Collaborator

Please, Abdullah, try to split this PR into smaller ones.
It is quite hard to review such big PR while maintaining the quality in the review.

Since you have basically have 5 new main scripts, try to have a PR for each one of them, and maybe another one for creating the new workflow.

@fernando79513 fernando79513 added the waiting-for-changes The review has been completed but the PR is waiting for changes from the author label Feb 14, 2025
@motjuste
Copy link
Copy Markdown
Contributor Author

I understand your complaint, and would like to apologise. I have created a new PR #1725 with the first important pieces. Please have a look. More PRs will follow after that one is merged. Closing this for the moment then.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

waiting-for-changes The review has been completed but the PR is waiting for changes from the author

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants