Migrate DSS GPU setup to setup-phase (New)

[motjuste]

Description

This PR migrates jobs related to GPU-setup from the normal test-plan in the DSS provider to the setup test-plan (which will eventually be run in the setup-phase of Checkbox). This includes installing NVIDIA GPU operator and or the Intel GPU plugin based on results from the newly added manifest entry jobs. Setting up the relevant GPU is implemented in the newly added k8s_gpu_setup.py script.

New manifest entries

Since we cannot use Checkbox resources during the setup-phase, two new manifest entry jobs have been added to the provider: has_intel_gpus and has_nvidia_gpus. The gpgpu provider was used as a reference for this.

These manifest entries are only used in the setup jobs. Normal test jobs still use resources to detect which GPUs are available. This is done to catch any mistakes made in the setup phase that didn't fail the setup jobs.

GPU Setup

NVIDIA GPU Operator is setup using Helm so that it can work with all Kubernetes clusters, but requires some special handling for Microk8s similar to how Microk8s' addon works.

Enabling Intel GPU plugin is done using kustomize.

Updates to Testflinger job

The GitHub workflow for running the DSS regression tests from this provider has been updated with:

• Choices to specify which version of the NVIDIA GPU Operator and Intel GPU Plugin to install.
• Default manifest files for each of the used devices, which are used when the appropriate manifest is not available on C3.

Resolved issues

CHECKBOX-1902 and CHECKBOX-1903

Documentation

No changes to Checkbox's documentation. Information about GPU setup has been added to the README of the provider.

Tests

Full run of the GitHub Workflow

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 52.02%. Comparing base (fec2b5c) to head (56b6459).
⚠️ Report is 104 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2066      +/-   ##
==========================================
+ Coverage   51.86%   52.02%   +0.16%     
==========================================
  Files         387      389       +2     
  Lines       41674    41819     +145     
  Branches     7741     7741              
==========================================
+ Hits        21613    21758     +145     
  Misses      19294    19294              
  Partials      767      767              

Flag	Coverage Δ
provider-dss	100.00% <100.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[wctaylor]

I don't have the full context for how everything works, so I won't approve this. That said, I do think everything looks good. I just have one nitpick

[fernando79513]

Good job here!
I'v just added a couple of small comments and answered in the variable renaming, but the rest looks nice.

[motjuste]

I have incorporated the feedback: mainly removing the unused global variables that were leftover by mistake from the previous implementation. The changes don't change anything functional, but I have triggered a new run just to confirm it.

[motjuste]

I had some bad luck with those triggered runs in the first few attempts, but re-running them recently succeeded. The earlier attempts had failed in fetching docker images for DSS notebooks, which were not related to the changes in this PR, but I chose to re-run the jobs again, and now they all pass.

[fernando79513]

LGTM +1!