Enable test customization

[forsyth2]

Summary

Objectives:

• Enable test customization by allowing the tester to choose which cfgs to run and which sets to run the image checker on. This will be quite useful when one test fails and we want to re-run it without running all the other jobs and/or tests unnecessarily.

Issue resolution:

• N/A

Select one: This pull request is...

• a bug fix: increment the patch version
• a small improvement: increment the minor version
• a new feature: increment the minor version
• an incompatible (non-backwards compatible) API change: increment the major version

Small Change

• To merge, I will use "Squash and merge". That is, this change should be a single commit.
• Logic: I have visually inspected the entire pull request myself.
• Pre-commit checks: All the pre-commits checks have passed.

[forsyth2]

I think this is good to merge; I just want to run on the full test suite. Since the code changes only affect the tests and not user-facing code, it won't be a problem if this merges after the Unified testing deadline.

[forsyth2]

@chengzhuzhang Let me know if you want to review this test-side-only change. Otherwise, I think this should be good to merge (aside from a couple dict keys I think I'll rename first).

Key points:

• Most of the things that need to be changed between test runs are now at the very top of the utils file. That is, the environments, the cfgs to run, the tasks/sets to run, and the unique test run id.
• This change allows a tester to auto-generate only the requested cfgs and to run the image checker on only the requested results directories & task subdirectories. This is currently possible via cumbersome commenting of the test code. These changes make it easier to toggle that functionality on/off. This would be useful for example when most of the tests are passing, but we just want to test a small change in ilamb (i.e., no need to re-run e3sm_diags too).
• I tested this commit on mpas_analysis, global_time_series, and ilamb during my testing of E3SM-Project/zppy-interfaces#26. I only left out e3sm_diags because of the diffs noted in the thread at #634 (comment). As expected, e3sm_diags tests were skipped rather than failed.

[forsyth2]

I guess this might be better labeled "tasks_to_run"

[forsyth2]

I guess this might be better labeled "cfgs_to_run"

[forsyth2]

Let me know if you want to review this test-side-only change. Otherwise, I think this should be good to merge

@chengzhuzhang I just want to re-ping you on this. I'd like to include this change for this week's weekly testing and other PR testing.

[chengzhuzhang]

I had a visual check, the changes look good for me.

[forsyth2]

Latest commit 093ae2f just changes the (internal testing) parameter names to be better descriptors. Visual inspection/search confirms all were changed. This is now good to merge.