Skip to content

Test metadata templating #470

@comps

Description

@comps

Currently, we copy/paste per-profile definitions like this:

main.fmf snippet
/anssi_bp28_high:

/anssi_bp28_enhanced:
    tag+:
      - subset-profile

/anssi_bp28_intermediary:
    tag+:
      - subset-profile

/anssi_bp28_minimal:
    tag+:
      - subset-profile

/bsi:
    adjust+:
      - when: distro != rhel-9
        enabled: false
        because: BSI profile is currently only on RHEL-9

/cis:

/cis_server_l1:
    tag+:
      - subset-profile

/cis_workstation_l2:

/cis_workstation_l1:
    tag+:
      - subset-profile

/cui:
    tag+:
      - fips
    adjust+:
      - when: distro >= rhel-10
        enabled: false
        because: there is no CUI profile on RHEL-10+

/e8:

/hipaa:

/ism_o:
    tag+:
      - fips

/ospp:
    tag+:
      - fips
    adjust+:
      - when: distro >= rhel-10
        enabled: false
        because: there is no OSPP profile on RHEL-10+

/pci-dss:

/stig:
    tag+:
      - fips

/stig_gui:
    adjust+:
      - enabled: false
        because: not supported without GUI, use stig instead

/ccn_advanced:
    adjust+:
      - when: distro == rhel-8 or distro == rhel-10
        enabled: false
        because: CCN profiles are not present on RHEL-8 and on RHEL-10

/ccn_intermediate:
    tag+:
      - subset-profile
    adjust+:
      - when: distro == rhel-8 or distro == rhel-10
        enabled: false
        because: CCN profiles are not present on RHEL-8 and on RHEL-10

/ccn_basic:
    tag+:
      - subset-profile
    adjust+:
      - when: distro == rhel-8 or distro == rhel-10
        enabled: false
        because: CCN profiles are not present on RHEL-8 and on RHEL-10

and that is not sustainable when CaC/content upstream keeps adding more profiles.

At the same time, FMF doesn't natively support any kind of multi-file templating, and while TMT has something in the works (they call it "policy", I believe, teemtee/tmt#3538), it doesn't work on generic FMF metadata.

As such, maybe we should create our own templating logic, perhaps using python code in scripts/*.py as configured by some conf/*.py file, that would generate / alter main.fmf files across the hierachy, adding per-profile tests.

Maybe the solution would be something like

  1. conf/metadata.py having
    1. a python list of .fmf file paths to modify
    2. a python list/dict/whatever of metadata to add, while NOT using any lib.* functionality, to make the metadata-building independent of OS, so the generated code is still fmf/tmt compatible
  2. Github CI deployment logic that automatically builds and deploys pre-built Contest somewhere, so we would have deterministic location to fetch built Contest from in CI pipelines
  3. Some way to build this manually, maybe a Makefile or manual ./scripts/build_metadata.py invocation, or whatever

Maybe we could store the configuration in the building script itself, instead of conf/, if conf/ seems too weird for something that external to the test suite.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions