CI Test Results Comparison: Local vs build_and_test_epmt vs docker_build_test (v4.11.0, main, 2026-03-06)

[ilaflott]

CI Test Results: Local vs build_and_test_epmt vs docker_build_test

Comparison of test outcomes across three environments for EPMT v4.11.0 on the main branch (March 6, 2026).

• Local: editable install in conda env epmt4.11_pip, Rocky 8 workstation
• build_and_test_epmt CI: bare Rocky 8 container, pip install from source (run 22780457052)
• docker_build_test CI: Docker test-release image on slurm-cluster runner (run 22780457065)

Unit Tests

Suite	Local	build_and_test_epmt	docker_build_test
test_anysh	2 passed	2 passed	1 failed (GUARDED)
test_cmds	13 passed	13 passed	13 passed
test_db_migration	2 skipped	2 skipped	2 skipped
test_db_schema	1 passed, 1 skipped	1 passed, 1 skipped	1 passed, 1 skipped
test_explore	2 passed	2 passed	2 passed
test_lib	9 passed	9 passed	9 passed
test_outliers	25 passed	25 passed	25 passed
test_query	29 passed	29 passed	29 passed
test_run	8 passed	8 passed	1 failed (GUARDED)
test_settings	3 passed	3 passed	3 passed
test_shell	1 skipped	1 skipped	1 skipped
test_stat	9 passed	9 passed	9 passed
test_submit	12 passed	12 passed	12 passed

Integration Tests

Suite	Local	build_and_test_epmt	docker_build_test
integration, basic	6 passed	6 passed	1 error (GUARDED)
integration, escape	1 passed, 1 skipped	1 passed, 1 skipped	1 failed (GUARDED)
integration, concat	6 skipped	6 skipped	6 skipped
integration, daemon	3 passed	3 passed	3 passed
integration, shell	1 passed	1 passed	1 passed
integration, annotate	8 passed	8 passed	1 error (GUARDED)
integration, explore	1 passed	1 passed	1 error (GUARDED)
integration, slurm	9 skipped	9 skipped	1 failed (GUARDED)
integration, collate-tsv	1 passed	1 passed	1 failed (GUARDED)
integration, kernel-compile	2 skipped	2 skipped	2 skipped

Key Observations

1. Local and build_and_test_epmt results are identical — every suite matches exactly in pass/skip/fail counts.

2. docker_build_test has 8 failures/errors, all on steps marked GUARDED (continue-on-error: true), which is why the workflow still reports overall success.

3. Root causes in the Docker test-release image:

   • test_anysh / test_run: epmt_run returns exit code 127 (command not found). The LD_PRELOAD'd papiex libraries cannot locate binaries on the container's $PATH.
   • integration basic, annotate, explore: Collection errors — likely import or path issues inside the test-release container.
   • integration escape, slurm, collate-tsv: Test failures — slurm and collate-tsv tests require slurm/papiex infrastructure not fully wired in the container; escape has a similar wrapper issue.

4. The GUARDED pattern is masking real breakage. The bare Rocky 8 pipeline (build_and_test_epmt) passes everything that isn't legitimately skipped, but the Docker test-release image has environment issues (missing binaries on $PATH, papiex library incompatibilities) that cause 8 suites to fail silently.

5. Legitimately skipped suites (consistent across all three environments):

   • test_db_migration (2 skipped) — requires Alembic migration setup
   • test_shell (1 skipped) — requires interactive shell
   • integration, concat (6 skipped) — requires concat infrastructure
   • integration, kernel-compile (2 skipped) — requires kernel compilation toolchain
   • test_db_schema has 1 skipped (PostgreSQL-specific test)