Testing model build 2026.01.000 on `dev-MC_100km_jra_ryf` by chrisb13 · Pull Request #1033 · ACCESS-NRI/access-om3-configs

chrisb13 · 2026-01-10T01:57:05Z

Closes #1032. PR to test new om3 build 2026.01.000.

trivial commit

ACCESS-NRI/ACCESS-OM3#176 (comment) Testing `openmpi` version '@5.0.8'

chrisb13 · 2026-01-10T03:07:44Z

Can someone tell me why this failed? Here's the error:

AssertionError: Failed to find release artefact for model version at https://github.com/ACCESS-NRI/ACCESS-OM3/releases/tag/pr176-2
assert 404 == 200

Rather reads like it can't find my pre-release?

Also, @anton-seaice has written:

A draft-PR to dev-MC_100km_jra_ryf can be a good choice as it runs all (historical, determinism and determinism over restart) reproducability tests.

How do I run all these tests?

dougiesquire · 2026-01-11T21:31:50Z

Can someone tell me why this failed? Here's the error:

You need to update the Payu manifests: payu setup && payu sweep (note that if we don't have metadata:enable:false in the config.yaml then doing this will add a commit)

dougiesquire · 2026-01-11T21:31:58Z

!test repro

github-actions · 2026-01-11T21:38:26Z

✅ The Bitwise Reproducibility Check Succeeded ✅

When comparing:

1032-model-buid-update-test (checksums created using commit 92f0834), against
dev-MC_100km_jra_ryf (checksums in commit 100307f)

Further information

The experiment can be found on Gadi at /scratch/tm70/repro-ci/experiments/access-om3-configs/92f083494e8c4c3cab2b3a6891c5749d5c9c515a, and the test results at https://github.com/ACCESS-NRI/access-om3-configs/runs/60050026490.

The checksums generated by this !test command are found in the testing/checksum directory of https://github.com/ACCESS-NRI/access-om3-configs/actions/runs/20902209290/artifacts/5091750335.

The checksums compared against are found here https://github.com/ACCESS-NRI/access-om3-configs/tree/100307f72806dca4694ab2fb6bd5bf2366ba524a/testing/checksum

Test summary:
✅ test_repro_historical
✅ test_repro_determinism
✅ test_repro_restart

dougiesquire · 2026-01-11T21:40:14Z

How do I run all these tests?

From above:

✅ test_repro_historical
✅ test_repro_determinism
✅ test_repro_restart

anton-seaice · 2026-01-11T21:45:55Z

Can someone tell me why this failed? Here's the error:
AssertionError: Failed to find release artefact for model version at https://github.com/ACCESS-NRI/ACCESS-OM3/releases/tag/pr176-2
assert 404 == 200
Rather reads like it can't find my pre-release?

This just mean's it can't find the pre-release in the ACCESS-OM3 releases - which is expected obviously

anton-seaice · 2026-01-11T21:49:36Z

Can someone tell me why this failed? Here's the error:

You need to update the Payu manifests: payu setup && payu sweep (note that if we don't have metadata:enable:false in the config.yaml then doing this will add a commit)

I don't think this is correct, exe checksums are not being checked, (it's turned off)

From config.yaml:

manifest:
  reproduce:
    exe: false

dougiesquire · 2026-01-11T21:53:07Z

I don't think this is correct, exe checksums are not being checked, (it's turned off)

From config.yaml:
manifest:
  reproduce:
    exe: false

My comment is about the metadata field, not the manifest field. I'm trying to say that if metadata:enable:false is not set, then doing payu setup will add a commit.

chrisb13 · 2026-01-11T22:23:14Z

Thanks @anton-seaice and @dougiesquire. Sweet, confirmed (expected) that openmpi going from 4.1.7 to 5.0.8 does not change answers. Few questions:

You need to update the Payu manifests: payu setup && payu sweep

So did you do this? (I think not.) Is it something in here? Normally, payu setup && payu sweep is something I do in a cli so I'm not sure how it applies here.

This just mean's it can't find the pre-release in the ACCESS-OM3 releases - which is expected obviously

Ok, so effectively the payu setup && payu sweep bypasses this check somehow?

dougiesquire · 2026-01-11T22:32:34Z

You need to update the Payu manifests: payu setup && payu sweep

So did you do this? (I think not.) Is it something in here? Normally, payu setup && payu sweep is something I do in a cli so I'm not sure how it applies here.

Yeah, you need to do this at the command line.

This just mean's it can't find the pre-release in the ACCESS-OM3 releases - which is expected obviously

Ok, so effectively the payu setup && payu sweep bypasses this check somehow?

Sorry, it was misleading for me to make my comment a reply to yours. Updating the manifest will not fix the test_access_om3_manifest_exe_in_release_spack_location error

chrisb13 · 2026-01-11T22:37:09Z

Sorry, it was misleading for me to make my comment a reply to yours. Updating the manifest will not fix the test_access_om3_manifest_exe_in_release_spack_location error

I'm confused, so what does payu setup && payu sweep achieve here? Are you suggesting I could go to a cli, checkout 1032-model-buid-update-test do payu setup && payu sweep and then commit the change? (I would have guessed it wouldn't result in a change but would check that the config can find input files etc.)

On the test_access_om3_manifest_exe_in_release_spack_location error, we just ignore that then because this is a pre-release?

dougiesquire · 2026-01-11T22:45:14Z

I'm confused, so what does payu setup && payu sweep achieve here? Are you suggesting I could go to a cli, checkout 1032-model-buid-update-test do payu setup && payu sweep and then commit the change? (I would have guessed it wouldn't result in a change but would check that the config can find input files etc.)

It will mean that the manifests are correct. It's good to get in the habit of doing this. E.g. in many of our other configs we have:

manifest:
  reproduce:
    exe: True
    input: True

so updating exe/inputs and failing to update the manifests would mean that the config won't run.

I was trying to give some general helpful advice, but it would appear I've only added confusion...

On the test_access_om3_manifest_exe_in_release_spack_location error, we just ignore that then because this is a pre-release?

Yup

Testing this pre-release build: ACCESS-NRI/ACCESS-OM3#176 (comment) i.e. 7160dae ACCESS-NRI/ACCESS-OM3@7160dae and 4726d44 ACCESS-NRI/ACCESS-OM3@4726d44

helenmacdonald · 2026-01-12T00:53:46Z

!test repro

Testing generic tracers

github-actions · 2026-01-12T01:00:09Z

✅ The Bitwise Reproducibility Check Succeeded ✅

When comparing:

1032-model-buid-update-test (checksums created using commit 9c48fdb), against
dev-MC_100km_jra_ryf (checksums in commit 100307f)

Further information

The experiment can be found on Gadi at /scratch/tm70/repro-ci/experiments/access-om3-configs/9c48fdb262ad55b392ddf1ffe0ea0fca341e7a9f, and the test results at https://github.com/ACCESS-NRI/access-om3-configs/runs/60056799503.

The checksums generated by this !test command are found in the testing/checksum directory of https://github.com/ACCESS-NRI/access-om3-configs/actions/runs/20904863287/artifacts/5092585441.

The checksums compared against are found here https://github.com/ACCESS-NRI/access-om3-configs/tree/100307f72806dca4694ab2fb6bd5bf2366ba524a/testing/checksum

Test summary:
✅ test_repro_historical
✅ test_repro_determinism
✅ test_repro_restart

github-actions · 2026-01-12T01:02:41Z

❌ !test Command Failed ❌ The command given could not be parsed correctly. Usage: !test TYPE [commit] See https://github.com/ACCESS-NRI/access-om3-configs/actions/runs/20904985630

dougiesquire · 2026-01-12T01:04:19Z

!test repro

Testing generic tracers

Just a note that this configuration does not actually have WOMBAT configured

Testing updates to `access3-share`. See details: d84ae63 ACCESS-NRI/ACCESS-OM3@d84ae63

chrisb13 · 2026-01-19T05:05:19Z

!test repro

chrisb13 · 2026-01-19T05:11:39Z

I'm confused, so what does payu setup && payu sweep achieve here? Are you suggesting I could go to a cli, checkout 1032-model-buid-update-test do payu setup && payu sweep and then commit the change? (I would have guessed it wouldn't result in a change but would check that the config can find input files etc.)

It will mean that the manifests are correct. It's good to get in the habit of doing this. E.g. in many of our other configs we have:
manifest:
  reproduce:
    exe: True
    input: True
so updating exe/inputs and failing to update the manifests would mean that the config won't run.

So currently we have:

collate: false
runlog: false
metadata: 
    enable: false
manifest:
  reproduce:
    exe: false
    inputs: false

Proposal:

collate: false
runlog: true
metadata: 
    enable: false
manifest:
  reproduce:
    exe: false
    inputs: true

So, I'm suggesting we could turn on runlog and manifest/inputs. @dougiesquire suggested manifest/exe but I think that will fail here, is that a concern (or a feature)?

Did I miss anything @dougiesquire? Maybe we don't want runlog in this context?

github-actions · 2026-01-19T05:18:50Z

❌ The Bitwise Reproducibility Check Failed ❌

When comparing:

1032-model-buid-update-test (checksums created using commit 2b139f5), against
dev-MC_100km_jra_ryf (checksums in commit 100307f)

Further information

The experiment can be found on Gadi at /scratch/tm70/repro-ci/experiments/access-om3-configs/2b139f5864588f24d4514dd887238bfe04e668e3, and the test results at https://github.com/ACCESS-NRI/access-om3-configs/runs/60747952581.

The checksums generated by this !test command are found in the testing/checksum directory of https://github.com/ACCESS-NRI/access-om3-configs/actions/runs/21125980042/artifacts/5172726205.

The checksums compared against are found here https://github.com/ACCESS-NRI/access-om3-configs/tree/100307f72806dca4694ab2fb6bd5bf2366ba524a/testing/checksum

Test summary:
❌ test_repro_historical
✅ test_repro_determinism
✅ test_repro_restart

dougiesquire · 2026-01-19T05:25:21Z

@dougiesquire suggested manifest/exe but I think that will fail here, is that a concern (or a feature)?

I don't think I did suggest this.

So, I'm suggesting we could turn on runlog and manifest/inputs
Maybe we don't want runlog in this context?

There's not much value in setting either of these in this PR if that's what you're asking?

chrisb13 · 2026-01-19T05:30:35Z

I don't think I did suggest this.

There's not much value in setting either of these in this PR if that's what you're asking?

Oh ok, sounds like it was more of a general comment then? (i.e. I'll do nothing here.)

dougiesquire · 2026-01-19T05:33:34Z

My (intended) point was that it's probably good practice to keep the configuration manifests up-to-date with configuration changes

chrisb13 · 2026-01-19T05:35:43Z

Okay from this test.

It seems

❌ test_repro_historical
✅ test_repro_determinism
✅ test_repro_restart

I was expecting the first one to pass as well.

The changes are here (I think!):
https://github.com/ACCESS-NRI/ACCESS-OM3/pull/176/changes

which means it's either access3-share, spack-packages or the combination of previous changes that were not answer changing (access-generic-tracers and/or openmpi).

Any guesses on what I should try first? Or I can just brute force it...

dougiesquire · 2026-01-19T05:45:26Z

Almost certainly the access3-share update

Looking to test the theory that the `access3-share` update broke historical repro, see: ACCESS-NRI/access-om3-configs#1033 (comment)

Looking to test the theory that the `access3-share` update broke historical repro, see: #1033 (comment)

chrisb13 · 2026-01-19T22:01:29Z

!test repro

github-actions · 2026-01-19T22:07:56Z

✅ The Bitwise Reproducibility Check Succeeded ✅

When comparing:

1032-model-buid-update-test (checksums created using commit 8fda7eb), against
dev-MC_100km_jra_ryf (checksums in commit 100307f)

Further information

The experiment can be found on Gadi at /scratch/tm70/repro-ci/experiments/access-om3-configs/8fda7ebc134798d7b3b8130fa038c2e61a01feed, and the test results at https://github.com/ACCESS-NRI/access-om3-configs/runs/60831742382.

The checksums generated by this !test command are found in the testing/checksum directory of https://github.com/ACCESS-NRI/access-om3-configs/actions/runs/21152568608/artifacts/5182431903.

The checksums compared against are found here https://github.com/ACCESS-NRI/access-om3-configs/tree/100307f72806dca4694ab2fb6bd5bf2366ba524a/testing/checksum

Test summary:
✅ test_repro_historical
✅ test_repro_determinism
✅ test_repro_restart

Testing: ACCESS-NRI/ACCESS-OM3#176 (comment) i.e. "Update the CESM_share 1.1.18" [dcbe86b](ACCESS-NRI/access3-share@dcbe86b)

chrisb13 · 2026-01-22T01:18:11Z

!test repro

Testing: ACCESS-NRI/ACCESS-OM3#176 (comment) i.e. "Update to CDEPS 1.0.86" [4daf043](ACCESS-NRI/access3-share@4daf043)

chrisb13 · 2026-01-22T01:30:26Z

!test repro

Testing: ACCESS-NRI/ACCESS-OM3#176 (comment) i.e. "Update to CMEPS 1.1.31" [ed7680e](ACCESS-NRI/access3-share@ed7680e) (we expect this to fail, it'll be weird if it doesn't!)

chrisb13 · 2026-01-22T01:33:59Z

!test repro

github-actions · 2026-01-22T02:00:52Z

✅ The Bitwise Reproducibility Check Succeeded ✅

When comparing:

1032-model-buid-update-test (checksums created using commit 9bc9044), against
dev-MC_100km_jra_ryf (checksums in commit 100307f)

Further information

The experiment can be found on Gadi at /scratch/tm70/repro-ci/experiments/access-om3-configs/9bc9044b1f27ddc6fdf0cd27856c15a464021229, and the test results at https://github.com/ACCESS-NRI/access-om3-configs/runs/61095605815.

The checksums generated by this !test command are found in the testing/checksum directory of https://github.com/ACCESS-NRI/access-om3-configs/actions/runs/21232254355/artifacts/5213494002.

The checksums compared against are found here https://github.com/ACCESS-NRI/access-om3-configs/tree/100307f72806dca4694ab2fb6bd5bf2366ba524a/testing/checksum

Test summary:
✅ test_repro_historical
✅ test_repro_determinism
✅ test_repro_restart

github-actions · 2026-01-22T02:01:54Z

❌ The Bitwise Reproducibility Check Failed ❌

When comparing:

1032-model-buid-update-test (checksums created using commit 8e79383), against
dev-MC_100km_jra_ryf (checksums in commit 100307f)

Further information

The experiment can be found on Gadi at /scratch/tm70/repro-ci/experiments/access-om3-configs/8e79383c67eccbd92bca3609e60ce4c06d1c31e3, and the test results at https://github.com/ACCESS-NRI/access-om3-configs/runs/61095672188.

The checksums generated by this !test command are found in the testing/checksum directory of https://github.com/ACCESS-NRI/access-om3-configs/actions/runs/21232614107/artifacts/5213501262.

The checksums compared against are found here https://github.com/ACCESS-NRI/access-om3-configs/tree/100307f72806dca4694ab2fb6bd5bf2366ba524a/testing/checksum

Test summary:
❌ test_repro_historical
✅ test_repro_determinism
✅ test_repro_restart

github-actions · 2026-01-22T02:01:59Z

❌ The Bitwise Reproducibility Check Failed ❌

When comparing:

1032-model-buid-update-test (checksums created using commit cd85b2a), against
dev-MC_100km_jra_ryf (checksums in commit 100307f)

Further information

The experiment can be found on Gadi at /scratch/tm70/repro-ci/experiments/access-om3-configs/cd85b2a6606ffa495de81650975e360b61f56dc1, and the test results at https://github.com/ACCESS-NRI/access-om3-configs/runs/61095678405.

The checksums generated by this !test command are found in the testing/checksum directory of https://github.com/ACCESS-NRI/access-om3-configs/actions/runs/21232529951/artifacts/5213501539.

The checksums compared against are found here https://github.com/ACCESS-NRI/access-om3-configs/tree/100307f72806dca4694ab2fb6bd5bf2366ba524a/testing/checksum

Test summary:
❌ test_repro_historical
✅ test_repro_determinism
✅ test_repro_restart

Checking this build: ACCESS-NRI/ACCESS-OM3#176 (comment) I expect this to run okay, I've reverted to the old version of CDEPS

chrisb13 · 2026-01-26T23:23:49Z

!test repro

github-actions · 2026-01-26T23:32:25Z

✅ The Bitwise Reproducibility Check Succeeded ✅

When comparing:

1032-model-buid-update-test (checksums created using commit 06535c4), against
dev-MC_100km_jra_ryf (checksums in commit 100307f)

Further information

The experiment can be found on Gadi at /scratch/tm70/repro-ci/experiments/access-om3-configs/06535c4b70c476346e235358027f2c4903b3b047, and the test results at https://github.com/ACCESS-NRI/access-om3-configs/runs/61538904039.

The checksums generated by this !test command are found in the testing/checksum directory of https://github.com/ACCESS-NRI/access-om3-configs/actions/runs/21377903827/artifacts/5264782762.

The checksums compared against are found here https://github.com/ACCESS-NRI/access-om3-configs/tree/100307f72806dca4694ab2fb6bd5bf2366ba524a/testing/checksum

Test summary:
✅ test_repro_historical
✅ test_repro_determinism
✅ test_repro_restart

Testing a newer CDEPS version: > ACCESS-NRI/ACCESS-OM3@f78bf8c

chrisb13 · 2026-01-29T01:12:12Z

!test repro

github-actions · 2026-01-29T01:19:39Z

✅ The Bitwise Reproducibility Check Succeeded ✅

When comparing:

1032-model-buid-update-test (checksums created using commit d0b5348), against
dev-MC_100km_jra_ryf (checksums in commit 100307f)

Further information

The experiment can be found on Gadi at /scratch/tm70/repro-ci/experiments/access-om3-configs/d0b534872ee57fa81fa7bc3493e2e4e289169725, and the test results at https://github.com/ACCESS-NRI/access-om3-configs/runs/61815946088.

The checksums generated by this !test command are found in the testing/checksum directory of https://github.com/ACCESS-NRI/access-om3-configs/actions/runs/21461735740/artifacts/5297111354.

The checksums compared against are found here https://github.com/ACCESS-NRI/access-om3-configs/tree/100307f72806dca4694ab2fb6bd5bf2366ba524a/testing/checksum

Test summary:
✅ test_repro_historical
✅ test_repro_determinism
✅ test_repro_restart

Update config.yaml

f571864

trivial commit

chrisb13 changed the base branch from main to dev-MC_100km_jra_ryf January 10, 2026 01:59

Update Payu configuration for module paths

92f0834

ACCESS-NRI/ACCESS-OM3#176 (comment) Testing `openmpi` version '@5.0.8'

chrisb13 marked this pull request as draft January 10, 2026 03:24

chrisb13 changed the title ~~Testing model build 2026.01.000 on~~ Testing model build 2026.01.000 on dev-MC_100km_jra_ryf Jan 12, 2026

Update config.yaml

9c48fdb

Testing this pre-release build: ACCESS-NRI/ACCESS-OM3#176 (comment) i.e. 7160dae ACCESS-NRI/ACCESS-OM3@7160dae and 4726d44 ACCESS-NRI/ACCESS-OM3@4726d44

Update config.yaml

2b139f5

Testing updates to `access3-share`. See details: d84ae63 ACCESS-NRI/ACCESS-OM3@d84ae63

chrisb13 mentioned this pull request Jan 19, 2026

Upstream model component update: January 2026. #1028

Open

chrisb13 added a commit to ACCESS-NRI/ACCESS-OM3 that referenced this pull request Jan 19, 2026

Update access3 package version in spack.yaml

eb8627e

Looking to test the theory that the `access3-share` update broke historical repro, see: ACCESS-NRI/access-om3-configs#1033 (comment)

Update module version from pr176-12 to pr176-13

8fda7eb

Looking to test the theory that the `access3-share` update broke historical repro, see: #1033 (comment)

chrisb13 mentioned this pull request Jan 21, 2026

Migrate from patches to forks on CDEPS, CMEPS, cesm_share ACCESS-NRI/access3-share#22

Closed

Update config.yaml

9bc9044

Testing: ACCESS-NRI/ACCESS-OM3#176 (comment) i.e. "Update the CESM_share 1.1.18" [dcbe86b](ACCESS-NRI/access3-share@dcbe86b)

Update config.yaml

cd85b2a

Testing: ACCESS-NRI/ACCESS-OM3#176 (comment) i.e. "Update to CDEPS 1.0.86" [4daf043](ACCESS-NRI/access3-share@4daf043)

Update config.yaml

8e79383

Testing: ACCESS-NRI/ACCESS-OM3#176 (comment) i.e. "Update to CMEPS 1.1.31" [ed7680e](ACCESS-NRI/access3-share@ed7680e) (we expect this to fail, it'll be weird if it doesn't!)

Update config.yaml

06535c4

Checking this build: ACCESS-NRI/ACCESS-OM3#176 (comment) I expect this to run okay, I've reverted to the old version of CDEPS

chrisb13 mentioned this pull request Jan 26, 2026

Rc 2026.01.000 reprobug ACCESS-NRI/access3-share#26

Draft

Update config.yaml

d0b5348

Testing a newer CDEPS version: > ACCESS-NRI/ACCESS-OM3@f78bf8c

Conversation

chrisb13 commented Jan 10, 2026

Uh oh!

chrisb13 commented Jan 10, 2026

Uh oh!

dougiesquire commented Jan 11, 2026

Uh oh!

dougiesquire commented Jan 11, 2026

Uh oh!

github-actions bot commented Jan 11, 2026

Uh oh!

dougiesquire commented Jan 11, 2026

Uh oh!

anton-seaice commented Jan 11, 2026

Uh oh!

anton-seaice commented Jan 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dougiesquire commented Jan 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chrisb13 commented Jan 11, 2026

Uh oh!

dougiesquire commented Jan 11, 2026

Uh oh!

chrisb13 commented Jan 11, 2026

Uh oh!

dougiesquire commented Jan 11, 2026

Uh oh!

helenmacdonald commented Jan 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jan 12, 2026

Uh oh!

github-actions bot commented Jan 12, 2026

Uh oh!

dougiesquire commented Jan 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chrisb13 commented Jan 19, 2026

Uh oh!

chrisb13 commented Jan 19, 2026

Uh oh!

github-actions bot commented Jan 19, 2026

Uh oh!

dougiesquire commented Jan 19, 2026

Uh oh!

chrisb13 commented Jan 19, 2026

Uh oh!

dougiesquire commented Jan 19, 2026

Uh oh!

chrisb13 commented Jan 19, 2026

Uh oh!

dougiesquire commented Jan 19, 2026

Uh oh!

chrisb13 commented Jan 19, 2026

Uh oh!

github-actions bot commented Jan 19, 2026

Uh oh!

chrisb13 commented Jan 22, 2026

Uh oh!

chrisb13 commented Jan 22, 2026

Uh oh!

chrisb13 commented Jan 22, 2026

Uh oh!

github-actions bot commented Jan 22, 2026

Uh oh!

github-actions bot commented Jan 22, 2026

Uh oh!

github-actions bot commented Jan 22, 2026

Uh oh!

chrisb13 commented Jan 26, 2026

Uh oh!

github-actions bot commented Jan 26, 2026

Uh oh!

chrisb13 commented Jan 29, 2026

Uh oh!

github-actions bot commented Jan 29, 2026

Uh oh!

Reviewers

anton-seaice commented Jan 11, 2026 •

edited

Loading

dougiesquire commented Jan 11, 2026 •

edited

Loading

helenmacdonald commented Jan 12, 2026 •

edited

Loading

dougiesquire commented Jan 12, 2026 •

edited

Loading