Fix ptscotch test by lindsayad · Pull Request #32394 · idaholab/moose

lindsayad · 2026-02-25T00:01:07Z

We need to obey the PETSc documentation which requests that the adjacency list be sorted (per row). @roystgnr I checked to see whether we call MatCreateMPIAdj in libMesh and I see that we do in one place. AFAICT we also don't do any sorting there, so that could be a landmine waiting to happen

lindsayad · 2026-02-25T01:23:23Z

Hmm this actually seems like it's going to have involve libMesh because I'm getting errors out of recent ptscotch if I attempt to sort the graph after build_graph. I'm guessing this is due to constructing things like _local_id_to_elem with the unsorted graph?

lindsayad · 2026-02-25T01:50:55Z

Hmm this actually seems like it's going to have involve libMesh because I'm getting errors out of recent ptscotch if I attempt to sort the graph after build_graph. I'm guessing this is due to constructing things like _local_id_to_elem with the unsorted graph?

Hmm no I don't see anything obvious that suggests if I sort _dual_graph rows post build_graph that it would break any other data structures

moosebuild · 2026-02-25T02:18:29Z

Job Documentation, step Docs: sync website on c31db84 wanted to post the following:

View the site here

This comment will be updated on new commits.

Closes idaholab#32393

This reverts commit 25ac6b2.

Also add some fuzzy comparison safety to floating point equality testing

These clones led to a fun bug where when I tried to use WorkBalance with the `BlockWeightedPartitioner` which relies on data computed in `initialize` instead of the constructor, leading to failed assertions during the WorkBalance calculations

lindsayad · 2026-03-01T05:25:39Z

@ChengHauYang your equal_value_boundary_constraint.error_multiple_primary_nodes test does not work with --distributed-mesh run in serial, which is kind of fascinating. Can you figure out what's going on there?

ChengHauYang · 2026-03-01T06:23:44Z

Thanks for catching the bugs, @lindsayad. In break_mesh_with_evbc.i, the test hard-codes secondary_node_ids = '1 2 3 4 5 6'. After BMBB duplicates the interface node, the duplicates get different IDs on a distributed mesh than on a replicated mesh. As a result, under distributed serial, that hard-coded node list no longer contains both coincident nodes at the requested primary_node_coord, so EqualValueBoundaryConstraint (EVBC) only sees one valid candidate and does not trigger the expected "Multiple nodes found" error.
In short: the failure is caused by mesh-dependent node numbering in the test setup, not by the EVBC or BMBB logic itself.
The short fix is to avoid using BMBB in the input file for this particular test, which would require hard-coding secondary_node_ids. I have tested this on multiple processor counts (including serial) on my local machine. I hope it can work. The fixing patch is in 945a65b.

Thanks for your help!!

moosebuild · 2026-03-01T10:00:31Z

Job Coverage, step Generate coverage on c31db84 wanted to post the following:

Framework coverage

	801dc5	#32394 c31db8
	Total	Total	+/-	New
Rate	85.78%	85.78%	+0.00%	100.00%
Hits	128547	128554	+7	30
Misses	21316	21317	+1	0

Diff coverage report

Full coverage report

Modules coverage

Thermal hydraulics

	801dc5	#32394 c31db8
	Total	Total	+/-	New
Rate	88.88%	88.87%	-0.01%	-
Hits	15430	15429	-1	0
Misses	1931	1932	+1	0

Diff coverage report

Full coverage report

Full coverage reports

Reports

This comment will be updated on new commits.

ChengHauYang · 2026-03-01T15:52:02Z

Hi @lindsayad,

I found a simpler fix than the previous fix patch: 51a4d1d.

The way is to list all node IDs inside secondary_node_ids. I actually did this at the beginning, but I accidentally incremented each node ID by 1.

Thanks for your help! I hope this fixes the issue.

lindsayad · 2026-03-02T19:18:28Z

The fixing patch is in 945a65b.

I just fetched from your repository and that commit is not visible to me. Do you have a branch tip that includes that commit?

ChengHauYang · 2026-03-02T19:28:56Z

Sorry, @lindsayad. I think maybe you can go with this patch: 51a4d1d. The branch is "fix_evbc_simple".

Thanks for your help!

lindsayad · 2026-03-02T19:58:02Z

Thank you for being so responsive! It's awesome having you as a MOOSE contributor

ChengHauYang · 2026-03-02T20:22:04Z

You are very welcome, @lindsayad! Thanks for your kind words!

lindsayad · 2026-03-03T18:05:22Z

Griffin patch at https://github.inl.gov/ncrc/griffin/pull/3000

moosebuild · 2026-03-04T06:03:11Z

Job Test, step Results summary on c31db84 wanted to post the following:

Framework test summary

Compared against 801dc58 in job civet.inl.gov/job/3617958.

Removed tests

Test	Time (s)	Memory (MB)
`partitioners/petsc_partitioner.ptscotch_weight_elment`	0.82	0.00

Added tests

Test	Time (s)	Memory (MB)
`partitioners/petsc_partitioner.ptscotch_weight_element`	0.85	0.00

Modules test summary

Compared against 801dc58 in job civet.inl.gov/job/3617958.

No change

moosebuild · 2026-03-04T09:33:02Z

Job Coverage, step Verify coverage on c31db84 wanted to post the following:

The following coverage requirement(s) failed:

Failed to generate richards coverage rate (required: 93.0%)

lindsayad · 2026-03-05T00:14:16Z

@loganharbour it seems like there are some CI problems with the sweeps for processor counts >= 11. Maybe resource issues? I do get failures when I run on my system locally, but they are definitely not the same

Failed Tests:
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
outputs/exodus.hdf5 .............................................................................................................................................................. FAILED (EXIT CODE 77 != 0)
vectorpostprocessors/work_balance.work_balance/replicated ..................................................................................................................... [max_cpus=2] FAILED (CSVDIFF)
multiapps/sub_cycling.group/test ................................................................................................................................................ FAILED (EXIT CODE 134 != 0)
multisystem/restore_multiapp.nl_sol ............................................................................................................................................... FAILED (EXIT CODE 1 != 0)
multisystem/picard/linearfv_nonlinearfv.tightly_coupled ........................................................................................................................... FAILED (EXIT CODE 1 != 0)
transfers/multiapp_copy_transfer/linear_sys_to_aux.test ........................................................................................................................... FAILED (EXIT CODE 1 != 0)
transfers/multiapp_variable_value_sample_transfer.array_sample_test/input_positions ............................................................................................... FAILED (EXIT CODE 1 != 0)
multiapps/linearfv_nonlinearfv.linearfv/linearfv_mainapp .......................................................................................................................... FAILED (EXIT CODE 1 != 0)
multiapps/linearfv_nonlinearfv.linearfv/nonlinearfv_mainapp ....................................................................................................................... FAILED (EXIT CODE 1 != 0)
variables/linearfv.basic_aux ...................................................................................................................................................... FAILED (EXIT CODE 1 != 0)
restart/kernel_restartable.thread_error/with_threads .................................................................................................. [min_threads=4,max_cpus=1] FAILED (EXIT CODE -6 != 0)
multiapps/steffensen_postprocessor.pp_transient/app_begin_transfers_begin_steffensen_sub ........................................................................................ FAILED (EXIT CODE 143 != 0)
meshgenerators/coarsen_block_generator.coarsen_hex/multiple_levels ......................................................................................................................... FAILED (TIMEOUT)
transfers/multiapp_variable_value_sample_transfer.block_restricted_primary/multiapp ............................................................................................... FAILED (EXIT CODE 1 != 0)
tag.controls-tagging ............................................................................................................................................................ FAILED (EXIT CODE 134 != 0)
tag.linear-fv ..................................................................................................................................................................... FAILED (EXIT CODE 1 != 0)
transfers/multiapp_variable_value_sample_transfer.block_restricted_primary/multiapp_and_var ....................................................................................... FAILED (EXIT CODE 1 != 0)
ics/random_ic_test.test_threaded ....................................................................................................................... [FINISHED,min_threads=2] FAILED (EXIT CODE 139 != 0)

loganharbour · 2026-03-05T00:42:41Z

@loganharbour it seems like there are some CI problems with the sweeps for processor counts >= 11. Maybe resource issues? I do get failures when I run on my system locally, but they are definitely not the same

Failed Tests:
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
outputs/exodus.hdf5 .............................................................................................................................................................. FAILED (EXIT CODE 77 != 0)
vectorpostprocessors/work_balance.work_balance/replicated ..................................................................................................................... [max_cpus=2] FAILED (CSVDIFF)
multiapps/sub_cycling.group/test ................................................................................................................................................ FAILED (EXIT CODE 134 != 0)
multisystem/restore_multiapp.nl_sol ............................................................................................................................................... FAILED (EXIT CODE 1 != 0)
multisystem/picard/linearfv_nonlinearfv.tightly_coupled ........................................................................................................................... FAILED (EXIT CODE 1 != 0)
transfers/multiapp_copy_transfer/linear_sys_to_aux.test ........................................................................................................................... FAILED (EXIT CODE 1 != 0)
transfers/multiapp_variable_value_sample_transfer.array_sample_test/input_positions ............................................................................................... FAILED (EXIT CODE 1 != 0)
multiapps/linearfv_nonlinearfv.linearfv/linearfv_mainapp .......................................................................................................................... FAILED (EXIT CODE 1 != 0)
multiapps/linearfv_nonlinearfv.linearfv/nonlinearfv_mainapp ....................................................................................................................... FAILED (EXIT CODE 1 != 0)
variables/linearfv.basic_aux ...................................................................................................................................................... FAILED (EXIT CODE 1 != 0)
restart/kernel_restartable.thread_error/with_threads .................................................................................................. [min_threads=4,max_cpus=1] FAILED (EXIT CODE -6 != 0)
multiapps/steffensen_postprocessor.pp_transient/app_begin_transfers_begin_steffensen_sub ........................................................................................ FAILED (EXIT CODE 143 != 0)
meshgenerators/coarsen_block_generator.coarsen_hex/multiple_levels ......................................................................................................................... FAILED (TIMEOUT)
transfers/multiapp_variable_value_sample_transfer.block_restricted_primary/multiapp ............................................................................................... FAILED (EXIT CODE 1 != 0)
tag.controls-tagging ............................................................................................................................................................ FAILED (EXIT CODE 134 != 0)
tag.linear-fv ..................................................................................................................................................................... FAILED (EXIT CODE 1 != 0)
transfers/multiapp_variable_value_sample_transfer.block_restricted_primary/multiapp_and_var ....................................................................................... FAILED (EXIT CODE 1 != 0)
ics/random_ic_test.test_threaded ....................................................................................................................... [FINISHED,min_threads=2] FAILED (EXIT CODE 139 != 0)

Signal 15 is a kill. I bet they were killed due to the host being OOM...

lindsayad · 2026-03-09T17:11:24Z

In this case I can't blame my navier-stokes test 😆

lindsayad · 2026-03-09T17:12:09Z

You think after all the memory enforcement policies you're working on trickle through that we shouldn't run into recipe failures like this?

loganharbour · 2026-03-09T17:13:36Z

Eventually. There's a lot more to go... valgrind and apps. And then you have crap like this:

20 Heaviest Jobs (memory/slot):
--------------------------------------------------------------------------------------------------------------
[172.8s] [4634MB]       OK htgr/mhtgr/3D_mesh.3D_MHTGR_mesh [FINISHED]
[1014s ] [4143MB]       OK sfr/subchannel/multiple_SCM_assemblies/19assemblies.SCM19 [FINISHED,min_cpus=10]
[52.08s] [4027MB]       OK sfr/subchannel/multiple_SCM_assemblies/7assemblies.Master_app_syntax SYNTAX PASS
[128.8s] [3737MB]       OK htgr/mhtgr/3D_mesh.3D_MHTGR_syntax SYNTAX PASS [FINISHED]
[267.3s] [2664MB]       OK htgr/httf/inputs.core [FINISHED,min_cpus=16]
[467.3s] [2005MB]       OK sfr/subchannel/multiple_SCM_assemblies/7assemblies.SCM7 [min_cpus=7,FINISHED]
[196.3s] [1993MB]       OK msr/msre/pipe_cardinal.solid_mechanics [FINISHED]
[90.62s] [1640MB]       OK htgr/httf/inputs.core_syntax SYNTAX PASS
[44.42s] [1376MB]       OK research_reactors/agn.3D_AGN-201_mesh

lindsayad · 2026-03-09T17:14:41Z

that's vtb right?

lindsayad force-pushed the fix-ptscotch-test branch from c529b81 to 390e45f Compare February 27, 2026 07:13

lindsayad added 13 commits February 27, 2026 01:44

Avoid enum-scalar arithmetic warnings

4a45474

Sort adjacency list for each row

e7d21e2

Closes idaholab#32393

Clean up extraneous lines in PetscExternalPartitioner

ea66d6f

Don't request to use edge weights if they don't exist

5ab504b

Revert "Sort adjacency list for each row"

0f516c7

This reverts commit 25ac6b2.

Sort graph in-place after build_graph

a2bca47

Fix side weightins such that ordering matches the graph's ordering

b6ec557

Also add some fuzzy comparison safety to floating point equality testing

Remove cruft

c61dd4c

Update to new KSPSetDMActive API in Petsc 3.25

a835d2d

typo fixes

f0b8970

Make block_weighted_partitioner tests better

7c6968d

Ignore seemingly new test-related yaml files

705ca4d

lindsayad force-pushed the fix-ptscotch-test branch from 38fe6d9 to 705ca4d Compare February 27, 2026 08:45

lindsayad added 3 commits February 27, 2026 13:41

Sort graph in DistributedRectilinearMeshGenerator as well

c0bba6e

Use linear partitioner for phase field voronoi adaptivity tests

2c41a28

Disable partitioning in a way that impacts recover as well

e63bb8c

Fix EVBC duplicate-node test for replicated and distributed meshes

3129475

Update contact module for PETSc 3.25 changes

c31db84

GiudGiud assigned lindsayad Mar 8, 2026

Conversation

lindsayad commented Feb 25, 2026

Uh oh!

lindsayad commented Feb 25, 2026

Uh oh!

lindsayad commented Feb 25, 2026

Uh oh!

moosebuild commented Feb 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lindsayad commented Mar 1, 2026

Uh oh!

ChengHauYang commented Mar 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

moosebuild commented Mar 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Framework coverage

Modules coverage

Thermal hydraulics

Full coverage reports

Uh oh!

ChengHauYang commented Mar 1, 2026

Uh oh!

lindsayad commented Mar 2, 2026

Uh oh!

ChengHauYang commented Mar 2, 2026

Uh oh!

lindsayad commented Mar 2, 2026

Uh oh!

ChengHauYang commented Mar 2, 2026

Uh oh!

lindsayad commented Mar 3, 2026

Uh oh!

moosebuild commented Mar 4, 2026

Framework test summary

Removed tests

Added tests

Modules test summary

Uh oh!

moosebuild commented Mar 4, 2026

Uh oh!

lindsayad commented Mar 5, 2026

Uh oh!

loganharbour commented Mar 5, 2026

Uh oh!

lindsayad commented Mar 9, 2026

Uh oh!

lindsayad commented Mar 9, 2026

Uh oh!

loganharbour commented Mar 9, 2026

Uh oh!

lindsayad commented Mar 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

moosebuild commented Feb 25, 2026 •

edited

Loading

ChengHauYang commented Mar 1, 2026 •

edited

Loading

moosebuild commented Mar 1, 2026 •

edited

Loading