fix(ci): Unify noir contract compilation error snapshot testing#22733
Merged
vezenovm merged 9 commits intomerge-train/fairiesfrom Apr 24, 2026
Merged
Conversation
…pdates, plus snap updates that were previously being removed
…t-error-snapshot-testing
…into mv/expand-contract-compile-failure-test-coverage
nchamo
approved these changes
Apr 24, 2026
nchamo
reviewed
Apr 24, 2026
Co-authored-by: Nicolas Chamo <nicolas@chamo.com.ar>
) ## Motivation PR #22733 unified the compile-failure runner and migrated four legacy cases into `noir-contracts-comp-failures/`, bringing total coverage to 22 contracts. That left ~28 user-reachable `panic!` sites in `aztec-nr/aztec/src/macros/` with no dedicated test. This PR closes those gaps so every user-reachable panic has a pinned snapshot. ## Approach For each gap, a minimal `contracts/<name>/` contract triggers the target panic and an `expected_error` file pins nargo's output verbatim. Contracts that would otherwise cascade into an unrelated panic (the generic `check_each_fn_macroified` message, or the "noinitcheck without initializer" message) are isolated with a `#[contract_library_method]` marker or a stub `#[initializer]` so each test pins exactly one target. ## Changes - **`noir-contracts-comp-failures/contracts/`**: 28 new contracts, one per previously-uncovered panic site (see diff for details). One contract (`authorize_once_before_external`) ships with an empty `expected_error` as a known-unreachable reproducer. - **Realistic cascading errors**: In addition to the 28 isolated cases, `contracts/bob_token/` pins the full cascade behavior of the current library on a realistic program. It's a token contract adapted from the https://docs.aztec.network/developers/docs/tutorials/contract_tutorials/token_contract with four broken functions. - Because the broken functions have realistic bodies that reference `self` and call each other, the snapshot captures 38 error lines (when we should only have 4 errors). - This gives us a baseline to measure against. After https://linear.app/aztec-labs/issue/F-578/aztec-nr-refactor-macro-validators-to-collect-errors-and-emit-a-dummy we should expect the snapshot to shrink. - **`noir-contracts-comp-failures/bootstrap.sh`**: empty `expected_error` now means "expect compile success"; `ACCEPT_SNAPSHOTS=1` writes an empty file when compilation succeeds. Wrong count or content falls through to the existing count-mismatch path. - **`noir-contracts-comp-failures/README.md`**: one-line note on the empty-snapshot convention. ## Follow-ups - [F-579](https://linear.app/aztec-labs/issue/F-579/authorize-once-attribute-ordering-panic-at-helpersnr55-is-unreachable): the `#[authorize_once]` attribute-ordering panic at `helpers.nr:55` does not fire for the documented trigger; the reproducer ships with an empty snapshot. - [F-580](https://linear.app/aztec-labs/issue/F-580/authorization-macro-is-pub-but-has-no-downstream-consumers-and-leaks): `#[authorization]` is `pub` but has no downstream consumers; its generated code references `crate::authwit::*` paths that don't resolve from a downstream crate, producing a 5-line cascade before the target diagnostic. Building off of #22733
AztecBot
pushed a commit
that referenced
this pull request
Apr 24, 2026
Resolves https://linear.app/aztec-labs/issue/F-577/unified-noir-contract-error-snapshot-testing Summary: - 4 cases moved into `noir-projects/noir-contracts-comp-failures/contracts/` - Old runner deleted: `noir-projects/aztec-nr/macro_compilation_failure_tests/` + both invocations in `noir-projects/aztec-nr/bootstrap.sh`. - Runner extended (`noir-projects/noir-contracts-comp-failures/bootstrap.sh`): - Matcher asserts **strict equality**, line-for-line, between the `error:` headlines nargo emits (prefix stripped, in emission order) and the non-blank lines of `expected_error`. Any difference in text, count, or order fails the test. - `ACCEPT_SNAPSHOTS=1 ./bootstrap.sh test` auto-extracts every `error:` headline and writes them one-per-line into `expected_error`. A `⚠` fallback writes full stderr when nargo emits no `error:` lines (ICE/panic), prompting a manual edit. - Devs no longer have to write their own `expected_error` file manually - CI guard: `ACCEPT_SNAPSHOTS=1` + `CI=1` is refused, so committed snapshots and live stderr can only diverge when a dev regenerates locally and commits. - `test` accepts an optional contract-name glob to run a subset: `./bootstrap.sh test reserved_public_dispatch` or `./bootstrap.sh test 'panic_on_*'`. No-arg behaviour is unchanged; `test_cmds` (CI wiring) still emits the no-arg form. Composes with `ACCEPT_SNAPSHOTS=1` for targeted snapshot updates. - Snapshots regenerated (13 `expected_error` files changed): - Added specificity where previous dropped by manual trims (e.g. `duplicate_storage` now pins `'Storage', got 'Storage2' instead.`, `event_selector_collision` pins both colliding event names, `marked_*_unconstrained` pins the function name). - Several cases gained cascading downstream `error:` lines that previous single-line trims were hiding (`Could not resolve 'init'/'ZERO' in path`, `Type annotation needed`). Kept them in the snapshot. This is part of the actual compiler behavior being asserted. - READMEs updated: aztec-nr README; comp-failures README Verification (done _manually_): ``` cd noir-projects/noir-contracts-comp-failures ``` 1. Happy case: ``` ./bootstrap.sh test ``` 2. Wrong-error path: I replaced the message in `contracts/reserved_public_dispatch/expected_error` with garbage ``` ./bootstrap.sh test Gives this output: ... ✓ contracts/panic_on_owned_state_var_in_storage/: Compilation failed as expected with correct error(s) ✓ contracts/public_function_selector_collision/: Compilation failed as expected with correct error(s) ✓ contracts/reserved_emit_public_init_nullifier/: Compilation failed as expected with correct error(s) ✗ contracts/reserved_public_dispatch/: error 1 does not match Expected: Function name '' is reserved for internal use Actual: Function name 'public_dispatch' is reserved for internal use ``` 4. Success-compile path: Change reserved_public_dispatch to `contract ReservedPublicDispatch {}` so that it succeeds compilation. ``` ... ✓ contracts/public_function_selector_collision/: Compilation failed as expected with correct error ✓ contracts/reserved_emit_public_init_nullifier/: Compilation failed as expected with correct error ✗ contracts/reserved_public_dispatch/: Expected compilation to fail but it succeeded ``` 5. Missing error Removed "Could not resolve 'init' in path" from panic_on_owned_state_var_in_storage. Running bootstrap we get the following: ``` ✗ contracts/panic_on_owned_state_var_in_storage/: error count mismatch — expected 2, got 3 Expected lines: Type PrivateImmutable<Field, Context> implements OwnedStateVariable and hence cannot be placed in Storage struct without being wrapped in Owned. Define the type in storage as Owned<PrivateImmutable<Field, Context><..., Context>>, Context>. Type annotation needed Actual error headlines: Type PrivateImmutable<Field, Context> implements OwnedStateVariable and hence cannot be placed in Storage struct without being wrapped in Owned. Define the type in storage as Owned<PrivateImmutable<Field, Context><..., Context>>, Context>. Could not resolve 'init' in path Type annotation needed Full stderr: error: Type PrivateImmutable<Field, Context> implements OwnedStateVariable and hence cannot be placed in Storage struct without being wrapped in Owned. Define the type in storage as Owned<PrivateImmutable<Field, Context><..., Context>>, Context>. ┌─ std/panic.nr:8:12 │ 8 │ assert(false, message); │ ----- Assertion failed │ = Call stack: 1: PanicOnOwnedStateVarInStorage::#[storage] at src/main.nr:8:5 2: storage <omitted full call stack here> Aborting due to 3 previous errors ``` Bringing back the error bootstrap will succeed again. 6. Extra error ``` ✗ contracts/panic_on_owned_state_var_in_storage/: error count mismatch — expected 4, got 3 Expected lines: Type PrivateImmutable<Field, Context> implements OwnedStateVariable and hence cannot be placed in Storage struct without being wrapped in Owned. Define the type in storage as Owned<PrivateImmutable<Field, Context><..., Context>>, Context>. Could not resolve 'init' in path Type annotation needed Type annotation needed Actual error headlines: Type PrivateImmutable<Field, Context> implements OwnedStateVariable and hence cannot be placed in Storage struct without being wrapped in Owned. Define the type in storage as Owned<PrivateImmutable<Field, Context><..., Context>>, Context>. Could not resolve 'init' in path Type annotation needed Full stderr: error: Type PrivateImmutable<Field, Context> implements OwnedStateVariable and hence cannot be placed in Storage struct without being wrapped in Owned. Define the type in storage as Owned<PrivateImmutable<Field, Context><..., Context>>, Context>. ┌─ std/panic.nr:8:12 │ 8 │ assert(false, message); │ ----- Assertion failed │ = Call stack: <omitted full call stack here> Aborting due to 3 previous errors ``` 7. Valid number of errors but wrong error I swapped the two errors in `panic_on_owned_state_var_in_storage`, where we got: ``` ✗ contracts/panic_on_owned_state_var_in_storage/: error 2 does not match Expected: Type annotation needed Actual: Could not resolve 'init' in path ``` 8. `ACCEPT_SNAPSHOTS` path: Remove the function names from `contracts/reserved_public_dispatch/expected_error` and `contracts/reserved_emit_public_init_nullifier/expected_error` (e.g., `Function name '' is reserved for internal use`). ``` ACCEPT_SNAPSHOTS=1 ./bootstrap.sh test ``` Afterwards the `expected_error` for both tests will contain the appropriate error with the name (e.g., `Function name 'public_dispatch' is reserved for internal use`). All the invariants of running the script above resulted as expected. --------- Co-authored-by: Nicolas Chamo <nicolas@chamo.com.ar>
Collaborator
|
✅ Successfully backported to backport-to-v4-next-staging #22763. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves https://linear.app/aztec-labs/issue/F-577/unified-noir-contract-error-snapshot-testing
Summary:
noir-projects/noir-contracts-comp-failures/contracts/noir-projects/aztec-nr/macro_compilation_failure_tests/+ both invocations innoir-projects/aztec-nr/bootstrap.sh.noir-projects/noir-contracts-comp-failures/bootstrap.sh):error:headlines nargo emits (prefix stripped, in emission order) and the non-blank lines ofexpected_error. Any difference in text, count, or order fails the test.ACCEPT_SNAPSHOTS=1 ./bootstrap.sh testauto-extracts everyerror:headline and writes them one-per-line intoexpected_error. A⚠fallback writes full stderr when nargo emits noerror:lines (ICE/panic), prompting a manual edit.expected_errorfile manuallyACCEPT_SNAPSHOTS=1+CI=1is refused, so committed snapshots and live stderr can only diverge when a dev regenerates locally and commits.testaccepts an optional contract-name glob to run a subset:./bootstrap.sh test reserved_public_dispatchor./bootstrap.sh test 'panic_on_*'. No-arg behaviour is unchanged;test_cmds(CI wiring) still emits the no-arg form. Composes withACCEPT_SNAPSHOTS=1for targeted snapshot updates.expected_errorfiles changed):duplicate_storagenow pins'Storage', got 'Storage2' instead.,event_selector_collisionpins both colliding event names,marked_*_unconstrainedpins the function name).error:lines that previous single-line trims were hiding (Could not resolve 'init'/'ZERO' in path,Type annotation needed). Kept them in the snapshot. This is part of the actual compiler behavior being asserted.Verification (done manually):
I replaced the message in
contracts/reserved_public_dispatch/expected_errorwith garbageChange reserved_public_dispatch to
contract ReservedPublicDispatch {}so that it succeeds compilation.Removed "Could not resolve 'init' in path" from panic_on_owned_state_var_in_storage. Running bootstrap we get the following:
Bringing back the error bootstrap will succeed again.
I swapped the two errors in
panic_on_owned_state_var_in_storage, where we got:ACCEPT_SNAPSHOTSpath:Remove the function names from
contracts/reserved_public_dispatch/expected_errorandcontracts/reserved_emit_public_init_nullifier/expected_error(e.g.,Function name '' is reserved for internal use).Afterwards the
expected_errorfor both tests will contain the appropriate error with the name (e.g.,Function name 'public_dispatch' is reserved for internal use).All the invariants of running the script above resulted as expected.