Skip to content

fix(ci): Unify noir contract compilation error snapshot testing#22733

Merged
vezenovm merged 9 commits intomerge-train/fairiesfrom
mv/f-577-unified-noir-contract-error-snapshot-testing
Apr 24, 2026
Merged

fix(ci): Unify noir contract compilation error snapshot testing#22733
vezenovm merged 9 commits intomerge-train/fairiesfrom
mv/f-577-unified-noir-contract-error-snapshot-testing

Conversation

@vezenovm
Copy link
Copy Markdown
Contributor

@vezenovm vezenovm commented Apr 22, 2026

Resolves https://linear.app/aztec-labs/issue/F-577/unified-noir-contract-error-snapshot-testing

Summary:

  • 4 cases moved into noir-projects/noir-contracts-comp-failures/contracts/
  • Old runner deleted: noir-projects/aztec-nr/macro_compilation_failure_tests/ + both invocations in noir-projects/aztec-nr/bootstrap.sh.
  • Runner extended (noir-projects/noir-contracts-comp-failures/bootstrap.sh):
    • Matcher asserts strict equality, line-for-line, between the error: headlines nargo emits (prefix stripped, in emission order) and the non-blank lines of expected_error. Any difference in text, count, or order fails the test.
    • ACCEPT_SNAPSHOTS=1 ./bootstrap.sh test auto-extracts every error: headline and writes them one-per-line into expected_error. A fallback writes full stderr when nargo emits no error: lines (ICE/panic), prompting a manual edit.
      • Devs no longer have to write their own expected_error file manually
    • CI guard: ACCEPT_SNAPSHOTS=1 + CI=1 is refused, so committed snapshots and live stderr can only diverge when a dev regenerates locally and commits.
    • test accepts an optional contract-name glob to run a subset: ./bootstrap.sh test reserved_public_dispatch or ./bootstrap.sh test 'panic_on_*'. No-arg behaviour is unchanged; test_cmds (CI wiring) still emits the no-arg form. Composes with ACCEPT_SNAPSHOTS=1 for targeted snapshot updates.
  • Snapshots regenerated (13 expected_error files changed):
    • Added specificity where previous dropped by manual trims (e.g. duplicate_storage now pins 'Storage', got 'Storage2' instead., event_selector_collision pins both colliding event names, marked_*_unconstrained pins the function name).
    • Several cases gained cascading downstream error: lines that previous single-line trims were hiding (Could not resolve 'init'/'ZERO' in path, Type annotation needed). Kept them in the snapshot. This is part of the actual compiler behavior being asserted.
  • READMEs updated: aztec-nr README; comp-failures README

Verification (done manually):

cd noir-projects/noir-contracts-comp-failures
  1. Happy case:
./bootstrap.sh test
  1. Wrong-error path:
    I replaced the message in contracts/reserved_public_dispatch/expected_error with garbage
./bootstrap.sh test

Gives this output:
...
✓ contracts/panic_on_owned_state_var_in_storage/: Compilation failed as expected with correct error(s)
✓ contracts/public_function_selector_collision/: Compilation failed as expected with correct error(s)
✓ contracts/reserved_emit_public_init_nullifier/: Compilation failed as expected with correct error(s)
✗ contracts/reserved_public_dispatch/: error 1 does not match
  Expected: Function name '' is reserved for internal use
  Actual:   Function name 'public_dispatch' is reserved for internal use
  1. Success-compile path:
    Change reserved_public_dispatch to contract ReservedPublicDispatch {} so that it succeeds compilation.
...
✓ contracts/public_function_selector_collision/: Compilation failed as expected with correct error
✓ contracts/reserved_emit_public_init_nullifier/: Compilation failed as expected with correct error
✗ contracts/reserved_public_dispatch/: Expected compilation to fail but it succeeded
  1. Missing error
    Removed "Could not resolve 'init' in path" from panic_on_owned_state_var_in_storage. Running bootstrap we get the following:
✗ contracts/panic_on_owned_state_var_in_storage/: error count mismatch — expected 2, got 3
Expected lines:
  Type PrivateImmutable<Field, Context> implements OwnedStateVariable and hence cannot be placed in Storage struct without being wrapped in Owned. Define the type in storage as Owned<PrivateImmutable<Field, Context><..., Context>>, Context>.
  Type annotation needed
Actual error headlines:
  Type PrivateImmutable<Field, Context> implements OwnedStateVariable and hence cannot be placed in Storage struct without being wrapped in Owned. Define the type in storage as Owned<PrivateImmutable<Field, Context><..., Context>>, Context>.
  Could not resolve 'init' in path
  Type annotation needed
Full stderr:
error: Type PrivateImmutable<Field, Context> implements OwnedStateVariable and hence cannot be placed in Storage struct without being wrapped in Owned. Define the type in storage as Owned<PrivateImmutable<Field, Context><..., Context>>, Context>.
  ┌─ std/panic.nr:8:12
  │
8 │     assert(false, message);
  │            ----- Assertion failed
  │
  = Call stack:
    1: PanicOnOwnedStateVarInStorage::#[storage]
            at src/main.nr:8:5
    2: storage

<omitted full call stack here>
Aborting due to 3 previous errors

Bringing back the error bootstrap will succeed again.

  1. Extra error
✗ contracts/panic_on_owned_state_var_in_storage/: error count mismatch — expected 4, got 3
Expected lines:
  Type PrivateImmutable<Field, Context> implements OwnedStateVariable and hence cannot be placed in Storage struct without being wrapped in Owned. Define the type in storage as Owned<PrivateImmutable<Field, Context><..., Context>>, Context>.
  Could not resolve 'init' in path
  Type annotation needed
  Type annotation needed
Actual error headlines:
  Type PrivateImmutable<Field, Context> implements OwnedStateVariable and hence cannot be placed in Storage struct without being wrapped in Owned. Define the type in storage as Owned<PrivateImmutable<Field, Context><..., Context>>, Context>.
  Could not resolve 'init' in path
  Type annotation needed
Full stderr:
error: Type PrivateImmutable<Field, Context> implements OwnedStateVariable and hence cannot be placed in Storage struct without being wrapped in Owned. Define the type in storage as Owned<PrivateImmutable<Field, Context><..., Context>>, Context>.
  ┌─ std/panic.nr:8:12
  │
8 │     assert(false, message);
  │            ----- Assertion failed
  │
  = Call stack:

<omitted full call stack here>
Aborting due to 3 previous errors
  1. Valid number of errors but wrong error

I swapped the two errors in panic_on_owned_state_var_in_storage, where we got:

✗ contracts/panic_on_owned_state_var_in_storage/: error 2 does not match
  Expected: Type annotation needed
  Actual:   Could not resolve 'init' in path
  1. ACCEPT_SNAPSHOTS path:
    Remove the function names from contracts/reserved_public_dispatch/expected_error and contracts/reserved_emit_public_init_nullifier/expected_error (e.g., Function name '' is reserved for internal use).
ACCEPT_SNAPSHOTS=1 ./bootstrap.sh test

Afterwards the expected_error for both tests will contain the appropriate error with the name (e.g., Function name 'public_dispatch' is reserved for internal use).

All the invariants of running the script above resulted as expected.

@vezenovm vezenovm requested a review from nventuro as a code owner April 22, 2026 19:27
…pdates, plus snap updates that were previously being removed
@vezenovm vezenovm requested a review from nchamo April 22, 2026 20:12
@vezenovm vezenovm requested a review from dbanks12 April 22, 2026 20:24
Copy link
Copy Markdown
Contributor

@nchamo nchamo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Comment thread noir-projects/noir-contracts-comp-failures/bootstrap.sh Outdated
Comment thread noir-projects/noir-contracts-comp-failures/bootstrap.sh
vezenovm and others added 3 commits April 24, 2026 09:31
Co-authored-by: Nicolas Chamo <nicolas@chamo.com.ar>
)

## Motivation

PR #22733 unified the compile-failure runner and migrated four legacy
cases into `noir-contracts-comp-failures/`, bringing total coverage to
22 contracts. That left ~28 user-reachable `panic!` sites in
`aztec-nr/aztec/src/macros/` with no dedicated test. This PR closes
those gaps so every user-reachable panic has a pinned snapshot.

## Approach

For each gap, a minimal `contracts/<name>/` contract triggers the target
panic and an `expected_error` file pins nargo's output verbatim.
Contracts that would otherwise cascade into an unrelated panic (the
generic `check_each_fn_macroified` message, or the "noinitcheck without
initializer" message) are isolated with a `#[contract_library_method]`
marker or a stub `#[initializer]` so each test pins exactly one target.

## Changes

- **`noir-contracts-comp-failures/contracts/`**: 28 new contracts, one
per previously-uncovered panic site (see diff for details). One contract
(`authorize_once_before_external`) ships with an empty `expected_error`
as a known-unreachable reproducer.
- **Realistic cascading errors**: In addition to the 28 isolated cases,
`contracts/bob_token/` pins the full cascade behavior of the current
library on a realistic program. It's a token contract adapted from the
https://docs.aztec.network/developers/docs/tutorials/contract_tutorials/token_contract
with four broken functions.
- Because the broken functions have realistic bodies that reference
`self` and call each other, the snapshot captures 38 error lines (when
we should only have 4 errors).
- This gives us a baseline to measure against. After
https://linear.app/aztec-labs/issue/F-578/aztec-nr-refactor-macro-validators-to-collect-errors-and-emit-a-dummy
we should expect the snapshot to shrink.
- **`noir-contracts-comp-failures/bootstrap.sh`**: empty
`expected_error` now means "expect compile success";
`ACCEPT_SNAPSHOTS=1` writes an empty file when compilation succeeds.
Wrong count or content falls through to the existing count-mismatch
path.
- **`noir-contracts-comp-failures/README.md`**: one-line note on the
empty-snapshot convention.

## Follow-ups

-
[F-579](https://linear.app/aztec-labs/issue/F-579/authorize-once-attribute-ordering-panic-at-helpersnr55-is-unreachable):
the `#[authorize_once]` attribute-ordering panic at `helpers.nr:55` does
not fire for the documented trigger; the reproducer ships with an empty
snapshot.
-
[F-580](https://linear.app/aztec-labs/issue/F-580/authorization-macro-is-pub-but-has-no-downstream-consumers-and-leaks):
`#[authorization]` is `pub` but has no downstream consumers; its
generated code references `crate::authwit::*` paths that don't resolve
from a downstream crate, producing a 5-line cascade before the target
diagnostic.

Building off of
#22733
@vezenovm vezenovm enabled auto-merge (squash) April 24, 2026 14:08
@vezenovm vezenovm merged commit b34624d into merge-train/fairies Apr 24, 2026
12 checks passed
@vezenovm vezenovm deleted the mv/f-577-unified-noir-contract-error-snapshot-testing branch April 24, 2026 14:29
AztecBot pushed a commit that referenced this pull request Apr 24, 2026
Resolves
https://linear.app/aztec-labs/issue/F-577/unified-noir-contract-error-snapshot-testing

Summary:
- 4 cases moved into
`noir-projects/noir-contracts-comp-failures/contracts/`
- Old runner deleted:
`noir-projects/aztec-nr/macro_compilation_failure_tests/` + both
invocations in `noir-projects/aztec-nr/bootstrap.sh`.
- Runner extended
(`noir-projects/noir-contracts-comp-failures/bootstrap.sh`):
- Matcher asserts **strict equality**, line-for-line, between the
`error:` headlines nargo emits (prefix stripped, in emission order) and
the non-blank lines of `expected_error`. Any difference in text, count,
or order fails the test.
- `ACCEPT_SNAPSHOTS=1 ./bootstrap.sh test` auto-extracts every `error:`
headline and writes them one-per-line into `expected_error`. A `⚠`
fallback writes full stderr when nargo emits no `error:` lines
(ICE/panic), prompting a manual edit.
- Devs no longer have to write their own `expected_error` file manually
- CI guard: `ACCEPT_SNAPSHOTS=1` + `CI=1` is refused, so committed
snapshots and live stderr can only diverge when a dev regenerates
locally and commits.
- `test` accepts an optional contract-name glob to run a subset:
`./bootstrap.sh test reserved_public_dispatch` or `./bootstrap.sh test
'panic_on_*'`. No-arg behaviour is unchanged; `test_cmds` (CI wiring)
still emits the no-arg form. Composes with `ACCEPT_SNAPSHOTS=1` for
targeted snapshot updates.
  - Snapshots regenerated (13 `expected_error` files changed):
- Added specificity where previous dropped by manual trims (e.g.
`duplicate_storage` now pins `'Storage', got 'Storage2' instead.`,
`event_selector_collision` pins both colliding event names,
`marked_*_unconstrained` pins the function name).
- Several cases gained cascading downstream `error:` lines that previous
single-line trims were hiding (`Could not resolve 'init'/'ZERO' in
path`, `Type annotation needed`). Kept them in the snapshot. This is
part of the actual compiler behavior being asserted.
  - READMEs updated: aztec-nr README; comp-failures README

Verification (done _manually_):
```
cd noir-projects/noir-contracts-comp-failures
```

1. Happy case: 
```
./bootstrap.sh test
```

2. Wrong-error path:
I replaced the message in
`contracts/reserved_public_dispatch/expected_error` with garbage
```
./bootstrap.sh test

Gives this output:
...
✓ contracts/panic_on_owned_state_var_in_storage/: Compilation failed as expected with correct error(s)
✓ contracts/public_function_selector_collision/: Compilation failed as expected with correct error(s)
✓ contracts/reserved_emit_public_init_nullifier/: Compilation failed as expected with correct error(s)
✗ contracts/reserved_public_dispatch/: error 1 does not match
  Expected: Function name '' is reserved for internal use
  Actual:   Function name 'public_dispatch' is reserved for internal use
```

4. Success-compile path:
Change reserved_public_dispatch to `contract ReservedPublicDispatch {}`
so that it succeeds compilation.
```
...
✓ contracts/public_function_selector_collision/: Compilation failed as expected with correct error
✓ contracts/reserved_emit_public_init_nullifier/: Compilation failed as expected with correct error
✗ contracts/reserved_public_dispatch/: Expected compilation to fail but it succeeded
```

5. Missing error
Removed "Could not resolve 'init' in path" from
panic_on_owned_state_var_in_storage. Running bootstrap we get the
following:
```
✗ contracts/panic_on_owned_state_var_in_storage/: error count mismatch — expected 2, got 3
Expected lines:
  Type PrivateImmutable<Field, Context> implements OwnedStateVariable and hence cannot be placed in Storage struct without being wrapped in Owned. Define the type in storage as Owned<PrivateImmutable<Field, Context><..., Context>>, Context>.
  Type annotation needed
Actual error headlines:
  Type PrivateImmutable<Field, Context> implements OwnedStateVariable and hence cannot be placed in Storage struct without being wrapped in Owned. Define the type in storage as Owned<PrivateImmutable<Field, Context><..., Context>>, Context>.
  Could not resolve 'init' in path
  Type annotation needed
Full stderr:
error: Type PrivateImmutable<Field, Context> implements OwnedStateVariable and hence cannot be placed in Storage struct without being wrapped in Owned. Define the type in storage as Owned<PrivateImmutable<Field, Context><..., Context>>, Context>.
  ┌─ std/panic.nr:8:12
  │
8 │     assert(false, message);
  │            ----- Assertion failed
  │
  = Call stack:
    1: PanicOnOwnedStateVarInStorage::#[storage]
            at src/main.nr:8:5
    2: storage

<omitted full call stack here>
Aborting due to 3 previous errors
```
Bringing back the error bootstrap will succeed again. 

6. Extra error 
```
✗ contracts/panic_on_owned_state_var_in_storage/: error count mismatch — expected 4, got 3
Expected lines:
  Type PrivateImmutable<Field, Context> implements OwnedStateVariable and hence cannot be placed in Storage struct without being wrapped in Owned. Define the type in storage as Owned<PrivateImmutable<Field, Context><..., Context>>, Context>.
  Could not resolve 'init' in path
  Type annotation needed
  Type annotation needed
Actual error headlines:
  Type PrivateImmutable<Field, Context> implements OwnedStateVariable and hence cannot be placed in Storage struct without being wrapped in Owned. Define the type in storage as Owned<PrivateImmutable<Field, Context><..., Context>>, Context>.
  Could not resolve 'init' in path
  Type annotation needed
Full stderr:
error: Type PrivateImmutable<Field, Context> implements OwnedStateVariable and hence cannot be placed in Storage struct without being wrapped in Owned. Define the type in storage as Owned<PrivateImmutable<Field, Context><..., Context>>, Context>.
  ┌─ std/panic.nr:8:12
  │
8 │     assert(false, message);
  │            ----- Assertion failed
  │
  = Call stack:

<omitted full call stack here>
Aborting due to 3 previous errors
```

7. Valid number of errors but wrong error

I swapped the two errors in `panic_on_owned_state_var_in_storage`, where
we got:
```
✗ contracts/panic_on_owned_state_var_in_storage/: error 2 does not match
  Expected: Type annotation needed
  Actual:   Could not resolve 'init' in path
```

8. `ACCEPT_SNAPSHOTS` path:
Remove the function names from
`contracts/reserved_public_dispatch/expected_error` and
`contracts/reserved_emit_public_init_nullifier/expected_error` (e.g.,
`Function name '' is reserved for internal use`).
```
ACCEPT_SNAPSHOTS=1 ./bootstrap.sh test
```
Afterwards the `expected_error` for both tests will contain the
appropriate error with the name (e.g., `Function name 'public_dispatch'
is reserved for internal use`).

All the invariants of running the script above resulted as expected.

---------

Co-authored-by: Nicolas Chamo <nicolas@chamo.com.ar>
@AztecBot
Copy link
Copy Markdown
Collaborator

✅ Successfully backported to backport-to-v4-next-staging #22763.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants