refactor: unify contract compilation pipeline via bb aztec_process#22590
refactor: unify contract compilation pipeline via bb aztec_process#22590benesjan merged 1 commit intomerge-train/fairiesfrom
Conversation
Both the bootstrap scripts and the `aztec compile` CLI independently called the transpiler and stripped `__aztec_nr_internals__` prefixes from contract artifacts. This duplication was brittle and a source of tech debt. Move prefix stripping into `bb aztec_process` (which already handles transpilation and VK generation), then simplify both callers to use that single command. This eliminates the manual VK generation shell pipeline (~50 lines of bash), the separate strip script, and the TypeScript stripInternalPrefixes function. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
| # Set common flags for parallel. | ||
| export PARALLEL_FLAGS="-j${PARALLELISM:-16} --halt now,fail=1 --memsuspend $(memsuspend_limit)" | ||
|
|
||
| # This computes a vk and adds it to the input function json if it's private, else returns same input. |
There was a problem hiding this comment.
This is a lot of bootstrap code removed. Have we lost per-vk CI caches? will this impact bootstrap performance?
There was a problem hiding this comment.
No meaningful perf loss — likely a slight improvement.
The old per-VK cache was actually redundant with the outer contract cache:
Old flow — two caches, both keyed by $contract_hash:
- Outer
contract-$contract_hash.tar.gz: compiled+transpiled JSON without VKs - Inner
vk-$contract_hash-$funchash.tar.gz: one per private function
The per-VK key included $contract_hash as a namespace, so it could only hit when $contract_hash matched a prior run — i.e., exactly the same scenarios where the outer cache also hit. Cross-contract_hash VK reuse was never possible. The per-VK cache was only necessary because the outer cache didn't contain VKs.
New flow — one cache, same key:
contract-$contract_hash.tar.gznow contains the fully processed JSON including VKs (produced in one shot bybb aztec_process).
Perf comparison:
- Cold build (contract_hash changed): equivalent work in both. New path uses C++ threads with nested parallelism in
generate_vks_for_functionsinstead of GNU parallel shelling out per function — likely a bit faster with less overhead. - Warm build: strictly faster — one tarball download instead of 1 + N.
- Bonus: there's a new local VK cache at
~/.bb/$BB_VERSION/vk_cache/<bytecode_hash>.vkkeyed on pure bytecode hash (nocontract_hashnamespace), giving dev-machine reuse across contracts that the old flow didn't have.
The only theoretical regression would be if the old per-VK cache were keyed on just $funchash (enabling cross-contract_hash reuse) — but it wasn't, so nothing's lost.
mverzilli
left a comment
There was a problem hiding this comment.
LGTM, though please note I currently have very little context on this code
|
❌ Failed to cherry-pick to |
This commit records the raw output of git cherry-pick 9c8bb7b for PR #22590 (refactor: unify contract compilation pipeline via bb aztec_process). Conflicts left intact in this commit: - noir-projects/noir-contracts/bootstrap.sh — content conflict - noir-projects/protocol-fuzzer/SANDBOX_INSTRUCTIONS.md — modify/delete (deleted in HEAD) - noir-projects/protocol-fuzzer/setup-nightly-sandbox.sh — modify/delete (deleted in HEAD) Subsequent commits resolve the conflicts and fix the build.
- noir-contracts/bootstrap.sh: accept incoming removal of the legacy process_function VK-generation block. 'bb aztec_process' now handles VK generation directly, so the tmp_dir / jq / parallel plumbing that was still present on v4-next is no longer needed. - protocol-fuzzer/SANDBOX_INSTRUCTIONS.md and protocol-fuzzer/setup-nightly-sandbox.sh: drop. These files were introduced in a commit that landed on next after v4-next diverged, so they never existed on v4-next. PR #22590 only touched them to stay consistent with that later work; since v4-next doesn't have them, keep them absent.
PR #22590 removed 'mkdir -p $tmp_dir' from bootstrap.sh's build function. On next that's fine because next's bootstrap.sh has no pinned-contracts logic. v4-next however still extracts pinned-protocol-contracts.tar.gz right after 'rm -rf target', so without the implicit target/ creation the tar step fails with: tar: target: Cannot open: No such file or directory Add an explicit 'mkdir -p target' after the rm -rf to restore the v4-next build.
…ackport #22590) (#22634) Backport of #22590 to v4-next. ## Original PR summary Moves `__aztec_nr_internals__` prefix stripping into `bb aztec_process` (C++), so contract compilation collapses from four steps (`nargo compile` → `transpile` → `strip prefixes` → `write_vk`) to two (`nargo compile` → `bb aztec_process`). This removes the duplicated pipeline between `noir-projects/noir-contracts/bootstrap.sh` and `aztec compile` (CLI), deletes `strip_aztec_nr_prefix.sh`, and removes the TypeScript `stripInternalPrefixes()` helper. ## Backport structure (3 commits) 1. **`chore: cherry-pick PR #22590 (conflicts unresolved)`** — raw `git cherry-pick` output with conflict markers, preserved for review. 2. **`fix: resolve cherry-pick conflicts for PR #22590`** — resolves only the conflict markers. 3. No build-fix commit was needed — bash syntax, grep for dangling references, and the already-present prefix-stripping logic in `aztec_process.cpp` all check out. ## Conflicts resolved - **`noir-projects/noir-contracts/bootstrap.sh`** — content conflict. v4-next still had the legacy `process_function` block plus `tmp_dir` / `jq` / `parallel` plumbing for VK generation. Accepted the incoming PR's removal since `bb aztec_process` now handles VK generation directly. - **`noir-projects/protocol-fuzzer/SANDBOX_INSTRUCTIONS.md`** and **`noir-projects/protocol-fuzzer/setup-nightly-sandbox.sh`** — modify/delete. These files were introduced on `next` after v4-next diverged (commit `5b0c0e166e`, "feat(fuzz): protocol fuzzer with bridge server and parallel batching") and never existed on v4-next. PR #22590 only touched them to stay consistent with that later work, so they're dropped here. ## Diff vs. original PR | | Original PR | This backport | |---|---|---| | Files | 10 | 8 | | Insertions | 42 | 38 | | Deletions | 160 | 142 | The size delta is entirely from the two protocol-fuzzer files that don't exist on v4-next. ## Verification - `bash -n` on all modified shell scripts: OK - No dangling references to `strip_aztec_nr_prefix`, `STRIP_AZTEC_NR_PREFIX`, `stripInternalPrefixes`, or the removed protocol-fuzzer files anywhere in the tree - `__aztec_nr_internals__` prefix-stripping logic is present in `barretenberg/cpp/src/barretenberg/api/aztec_process.cpp` (applied cleanly from the PR) ClaudeBox log: https://claudebox.work/s/0347afb057f4542d?run=1
BEGIN_COMMIT_OVERRIDE fix(pxe): stop block synchronizer on PXE shutdown (#22604) fix(aztec): respect TEST_ACCOUNTS env var in local network mode (#22600) fix: check all aztec-nr dependency tags, not just aztec (#22483) fix: reuse anchor block in kernel oracle (#22631) refactor: unify contract compilation pipeline via bb aztec_process (#22590) fix(pxe): queue registerSender wipe to avoid racing with in-flight jobs (#22623) fix(pxe): bounds-check PropertySelector in pick_notes (#22614) fix(pxe): guard private event store rollback against in-flight jobs (#22615) refactor(pxe): rename sideEffectCounter to initialSideEffectCounter (#22599) fix(pxe): correct stale authwitness comment and inverted tagging error (#22537) fix(aztec-nr): range-check auth witness fields before byte cast (#22624) fix(pxe): serialize block stream event handling to prevent race conditions (#22635) fix(pxe): throw error on origin/contract address mismatch in simulation (#22637) END_COMMIT_OVERRIDE
When working on another PR of mine I realized that we are having a problem and that is that we have 2 paths by which contracts get compiled:
aztec compileThis started to become problematic as our contract compilation pipeline was getting more complicated:
This meant that the duplication started to have real cost. In this PR I clean this up a bit by moving the prefix stripping into the
bb aztec_processcommand. This results that the compilation pipeline is simply:nargo compile,bb aztec_processGiven that it's just 2 commands it is fine to have it be duplicated between
bootstrap.shandaztec compile. Once this is merged I will try to refactor #22550 such that the version is also injected only inbb aztec_process.AI Summary
__aztec_nr_internals__prefix stripping intobb aztec_process(C++), which already handles transpilation and VK generationcompilefunction to just callnargo compile+bb aztec_process, removing ~50 lines of manual VK generation shell codestripInternalPrefixes()TypeScript function from the CLI pathstrip_aztec_nr_prefix.shscriptBoth compilation paths (bootstrap and
aztec compile) now go throughbb aztec_processfor all post-nargo processing, eliminating the duplication.