harden various operators and add tests for large operands

[arvidn]

The main parts of this PR are:

• move cost checks earlier, before computation starts
• add a test that pass large buffers to operators

---

Note

Medium Risk
Touches CLVM operator cost accounting and evaluation order; incorrect cost math or earlier failures could change consensus-critical behavior and reject/accept programs differently.

Overview
Tightens cost accounting across several operators to charge per-argument bytes immediately and fail fast on max_cost, reducing risk of expensive computation or allocation before cost is enforced. This adjusts concat/arith/mul/sha cost calculations (including op_unknown’s concat mode) and adds max_cost enforcement to op_pubkey_for_exp.

Adds parameterized stress tests (via rstest) that build very large atoms and assert expected CostExceeded/OutOfMemory behavior across many ops. Separately hardens TreeCache pair serialized_length computation by using saturating addition to avoid overflow.

^{Written by Cursor Bugbot for commit b4eb8d3. This will update automatically on new commits. Configure here.}

[coveralls-official]

Pull Request Test Coverage Report for Build 21491820903

Details

• 46 of 46 (100.0%) changed or added relevant lines in 2 files are covered.
• 4 unchanged lines in 1 file lost coverage.
• Overall coverage increased (+0.08%) to 90.621%

---

Files with Coverage Reduction	New Missed Lines	%
src/more_ops.rs	4	98.83%

Totals
Change from base Build 21360650640:	0.08%
Covered Lines:	6425
Relevant Lines:	7090

---

💛 - Coveralls

[richardkiss]

Seems reasonable. I didn't look too closely at whether semantics or costs changed since I know we have a lot of tests for those.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable Autofix in the Cursor dashboard.}