Skip to content

improve scaling of new clvm compression implementation#707

Merged
arvidn merged 3 commits intomainfrom
block-builder
Feb 23, 2026
Merged

improve scaling of new clvm compression implementation#707
arvidn merged 3 commits intomainfrom
block-builder

Conversation

@arvidn
Copy link
Copy Markdown
Contributor

@arvidn arvidn commented Feb 23, 2026

at the expense of not always finding the best compression.

If the number of parents aren't capped, that list can become very long and very expensive to maintain and traverse. Not having access to every parent when finding a path is not optimal, but I believe this will give a good result most of the time, in the common cases. It definitely makes the algorithm scale a lot better.

The incremental serializer compares against node_to_bytes_backrefs():

Benchmarking many_spends/node_to_bytes_backrefs: Warming up for 1.0000 ns
Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 775.1s.
many_spends/node_to_bytes_backrefs
                        time:   [64.284 s 67.147 s 70.449 s]
Benchmarking many_spends/incremental_serializer: Warming up for 1.0000 ns
Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 10.0s.
many_spends/incremental_serializer
                        time:   [942.38 ms 952.31 ms 962.57 ms]
                        No change in performance detected.

Note

Medium Risk
Changes core compression path selection by dropping some parent edges, which can subtly affect compression ratios and edge-case behavior even though the logic remains bounded and well-scoped.

Overview
Improves scalability of the new incremental CLVM compression by capping TreeCache parent links per node (MAX_PARENTS = 8) and evicting older entries, trading optimal back-reference selection for bounded memory/CPU during find_path().

Adds a Criterion benchmark (many_spends) to stress incremental serialization on large spend-lists and removes the now-unneeded serializer-cmp fuzz target. CI wheel builds are also stabilized by pinning maturin to 1.12.2 (avoids --sdist artifact changes in 1.12.3).

Written by Cursor Bugbot for commit dd3679c. This will update automatically on new commits. Configure here.

@arvidn arvidn requested a review from richardkiss February 23, 2026 00:09
Comment thread fuzz/fuzz_targets/serializer_cmp.rs Outdated
@coveralls-official
Copy link
Copy Markdown

coveralls-official bot commented Feb 23, 2026

Pull Request Test Coverage Report for Build 22324685918

Details

  • 13 of 15 (86.67%) changed or added relevant lines in 1 file are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage decreased (-0.01%) to 88.222%

Changes Missing Coverage Covered Lines Changed/Added Lines %
src/serde/tree_cache.rs 13 15 86.67%
Totals Coverage Status
Change from base Build 22176657628: -0.01%
Covered Lines: 6831
Relevant Lines: 7743

💛 - Coveralls

Comment thread src/serde/tree_cache.rs
Comment thread src/serde/tree_cache.rs
…e of not always finding the best compression
@arvidn
Copy link
Copy Markdown
Contributor Author

arvidn commented Feb 23, 2026

@cursor review

Comment thread src/serde/tree_cache.rs
Comment thread .github/workflows/build-test.yml Outdated
Comment thread src/serde/tree_cache.rs
Comment thread fuzz/Cargo.toml
@arvidn arvidn force-pushed the block-builder branch 2 times, most recently from 5de8926 to 8abf1c3 Compare February 23, 2026 16:26
Copy link
Copy Markdown

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable Autofix in the Cursor dashboard.

Comment thread .github/workflows/build-test.yml Outdated
Comment thread .github/workflows/build-test.yml Outdated
@richardkiss
Copy link
Copy Markdown
Contributor

Okay, I think I see more or less how it works. You're evicting a random parent so there are fewer things to look at. There might be a way to smartly evict potential parents that could never be reached. But also, I think this serialization format is a dead-end: serde 2026 is way better, both faster and better compression, and dead-simple now that interning is implemented. I have a draft PR here #708

richardkiss
richardkiss previously approved these changes Feb 23, 2026
@arvidn arvidn enabled auto-merge February 23, 2026 20:59
Comment thread .github/workflows/build-test.yml Outdated
Co-authored-by: Richard Kiss <him@richardkiss.com>
@arvidn arvidn merged commit 114377b into main Feb 23, 2026
31 checks passed
@arvidn arvidn deleted the block-builder branch February 23, 2026 21:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants