feat(consensus): assume-valid swift sync aggregator

[JoseSK999]

Description and Notes

• Implement SwiftSyncAgg, which is made of two u64 values. Each of the two limbs is the result of a single 128-bit SipHash24 key. The secret salt is the 32-byte pair of SipHash24 keys.
• Implement verify_block_transactions_swiftsync and process_block_swiftsync for AssumeValid SwiftSync validation.
• New check_block function to reduce duplication (merkle root, BIP34 height, witness commitment, max weight).
• New max_supply_at_height function required for SwiftSync supply validation.
• Add the first 176 mainnet blocks in order to test the SwiftSync aggregator and supply.
• Rename old blocks.txt to regtest_blocks.txt.
• Implement Consensus::is_unspendable and Consensus::total_out_value.

The aggregator addition/subtraction is implemented as wrapping arithmetic with the result of SipHash24(txid || vout_le), with each of the two keys and limbs. In my old computer one OutPoint hash takes ~60ns (two SipHash24 hashes).

For outputs I've optimized hashing to reuse the midstate after processing the txid, across vouts.

[rustaceanrob]

Is there a particular reason SHA256 was used here? Another option would be Siphash24, which can be salted with random k0 and k1 values at startup. I only point this out as SHA256 has a lower throughput than Siphash24, which effects performance given the number of hashes being performed here. The Bitcoin Core reference implementation I implemented also uses Siphash24 via SaltedOutpointHasher.

Some results from a local benchmark of rust-bitcoin

sha256/engine_input/10  time:   [7.3749 ns 7.3926 ns 7.4070 ns]
                        thrpt:  [1.2574 GiB/s 1.2598 GiB/s 1.2628 GiB/s]
sha256/engine_input/1024
                        time:   [497.33 ns 497.40 ns 497.48 ns]
                        thrpt:  [1.9170 GiB/s 1.9173 GiB/s 1.9176 GiB/s]

siphash24/hash_with_keys/1k
                        time:   [331.07 ns 331.18 ns 331.30 ns]
                        thrpt:  [2.8786 GiB/s 2.8796 GiB/s 2.8806 GiB/s]
Found 2 outliers among 100 measurements (2.00%)
  2 (2.00%) low severe
siphash24/hash_to_u64_with_keys/1k
                        time:   [331.27 ns 331.44 ns 331.63 ns]
                        thrpt:  [2.8757 GiB/s 2.8773 GiB/s 2.8789 GiB/s]
Found 4 outliers among 100 measurements (4.00%)
  4 (4.00%) high mild

[JoseSK999]

Hi @rustaceanrob, thanks for pointing this out! Are the 64 bits of output length good enough or are you doing something like hashing it 4 times (with a counter) to get the four 64-bit limbs?

I guess the output length is not that relevant here since it's keyed with 128 bits anyway

[rustaceanrob]

As long as the salt is randomized I think 64 bits is sufficient, also considering this is an assume-valid implementation. SHA256 is certainly more secure, but I just wanted to point out the performance difference if sync speed is the priority for users.

[luisschwab]

SHA256 is certainly more secure, but I just wanted to point out the performance difference if sync speed is the priority for users.

I don't think this would make a difference in practice, since the bottleneck becomes fetching blocks from P2P instead of waiting on the CPU.

[JoseSK999]

So I have done some benches on my old computer and this would be the OutPoint hashing time in a block with 20,000 hinted-as-spent outputs and 20,000 inputs:

• SHA256: ~14.4ms (360ns per OutPoint).
• Two SipHash24 digests: ~2.3ms (57ns per OutPoint).
• One SipHash24 digest: ~1.2ms (30ns per OutPoint).

Either way we will finish much faster than the blocks arrive (we can also parallelize), so I agree with @luisschwab.

But SipHash24 is nice as it's designed to be keyed. I have implemented this with an ugly 19-bytes salt, so that SHA256 runs in one compression round. But we can keep the agg with 128 bits, while actually using a salt of 32 bytes: two SipHash24 keys. That's perhaps a nicer approach?

[luisschwab]

Indeed, I'd go with SipHash24. And it will be more efficient when it comes to CPU cycles and power consumption, even if minimally.

[JoseSK999]

Updated my comment, a better benchmark shows that two SipHash24 run 6x faster than a single SHA256 for me, and only one hash is 12x.

[rustaceanrob]

Perhaps pedantic, but I would not expect the + operator to wrap.

[JoseSK999]

Since we add two SwiftSyncAggs here (not integers), I think this signals the logical operation of adding/combining them. IMO it's reasonable.

[JoseSK999]

Pushed 99ac86a

• Made SwiftSyncAgg::zero() a const fn as suggested by @rustaceanrob
• Changed the OutPoint hash method to SipHash24 instead of SHA256 (thanks @rustaceanrob). We use two u64 now (instead of the single u128), which is more natural. Each limb is a half-aggregator for the same outpoints resulting from a 128-bit SipHash24 key. The two keys are our 32-byte secret salt now (we increase secret size from 19 to 32 bytes, while reducing CPU time by a factor of >6x).
• On top of that I've optimized the outputs OutPoint hashing to reuse the SipHash24 midstate after processing the txid, across vouts. With enough spent outputs this means ~57% less work, as we amortize the cost of the shared txid.
• Implement Consensus::is_unspendable and Consensus::total_out_value.

[JoseSK999]

Thanks to the SipHash24 midstate for txid, even tho we compute two SipHash24 hashes, for transactions with >4 spent outputs we would be doing almost the same CPU work as with a single hash (in my case 35ns vs 30ns).

[JoseSK999]

Derived Default for SipHashKeys and added a test for Consensus::total_out_value

[luisschwab]

@moisesPompilio has a fat finger?

[moisesPompilio]

@moisesPompilio has a fat finger?

Sorry, I pressed the wrong button.

[Davidson-Souza]

It should ask you if you're sure about it. Terrible UX, specially since the Close button is bigger than the comment one. I usually don't have problem with it because I use CTR + Enter, but it's easy to press close by mistake.

[JoseSK999]

My agg code is not that terrible 😅