AttesterSlashings causes `gc` spiked on mainnet nodes

[twoeths]

Describe the bug

This is on the stable and unstable mainnet node, since v1.12.0 Nov 14 gc rate increased significantly

Epoch transition increased at the same time

However it's significantly better on the 1k and md16 md64 goerli node

Expected behavior

gc rate is the same or less compared to v1.11.x, or give an explanation on the difference

Steps to reproduce

No response

Additional context

No response

Operating system

Linux

Lodestar version or commit hash

v1.12.0

[twoeths]

Update: need to ignore 1k, md16, md64 nodes due to the spiked epoch transition time issue #6063

[twoeths]

This may not link to v1.12.0, could be because node processes more AtterterSlashing messages recently on mainnet. Found this on a node of 64 validators:

This links to the recent high AttesterSlashing messages

note that AtterterSlashing uses BigInt which may cause memory and performance issue to our node, see #5892

[twoeths]

also the gc increased at the same time

[twoeths]

Confirm that there are a lot of validators get slashed on Nov 14 https://beaconcha.in/validators/slashings

[twoeths]

node wasn't able to recover since Nov 14

[wemeetagain]

@twoeths can this be closed?