[mainnet] Epoch Transition time spiked

[twoeths]

Describe the bug

On a mainnet node, epoch transition time spiked and never come back

other nodes were restarted and this time got back to normal

Expected behavior

Epoch transition on mainnet node should be consistently < 2s

Steps to reproduce

No response

Additional context

No response

Operating system

Linux

Lodestar version or commit hash

v1.11 and unstable

[twoeths]

this also happens on stable 1k validator node where rss spiked (note that heap memory slowly increased) which caused gc time spiked and epoch transition time spiked

[twoeths]

Some more observation on the issue:

• It only happens with nodes that have validators
• It tends to happen when we have a wasted epoch transition, we populated 3 checkpoint states per epoch at some points
• large_object in V8 memory came up and never come back

this links to the finding in #6096

[twoeths]

haven't seen this issue for a while and code changed a lot, closing for now