computeNewStateRoot performance issue at slot 0 of epoch

[twoeths]

Describe the bug

At slot 0 of epoch, computeNewStateRoot somehow has a very big hashTreeRoot() time

produce_block_holesky.cpuprofile.zip

This happens a long time ago since #5793 (comment)

Expected behavior

Fast hashTreeRoot() call in computeNewStateRoot() as the pre state there should cache a lot of hash objects already

Steps to reproduce

No response

Additional context

No response

Operating system

Linux

Lodestar version or commit hash

v1.12.1

[twoeths]

the issue was reproduced with tuyen/produce_block_interop_key, could use this branch to reproduce other produce block issues later

[twoeths]

was able to catch a block like this on holesky where it takes 3.56s to produce a block

and it's exactly at slot 0 of epoch (slot=584288)

2023-12-18 22:37:40.525 | Dec-18 15:37:40.524[]                 info: Published block slot=584288, validator=0x8a02…d776, graffiti=Lodestar-v1.12.1/003a444, builderSelection=maxprofit, api=produceBlindedBlock |  
-- | -- | --
  |   | 2023-12-18 22:37:39.605 | Dec-18 15:37:39.605[]                debug: Produced block slot=584288, validator=0x8a020967f7a4e1f1b23491abc26d3e0dad0301dfe48ea75ac9d929728db4fdbeadc886f977bf6a629a31afa60357d776, builderSelection=maxprofit, api=produceBlindedBlock |  
  |   | 2023-12-18 22:37:39.601 | Dec-18 15:37:39.601[]                debug: HttpClient response routeId=produceBlindedBlock |  
  |   | 2023-12-18 22:37:39.599 | Dec-18 15:37:39.599[rest]            debug: Res req-fnca produceBlindedBlock - 200 |  
  |   | 2023-12-18 22:37:36.139 | Dec-18 15:37:36.139[rest]            debug: Req req-fnca 127.0.0.1 produceBlindedBlock |  
  |   | 2023-12-18 22:37:36.037 | Dec-18 15:37:36.037[]                debug: HttpClient request routeId=produceBlindedBlock