CoinVoice has recently learned that the Prysm team released a mainnet incident review report stating that on December 4th, during the Fusaka period of the Ethereum mainnet, almost all Prysm beacon nodes experienced resource exhaustion when processing specific attestations, leading to an inability to respond to validator requests in a timely manner, resulting in a large number of missing blocks and witnesses.
The incident affected epochs 411439 to 411480, with a total of 42 epochs and 248 blocks missing out of 1344 slots, resulting in a missing rate of approximately 18.5%; the network participation rate once dropped to 75%, and validators lost about 382 ETH in witness rewards. The root cause was that Prysm received attestations from nodes that may have been out of sync with the mainnet, which referenced block roots from the previous epoch.
To verify its legitimacy, Prysm repeatedly replayed old epoch states and executed high-cost epoch transitions, leading to resource exhaustion under high concurrency. The related defect originated from Prysm PR 15965, which was deployed to the testnet over a month ago but did not trigger the same scenario.
The temporary solution provided by the officials is to enable the --disable-last-epoch-target parameter in version v7.0; the subsequently released v7.1 and v7.1.0 include a long-term fix by validating attestations using head state, avoiding the replay of historical states.
Prysm stated that the issue gradually alleviated after December 4 at 4:45 UTC, and by epoch 411480, the network participation rate had recovered to over 95%.
The Prysm team pointed out that this incident highlights the importance of client diversity. If a single client accounts for more than one-third, it may lead to a temporary inability to finalize; if it exceeds two-thirds, there is a risk of invalid finality chains. They also reflected on the unclear communication of feature switches and the testing environment's failure to simulate large-scale unsynchronized nodes, and will subsequently improve testing strategies and configuration management. [Original link]
