Ethereum consensus client Prysm states that validators lost 382 ETH, which amounted to more than 1 million USD, after a software bug that caused network disruptions shortly after the latest Fusaka upgrade.
The incident described in the report “Fusaka Mainnet Prysm incident” was due to almost all Prysm nodes running out of resources. This led to missed blocks and attestations in the network.
What caused Prysm’s outage?
According to Offchain Labs, which develops Prysm, the problem began on December 4 when a previous bug caused delays in validator requests.
The delays caused the entire network to miss blocks and attestations.
“Prysm beacon node received attestations from nodes that may not have been synchronized with the network. These attestations referred to a block root from a previous epoch,” explained the project.
The disruption led to 41 epochs being missed, with 248 blocks missing out of 1,344 available. It resulted in a missed slot rate of 18.5% and network participation dropped to 75% during the incident.
Offchain Labs says that the bug that caused the incident was introduced and tested on the testnet about a month ago before it was triggered on the mainnet after the Fusaka upgrade.
A temporary solution reduced the immediate impact, but Prysm has now made permanent changes to the logic of attestation validation to prevent it from happening again.
Ethereum's client diversity
At the same time, the interruption has brought more focus on the concentration of Ethereum clients and the risk of software monocultures.
Offchain Labs points out that it could have been worse if Prysm had a larger share of Ethereum's validator base. The company states that Ethereum's client diversity has helped avoid major network issues.
“A client with more than one-third of the network would cause temporary finality loss and more missed blocks. A bug client with more than two-thirds could have approved an invalid chain,” it states.
Despite this measure, many now want greater client diversity.
Data from Miga Labs shows that Lighthouse is the largest Ethereum consensus client, with 51.39% of validators. Prysm accounts for 19.06%, Teku 13.71%, and Nimbus 9.25%.
Lighthouse's share is about 15% away from a threshold that some researchers assess poses a systemic risk.
Therefore, developers and ecosystem participants urge validators to switch to other clients. This reduces the risk that a single software bug could disrupt the blockchain's core functions.
