Prysm, which is an Ethereum consensus client, revealed that auditors missed rewards amounting to 382 ETH or over 1 million USD after a software error affected the network during the recent Fusaka upgrade.
This incident, as detailed in the post-incident report titled 'Fusaka Mainnet Prysm incident,' was caused by excessive resource usage on the system, affecting nearly every node of Prysm and leading to block loss and attestation.
What caused Prysm to crash?
According to data from Offchain Labs, which develops Prysm, this issue began on December 4, when a previously added software error caused delays in requests from validators.
This delay resulted in missing blocks and attestations throughout the network.
The team explained that Prysm beacon nodes received attestations from nodes that may not be synced with the network. These attestations reference the block root from the previous epoch.
This anomaly led to the loss of a total of 41 epochs and 248 missing blocks out of 1,344 slots, resulting in a slot loss rate of 18.5% and reducing network participation to 75% during the incident.
Offchain Labs indicated that the bug responsible for this behavior was added and tested in the testnet about 1 month before it occurred on the mainnet after the Fusaka upgrade.
Although the temporary fix helped mitigate the immediate impact, Prysm stated that it has permanently changed the logic for attestation verification to prevent such incidents from occurring again.
The diversity of clients in the Ethereum network
However, the incident raised concerns about the centralization of Ethereum clients and the risks of relying on a single software provider.
Offchain Labs stated that if Prysm captures more of Ethereum's validator base, this incident could have more severe consequences, highlighting client diversity as a key factor in preventing widespread network outages.
If any client dominates more than 1 in 3 of the network, there could be temporary halts in finality and increased block losses, while clients with more than two-thirds could lead to incorrect chain finalization.
Even with a temporary fix, this incident has accelerated calls for increased client diversity.
Data from Miga Labs shows that Lighthouse remains the Ethereum consensus client with the most validators, accounting for 51.39%, while Prysm stands at 19.06%, followed by Teku at 13.71% and Nimbus at 9.25%.
Lighthouse's share is just below the threshold that some researchers consider risky to the overall system, at around 15%.
Therefore, developers and ecosystem participants have urged validators to consider switching to alternative clients to help mitigate the risks that vulnerabilities from a single software could impact the core operations of the blockchain.
