Ethereum consensus client Prysm states that validators missed 382 ETH, equivalent to more than $1 million, after a software bug led to network disruptions shortly after the latest Fusaka upgrade.
The incident, described in a postmortem titled “Fusaka Mainnet Prysm incident,” stemmed from a resource exhaustion that affected nearly all Prysm nodes and led to blocks and attestations being missed.
What caused Prysm's outage?
According to Offchain Labs, the company behind Prysm, the problem arose on December 4 when a previously introduced bug caused delays in validator requests.
These delays led to missed blocks and attestations across the network.
“Prysm beacon nodes received attestations from nodes that may have been out of sync with the network. These attestations referred to a block root from the previous epoch,” the project explained.
The disruption led to 41 missed epochs, with 248 blocks missing out of 1344 available. This represented a miss rate of 18.5% and reduced overall network participation to 75% during the incident.
Offchain Labs states that the bug that caused this behavior was introduced and rolled out on the testnet about a month earlier, before it was triggered on the mainnet after the Fusaka upgrade.
Although a temporary solution reduced the immediate impact, Prysm states that they have now implemented permanent changes in their attestation validation logic to prevent recurrence.
Ethereum client diversity
At the same time, the outage has led to increased focus on the concentration of Ethereum clients and the risks associated with software monocultures.
Offchain Labs states that the outage could have had more serious consequences if Prysm had constituted a larger share of Ethereum's validator base. The company points out that Ethereum's client diversity is a crucial factor in preventing a larger network failure.
“A client with more than 1/3 of the network would have caused a temporary loss of finality and several missed blocks. A buggy client with over 2/3 could have made an invalid chain final,” they stated.
Even with this mitigation, the incident has led to stronger calls for increased client diversity.
Data from Miga Labs shows that Lighthouse remains the dominant Ethereum consensus client, accounting for 51.39% of validators. Prysm makes up 19.06%, followed by Teku at 13.71% and Nimbus at 9.25%.
Lighthouse's share places it about 15 percentage points away from a threshold that some researchers consider to be a systemic risk.
As a result, developers and others in the ecosystem have once again urged validators to consider switching to alternative clients to reduce the likelihood that a single software bug could disrupt the core functions of the blockchain.
