The Ethereum consensus client, Prysm, reported that validators lost 382 ETH, equivalent to over 1 million dollars, after a software error caused interruptions on the network shortly after the recent Fusaka update.
The incident, detailed in a report titled 'Fusaka Mainnet Prysm incident', was caused by resource exhaustion that affected nearly all Prysm nodes, resulting in lost blocks and validations.
What caused the Prysm outage?
According to Offchain Labs, the developer behind Prysm, the problem arose on December 4 when a previously introduced bug caused delays in validator requests. These delays resulted in lost blocks and validations across the network.
“Prysm's Beacon nodes received validations from nodes that possibly were not synchronized with the network. These validations referred to the root of a block from the previous epoch,” the project explained.
The disruption led to the loss of 41 epochs, with 248 blocks lost out of 1,344 available slots. This represented a failure rate of 18.5% and reduced the overall network participation to 75% during the incident.
Offchain Labs noted that the bug responsible for this behavior was introduced and deployed in the testnets almost a month earlier, and was activated on the mainnet after the Fusaka upgrade.
Although a temporary solution reduced the immediate impact, Prysm indicated that it has implemented permanent changes to its attestation validation logic to prevent this from happening again.
Ethereum client diversity
Meanwhile, the disruption has renewed attention on the concentration of Ethereum clients and the risks of relying on a single software.
Offchain Labs said that the disruption could have caused more severe consequences if Prysm had a larger share of the Ethereum validator base. The company noted that client diversity in Ethereum was a key factor in preventing a widespread network failure.
“A client with more than 1/3 of the network could have caused a temporary loss of finality and more blocks lost. A faulty client with more than 2/3 could finalize an invalid blockchain,” it noted.
Despite that solution, the incident has increased calls for greater client diversity. Data from Miga Labs shows that Lighthouse remains the dominant consensus client of Ethereum, with 51.39% of validators. Prysm represents 19.06%, followed by Teku with 13.71% and Nimbus with 9.25%.
Lighthouse's participation puts it approximately 15 percentage points away from a threshold that some researchers consider a systemic risk.
Therefore, developers and ecosystem members have once again called for validators to consider switching to other clients to reduce the likelihood that a single software error disrupts the main operations of the blockchain.
