A few days after Binance's BTC withdrawal issues because of Bitcoin network congestion, Ethereum's Beacon chain temporarily stopped finalizing transactions – twice a 24h on last Thursday and Friday.
For regular users, the impact was minimal. The only noticeable effect was a slightly longer transaction processing time and a somewhat lower guarantee of finality for about 25 minutes.
The beacon chain did not "halt", to suggest that it did is willfully spreading misinformation. Finalization was delayed for 3 epochs (about 19 minutes), and was recovered after turbulence. End users would have no idea this happened and transactions continued normally.
— superphiz.eth 🦇🔊🛡️ (@superphiz) May 12, 2023
For the developer community, though, the incident was viewed as a significant test of the network's strength.
Ethereum had difficulties in reaching a consensus, resulting in a brief yet severe decline in validator participation on the consensus layer.
“This led to a period of time where Ethereum blocks were not being finalized, but blocks were still being validated as normal on the execution layer,” – said Blockworks Research Senior Analyst.
On Friday, the same problem emerged again, which doesn't seem to surprise either the Ethereum community members…
…or the Bitcoin folks:
Building a complex and secure app on Ethereum. https://t.co/NumUZXHbmb
— Willy Woo (@woonomic) May 14, 2023
But what really happened there?
The precise cause is unknown yet at the time of writing, but it was likely caused by a coincidence of two or three issues. During the incident, validators faced difficulties with Prysm and Teku nodes that were resolved primarily after a restart. A Prysm developer @potuz1 explained (thread) that several valid but untimely attestations were broadcasted in the network, which takes place when a node attests with an outdated view of the chain.
Here you can find a detailed thread about what it looked like from the blockchain perspective:
Hotfixes in both clients involved have now been deployed. As for the lessons learned from the incident, the experts agree that the loss of finality could be prevented entirely if less than a third of all validators would run the same consensus layer software client, while currently, two clients, Prysm and Lighthouse, are estimated to be running on more than a third of Ethereum nodes, as per clientdiversity.org. So it's all about the significance of decentralization again.
Marius van der Wijden, a developer of the Go Ethereum execution layer client for Ethereum, called this incident "a great fire drill" for the network, emphasizing that client diversity was critical to the network's ability to recover from such an event.
