Infura, a hosted node cluster that also powers MetaMask, is suffering a major service outage on their Ethereum Mainnet API as per a notice on Nov 11. The blackout that started at 0812 HRS UTC is affecting ETH and ERC-20 price feeds, forcing some exchanges to halt coin withdrawals.
Identifying the Root Cause
At the time of writing, the Infura team had dispatched their technicians to identify the root cause. They have been successful and have since restored limited traffic on their API “to expedite service recovery” as per a 12:37 PM UTC update.
The service disruption was because of a failure in several components traced to a bug on the stable version of the go-ethereum client at block height 11234873. Geth versions 1.9.9 and 1.9.13 were affected.
The go-ethereum client is one of the three original implementations of the Ethereum protocol. It is a popular software client for running a node on the Ethereum network.
“The root cause was traced to several components within our infrastructure which were locked to an older stable version of the go-ethereum client which encountered a critical consensus bug at block 11234873. This affected several Geth versions including 1.9.9 and 1.9.13.”
Infura has since added capacity for the affected components as they triage the situation.
Following the service disruption, Binance and Bithumb temporarily halted ETH withdrawals.
At the same time, Changpeng Zhao of Binance has said it is investigating a possible fork at block height 11234873.
Ethereum Hard Forked
The lead developer at Blockchair, Nikita Zhavoronkov, said the split is due to code insertion by Ethereum developers.
The addition subsequently affected node operators who hadn’t upgraded. Infura happened to be one of those entities.
https://twitter.com/nikzh/status/1326465592841351168
Today’s glitch, Nikita said, “should be considered as the most serious issue Ethereum has faced since the DAO debacle 4 years ago.”
“At some point, Ethereum developers introduced a change in the code that led today to a chain split starting from block 11234873 (07:08 UTC). Those who haven’t upgraded [Blockchair, Infura, miners, and many others] got stuck on a minority chain.”
As reported, Eth2 clients developed an on-boarding problem during the Spadina test network, an issue that was “ironed-out” by Zinken.
