1.8.x post-mortem: undocumented hard-fork caused by Dogecoin Core 1.10.0

[patricklodder]

Summary

While testing different upgrade scenarios (from 1.6, 1.7, 1.8 and 1.10 to 1.14.6) I observed an incompatibility between all 1.8.x versions and 1.10.0 and later that I have not found any evidence of being documented or announced. The observed behavior is an accidental hard fork at block 1731044, that luckily was preceded by a successful softfork. This means that although the affected 1.8.x software could not be used to mine valid blocks on mainnet anymore for any 1.10 or later version, it was supposed to be able to validate blocks until today but unfortunately this is not the case. 1.8.x will not accept blocks from the 1.10 validated chain past block 1731043, will not be seeing the chain with the most work and could observe a different chain than 1.10 and 1.14 would because of that. This puts 1.8.x node operators at risk of operating on a forked chain.

Note to all 1.8 node operators

Dogecoin Core 1.8 is no longer compatible with Dogecoin Core 1.10 and 1.14.x, and will not be compatible with 1.21. Both 1.8 or 1.10 are EOL and archived on this repository. If you are still using this software today, your node will not accept any blocks that are part of the checkpointed main chain after 2017-05-26 12:42:44 UTC. Hopefully, operators have noticed that something is off by now, as it’s been over 6.5 years.

If you haven't noticed anything and are still running a 1.8.x node, and you want to be compatible with the software actively maintained on this repository, you’ll want to consider upgrading to 1.14.6. My spider has found 4 connectable 1.8.x nodes on the network over the past 2 months, one of which was my test node used to investigate this problem. I have no way to reliably estimate how many 1.8.x nodes are out there that are not connectable from the outside, so there are probably more nodes with this software running than the ones that I've seen.

Observation

Dogecoin Core versions 1.8.0 to 1.8.3 are supposed to be usable until today, but can only sync:

• mainnet until block b5518be1bee360ffa7458d06536e109165f7435316c6688ffcd6f65665aa8b59 (height 1731043)
• testnet until block 77a9fb02393235de6f15d3c3f1ce340e767b2f25325408a817a4aabef65324ff (height 770632)

The respective next blocks according to the 1.14.6-validated chain are:

• 10b320f69f1eb5e1b7bb45fbc108e4379b5a77dee5a3b0a406038989b9641539
• 45ce558d629c29c0a340e146e6b8a9bea168b7481da02706178d02b47fabb36b

These blocks fail on 1.8.x with the following error sequence:

ERROR: MergedMiningHeader missing from parent coinbase
ERROR: CheckProofOfWork() : AUX POW is not valid
ERROR: CheckBlockHeader() : proof of work failed
ERROR: ProcessBlock() : CheckBlock FAILED

Analysis

The cause of these rejections is a difference between the AuxPow code initially ported from united-scrypt-coin into 1.8.0, and the code that was later ported into a clean Bitcoin 0.10 base, from Namecoin, into 1.10.0.

The relevant 1.8.0 code in CAuxPow::Check:

dogecoin/src/auxpow.cpp

Lines 42 to 73 in 6522864

	// Check that the same work is not submitted twice to our chain.
	//
	CScript::const_iterator pcHead =
	std::search(script.begin(), script.end(), UBEGIN(pchMergedMiningHeader), UEND(pchMergedMiningHeader));
	CScript::const_iterator pc =
	std::search(script.begin(), script.end(), vchRootHash.begin(), vchRootHash.end());
	if (pcHead == script.end())
	return error("MergedMiningHeader missing from parent coinbase");
	if (pc == script.end())
	return error("Aux POW missing chain merkle root in parent coinbase");
	if (pcHead != script.end())
	{
	// Enforce only one chain merkle root by checking that a single instance of the merged
	// mining header exists just before.
	if (script.end() != std::search(pcHead + 1, script.end(), UBEGIN(pchMergedMiningHeader), UEND(pchMergedMiningHeader)))
	return error("Multiple merged mining headers in coinbase");
	if (pcHead + sizeof(pchMergedMiningHeader) != pc)
	return error("Merged mining header is not just before chain merkle root");
	}
	else
	{
	// For backward compatibility.
	// Enforce only one chain merkle root by checking that it starts early in the coinbase.
	// 8-12 bytes are enough to encode extraNonce and nBits.
	if (pc - script.begin() > 20)
	return error("Aux POW chain merkle root must start in the first 20 bytes of the parent coinbase");
	}

The relevant 1.8.1-1.8.3 code in CAuxPow::Check:

dogecoin/src/auxpow.cpp

Lines 42 to 62 in f626f8f

	// Check that the same work is not submitted twice to our chain.
	//
	CScript::const_iterator pcHead =
	std::search(script.begin(), script.end(), UBEGIN(pchMergedMiningHeader), UEND(pchMergedMiningHeader));
	CScript::const_iterator pc =
	std::search(script.begin(), script.end(), vchRootHash.begin(), vchRootHash.end());
	if (pcHead == script.end())
	return error("MergedMiningHeader missing from parent coinbase");
	if (pc == script.end())
	return error("Aux POW missing chain merkle root in parent coinbase");
	// Enforce only one chain merkle root by checking that a single instance of the merged
	// mining header exists just before.
	if (script.end() != std::search(pcHead + 1, script.end(), UBEGIN(pchMergedMiningHeader), UEND(pchMergedMiningHeader)))
	return error("Multiple merged mining headers in coinbase");
	if (pcHead + sizeof(pchMergedMiningHeader) != pc)
	return error("Merged mining header is not just before chain merkle root");

The relevant 1.10.0 code in CAuxPow::Check:

dogecoin/src/auxpow.cpp

Lines 130 to 155 in 22aa7b1

	// Check that the same work is not submitted twice to our chain.
	//
	CScript::const_iterator pcHead =
	std::search(script.begin(), script.end(), UBEGIN(pchMergedMiningHeader), UEND(pchMergedMiningHeader));
	CScript::const_iterator pc =
	std::search(script.begin(), script.end(), vchRootHash.begin(), vchRootHash.end());
	if (pc == script.end())
	return error("Aux POW missing chain merkle root in parent coinbase");
	if (pcHead != script.end()) {
	// Enforce only one chain merkle root by checking that a single instance of the merged
	// mining header exists just before.
	if (script.end() != std::search(pcHead + 1, script.end(), UBEGIN(pchMergedMiningHeader), UEND(pchMergedMiningHeader)))
	return error("Multiple merged mining headers in coinbase");
	if (pcHead + sizeof(pchMergedMiningHeader) != pc)
	return error("Merged mining header is not just before chain merkle root");
	} else {
	// For backward compatibility.
	// Enforce only one chain merkle root by checking that it starts early in the coinbase.
	// 8-12 bytes are enough to encode extraNonce and nBits.
	if (pc - script.begin() > 20)
	return error("Aux POW chain merkle root must start in the first 20 bytes of the parent coinbase");
	}

The if (pcHead == script.end()) check that exists only in 1.8.x makes sure that the "merge mining header", a 4-byte magic string, is present in every aux chain coinbase transaction, and always precedes the merkle root of all merge mined chain trees. For 1.8.1, #748 was raised by the Namecoin Core maintainer because the else statement after if (pcHead != script.end()) was unreachable (after all, the same condition was already executed prior, and errored out if true), and the entirety much cleaner. This cleaner version was good because Dogecoin didn’t need to support legacy Namecoin mechanics. For 1.10.0, the code was however copied straight from Namecoin Core, which doesn’t have the if (pcHead == script.end()) check at all, because that cannot be enforced on Namecoin.

Because of this, 1.10.0 and 1.14.x are less strict than 1.8.x in enforcing the presence of the 4 byte header sequence, and at the blocks mentioned above, these were the first time on the respective chains that a block made it in (without ending up on a forked chain tip.) On mainnet, I have found 6918 blocks mined without a merge mining header present (and thus incompatible with 1.8.x) as of block 5040000.

Timeline

Date	Event
2014-08-24	Release of v1.8.0, introduces "unitedscryptcoin" AuxPoW to Dogecoin
2014-09-11	AuxPoW activates at block 371337 on mainnet
2014-12-20	Release of v1.8.1, removes unreachable code in AuxPoW validation logic (#748)
2015-06-21	AuxPoW unit tests from Namecoin get ported into the 1.9 development branch (#1130)
2015-06-25	1.9 development is abandoned in favor of a 1.10 rewrite (see #927 (comment))
2015-07-05	AuxPoW gets ported from Namecoin on top of Bitcoin 0.10 (#1200), before tests are ready
2015-08-26	AuxPoW parametrization tests are added to 1.10 (#1268)
2015-11-10	Release of v1.10.0, introduces the v3 softfork for BIP66 (strict signatures)
2016-01-06	v3 softfork activates at block 1032512 on mainnet, invalidating blocks mined with 1.8.x software
2017-05-26	BTCC pool submits block 1731044 that does not validate with 1.8.x AuxPoW validation logic, causing a hardfork

Underlying causes in development process

Increased risk through constant rewrites

As discussed previously, the strategy of rewriting Dogecoin Core code on top of newer Bitcoin Core increases the risk for issues like this. As there is no succinct, well-documented, maintained, publicly available set of patches, every line of every change has to be reviewed and regression tested, and this simply does not happen, also not when the forking code was reviewed for 1.10 (#1200). The process has improved for the current iteration of this effort in 1.21-dev, where at least tests are required to succeed for every pull request, but there are no checks that the code and tests are consistent with previous versions of the software, and the regression tests have been disabled for now. (#2435)

Insufficient coverage in tests

Consensus tests for Dogecoin specific rules, such as the auxpow integration test that addresses the code this issue occurred on, is covering insufficient scenarios. For 1.8, there was no unit test, and no integration test.

Impact

Because the observed accidental hardfork was preceded by a widely carried softfork, impact is limited, but not without risk:

• 1.8.x nodes cannot sync up to the current block
• 1.8.x miners could create a 18956-block forked off chaintip from block 1731043 onward, that would be a valid fork on 1.14, until the checkpoint on block 1750000

---

Please do not hesitate to ask questions if you have any.

[slightlyskepticalpotat]

Are there possibly other previously-undocumented hard fork scenarios that have yet to be discovered?

[patricklodder]

That's possible. But with a track record of 2 intended and 2 unintended hard forks, let's hope that it stays this way and no more hard forks unless there is no other way, it's intentional, and successful.

[MishaBoar]

Hi Patrick, thanks for reporting this, I looked at the code you pointed out and from what I can assess with my limited skills this makes sense, and it is disturbing.

I would like to post a warning on Twitter to encourage users to upgrade. Here you write:

If you haven't noticed anything and are still running a 1.8.x node, and you want to be compatible with the software actively maintained on this repository, you’ll want to consider upgrading to 1.14.6

Is there some other concern I should point out for those that need to upgrade? E.g. those who might have noticed something and are now on a fork? It is unlikely my post will reach them, but you never know.

[patricklodder]

Is there some other concern I should point out for those that need to upgrade?

If people have been actively transacting on any forked chain, they should back up their entire datadir, including wallet file if they use that, because there won't be a way to get the block history back after upgrading. If further investigation of an individual case is needed, the data needs to be retained at the node that ran a fork, if any.

[MishaBoar]

Thank you!