contrib/signet/miner: grind will fail for high difficulty chain #30102

edilmedeiros · 2024-05-14T20:17:28Z

Is there an existing issue for this?

I have searched the existing issues

Current behaviour

Mining will fail with a Could not satisfy difficulty target.

❯ ~/2-development/bitcoin/bitcoin-core/contrib/signet/miner --cli "/Users/jose.edil/2-development/bitcoin/bitcoin-core/src/bitcoin-cli -datadir=/Users/jose.edil/2-development/bitcoin/signet-mining-experiments/signet-fix-hashing" generate --address tb1qylfujt900rjxzfxjj7sktpu7dpm2n9j60ch7jt --grind-cmd "/Users/jose.edil/2-development/bitcoin/bitcoin-core/src/bitcoin-util grind" --nbits 1d008d28 --ongoing
2024-05-14 16:29:05 INFO Mined block at height 10079; next in -324h56m20s (mine)
2024-05-14 16:31:22 INFO Mined block at height 10080; next in -324h56m7s (mine)
2024-05-14 16:32:34 INFO Mined block at height 10081; next in -324h54m49s (mine)
Could not satisfy difficulty target
Traceback (most recent call last):
  File "/Users/jose.edil/2-development/bitcoin/bitcoin-core/contrib/signet/miner", line 545, in <module>
    main()
  File "/Users/jose.edil/2-development/bitcoin/bitcoin-core/contrib/signet/miner", line 539, in main
    return args.fn(args)
           ^^^^^^^^^^^^^
  File "/Users/jose.edil/2-development/bitcoin/bitcoin-core/contrib/signet/miner", line 418, in do_generate
    block = finish_block(block, signet_solution, args.grind_cmd)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jose.edil/2-development/bitcoin/bitcoin-core/contrib/signet/miner", line 107, in finish_block
    newheadhex = subprocess.run(cmd, stdout=subprocess.PIPE, input=b"", check=True).stdout.strip()
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/Users/jose.edil/2-development/bitcoin/bitcoin-core/src/bitcoin-util', 'grind', '0000002063d19ae45a71b26fc04ddeef152919d2ecc22ed936ea7129d3bf291f03000000703be9fb35f03fc17b9b3210ac8364c1b76888d793397b7686128107d21c0b4a39e33166ac77031d00000000']' returned non-zero exit status 1.

This is a follow-up of the discussion in #30091: I started a signet with difficulty --nbits 1d008d28 and mined the first block with a date 30 days in the past. That would let the miner run as fast as it can until difficulty is so high it will converge to the 10 minute average time between blocks.

After each 2016 blocks, the difficulty is adjusted as per the network consensus and it gets increasingly harder to find new blocks. After 5 adjustments, the grinder eventually exhausts the nonce search space and fails, making the python script to fail as well.

I inspected the bitcoin-util grinder source, but could not find any problem there. Indeed, it does what one would expect: look for PoW and fail if the nonce search space is exhausted. That's where the stdout message comes.

Expected behaviour

I tracked down the problem to this section of the code.

The finish_block function should be able to catch the exception that can arise from the grinder subprocess and in case of failure try another block header. In case of signet, I believe it is a matter of resigning it as in and looking for PoW again until it finds something.

I started a fix to submit a PR, but wanted to see if someone else could reproduce this behavior in the meantime. It's not clear to me how to make a (unit) test to systematically expose the problem, though.

Steps to reproduce

Follow the outline discussed in #30091 and let the miner run (for a few hours) until the difficulty gets high enough so that one needs a bigger search space to find valid PoW for new blocks.

Relevant log output

Current chain info

❯ ~/2-development/bitcoin/bitcoin-core/src/bitcoin-cli -datadir="/Users/jose.edil/2-development/bitcoin/signet-mining-experiments/signet-fix-hashing" -signet getblockchaininfo
{
  "chain": "signet",
  "blocks": 10081,
  "headers": 10081,
  "bestblockhash": "000000031f29bfd32971ea36d92ec2ecd2192915efde4dc06fb2715ae49ad163",
  "difficulty": 0.2883904525532027,
  "time": 1714545315,
  "mediantime": 1714544565,
  "verificationprogress": 1,
  "initialblockdownload": true,
  "chainwork": "000000000000000000000000000000000000000000000000000000c3e464c5fc",
  "size_on_disk": 4153462,
  "pruned": false,
  "warnings": ""
}

How did you obtain Bitcoin Core

Compiled from source

What version of Bitcoin Core are you using?

v27.0

Operating system and version

MacOS Sonoma 14.4

Machine specifications

Macbook Pro M3 Pro
18GB Memory

The text was updated successfully, but these errors were encountered:

1ma · 2024-05-15T22:33:42Z

Can reproduce. Been running a signet network whose miner stalled a few dozens of blocks after the block 10080 difficulty adjustment.

The miner (v26.1 script) is ran as a systemd service like this, and the nbits parameter was determined by running the calibrating utility set to 600s. Despite this, before block 10080 the miner used to mine a block every 2.5 min instead of 10. I don't quite understand how it works, honestly.

miner --cli="bitcoin-cli" generate --address $rewards_addy --grind-cmd="bitcoin-util grind" --nbits=1d02257e --ongoing

edilmedeiros · 2024-05-16T02:59:15Z

Thanks for reporting.

About the 2m30s interval, see #30091.

1ma · 2024-05-16T18:31:33Z

Thanks for the pointer. A bit off-topic, but since there are not so many people yet who are familiar with operating a signet let me ask you a follow up question:

I'm don't really understand why the signet miner has so much logic and is so opinionated. My initial idea was to keep the difficulty as low as possible then fire a cronjob every 10 minutes to insta-mine a block at the current timestamp, so the consensus algorithm would be tricked into thinking that is the real difficulty.

I was never able to use the miner like this though, it seems to have a life of its own. No matter which nbits value I selected it mined a block every 2m30s (until it reached block 10080 and broke soon after at 10106). And if it stopped for some time it insta-mined blocks with a backdated timestamp until it catched up to the present.

Is it possible to have a signet mining setup like that, that is not constantly raising the difficulty and wasting CPU?

edilmedeiros · 2024-05-16T19:27:43Z

Yeah, I had some difficulty understanding its behavior and I believe I grasped after studying the miner source code.

It has an internal timer whose logic is capable of doing what you want. Instead of using the calibrate feature, go straight to using --min-nbits --ongoing. It will mine a block very fast and automatically wait for 10 minutes until it mines the next. That way, it keeps difficulty at minimum but can keep the network pace.

See https://edil.com.br/blog/creating-a-custom-bitcoin-signet.

The miner script will call `bitcoin-util grind` to compute PoW which will try to exhaust the block's nonce field and fail if it can't find a valid hash. This behavior does not appear for low difficulty chains, but make the miner unusable for higher difficulty settings. We capture `bitcoin-util grind` failure, build a new block header by changing the block time and try to grind again. Fixes bitcoin#30102

The miner script will call `bitcoin-util grind` to compute PoW which will try to exhaust the block's nonce field and fail if it can't find a valid hash. This behavior does not appear for low difficulty chains, but make the miner unusable for higher difficulty settings. We capture `bitcoin-util grind` exception, build a new block header with different time and try to grind again. Fixes bitcoin#30102

edilmedeiros · 2024-05-20T18:32:18Z

@1ma, since you seem to have a high difficulty signet chain running, may I ask for a review in #30130?

The miner script will call `bitcoin-util grind` to compute PoW which will try to exhaust the block's nonce field and fail if it can't find a valid hash. This behavior does not appear for low difficulty chains, but make the miner unusable for higher difficulty settings. We capture `bitcoin-util grind` exception, build a new block header with different time and try to grind again. Fixes bitcoin#30102

willcl-ark added Scripts and tools Questions and Help labels May 15, 2024

bitcoin deleted a comment from mozz30-tech May 15, 2024

edilmedeiros linked a pull request May 17, 2024 that will close this issue

contrib/signet/miner: increase miner search space #30130

Open

edilmedeiros mentioned this issue May 23, 2024

contrib/signet/miner updates #28417

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

contrib/signet/miner: grind will fail for high difficulty chain #30102

contrib/signet/miner: grind will fail for high difficulty chain #30102

edilmedeiros commented May 14, 2024 •

edited

1ma commented May 15, 2024 •

edited

edilmedeiros commented May 16, 2024

1ma commented May 16, 2024 •

edited

edilmedeiros commented May 16, 2024 •

edited

edilmedeiros commented May 20, 2024

contrib/signet/miner: grind will fail for high difficulty chain #30102

contrib/signet/miner: grind will fail for high difficulty chain #30102

Comments

edilmedeiros commented May 14, 2024 • edited

Is there an existing issue for this?

Current behaviour

Expected behaviour

Steps to reproduce

Relevant log output

How did you obtain Bitcoin Core

What version of Bitcoin Core are you using?

Operating system and version

Machine specifications

1ma commented May 15, 2024 • edited

edilmedeiros commented May 16, 2024

1ma commented May 16, 2024 • edited

edilmedeiros commented May 16, 2024 • edited

edilmedeiros commented May 20, 2024

edilmedeiros commented May 14, 2024 •

edited

1ma commented May 15, 2024 •

edited

1ma commented May 16, 2024 •

edited

edilmedeiros commented May 16, 2024 •

edited