Geth --dev does not work with CancunTime

[fselmo]

System information

Geth version: 1.13.11 - 1.13.15
CL client & version: --dev (simulated beacon)
OS & Version: OSX Ventura 13.5.2
Commit hash : (if develop)

Expected behaviour

I wouldn't expect Invalid parameters when performing sealing work and for transaction indexing to not get stuck.

Actual behaviour

Logs that differ from Shanghai rules:

WARN [04-17|09:09:38.382] Error performing sealing work            err="Invalid parameters"

Indefinite indexing:

WARN [04-17|09:09:38.397] Served eth_getTransactionReceipt         reqid=9 duration="129.591µs" err="transaction indexing is in progress" errdata="transaction indexing is in progress"
WARN [04-17|09:09:38.508] Served eth_getTransactionReceipt         reqid=10 duration="147.688µs" err="transaction indexing is in progress" errdata="transaction indexing is in progress"
WARN [04-17|09:09:38.612] Served eth_getTransactionReceipt         reqid=11 duration="182.516µs" err="transaction indexing is in progress" errdata="transaction indexing is in progress"
WARN [04-17|09:09:38.716] Served eth_getTransactionReceipt         reqid=12 duration="151.324µs" err="transaction indexing is in progress" errdata="transaction indexing is in progress"
WARN [04-17|09:09:38.822] Served eth_getTransactionReceipt         reqid=13 duration="145.97µs"  err="transaction indexing is in progress" errdata="transaction indexing is in progress"
WARN [04-17|09:09:38.926] Served eth_getTransactionReceipt         reqid=14 duration="109.269µs" err="transaction indexing is in progress" errdata="transaction indexing is in progress"
...

Steps to reproduce the behaviour

At the present moment, I need to set terminalTotalDifficulty in genesis to -1 and difficulty to 0 in order to trick it to even trigger Shanghai rules. Nothing else seems to work. Since there's a check for difficulty > terminalTotalDifficulty and difficulty seems to need to be 0 at the genesis for the rules to trigger, I had to try to set terminalTotalDifficulty to -1 and that somehow seems to work.

All other fork rules are set as expected, as you can see from the logs below they are picked up. I just don't believe they are being triggered.

I am using dev.period == 1. Sending any transaction and attempting to get the receipt does not work as it gets stuck indexing forever. But the invalid parameters when performing sealing work also aren't present under the working Shanghai rules. Turning on Cancun time seems to trigger all of this.

Backtrace

WARN [04-17|09:09:36.006] Unknown config environment variable      envvar=GETH_BINARY
WARN [04-17|09:09:36.006] Unknown config environment variable      envvar=GETH_BINARY_LOCAL
INFO [04-17|09:09:36.017] Starting Geth in ephemeral dev mode...
WARN [04-17|09:09:36.017] You are running Geth in --dev mode. Please note the following:

  1. This mode is only intended for fast, iterative development without assumptions on
     security or persistence.
  2. The database is created in memory unless specified otherwise. Therefore, shutting down
     your computer or losing power will wipe your entire block data and chain state for
     your dev environment.
  3. A random, pre-allocated developer account will be available and unlocked as
     eth.coinbase, which can be used for testing. The random dev account is temporary,
     stored on a ramdisk, and will be lost if your machine is restarted.
  4. Mining is enabled by default. However, the client will only seal blocks if transactions
     are pending in the mempool. The miner's minimum accepted gas price is 1.
  5. Networking is disabled; there is no listen-address, the maximum number of peers is set
     to 0, and discovery is disabled.

INFO [04-17|09:09:36.020] Maximum peer count                       ETH=50 total=50
DEBUG[04-17|09:09:36.025] FS scan times                            list="122.635µs" set="39.359µs" diff="5.676µs"
DEBUG[04-17|09:09:36.025] Failed to decode keystore key            path=/tmp/tmp1xsd44so/keystore/pw.txt err="invalid character 'w' looking for beginning of value"
TRACE[04-17|09:09:36.025] Handled keystore changes                 time="172.866µs"
TRACE[04-17|09:09:36.025] Started watching keystore folder         path=/tmp/tmp1xsd44so/keystore folder=/tmp/tmp1xsd44so/keystore
DEBUG[04-17|09:09:36.027] Sanitizing Go's GC trigger               percent=100
INFO [04-17|09:09:36.028] Set global gas cap                       cap=50,000,000
INFO [04-17|09:09:37.164] Using developer account                  address=0xdC544d1AA88Ff8bbd2F2AeC754B1F1e99e1812fd
INFO [04-17|09:09:37.165] Using pebble as the backing database
INFO [04-17|09:09:37.165] Allocated cache and file handles         database=/tmp/tmp1xsd44so/geth/chaindata cache=512.00MiB handles=5120
DEBUG[04-17|09:09:37.169] Chain freezer table opened               database=/tmp/tmp1xsd44so/geth/chaindata/ancient/chain table=bodies items=0 size=0.00B
DEBUG[04-17|09:09:37.170] Chain freezer table opened               database=/tmp/tmp1xsd44so/geth/chaindata/ancient/chain table=receipts items=0 size=0.00B
DEBUG[04-17|09:09:37.171] Chain freezer table opened               database=/tmp/tmp1xsd44so/geth/chaindata/ancient/chain table=diffs items=0 size=0.00B
DEBUG[04-17|09:09:37.172] Chain freezer table opened               database=/tmp/tmp1xsd44so/geth/chaindata/ancient/chain table=headers items=0 size=0.00B
DEBUG[04-17|09:09:37.174] Chain freezer table opened               database=/tmp/tmp1xsd44so/geth/chaindata/ancient/chain table=hashes items=0 size=0.00B
INFO [04-17|09:09:37.174] Opened ancient database                  database=/tmp/tmp1xsd44so/geth/chaindata/ancient/chain readonly=true
INFO [04-17|09:09:37.174] Initializing the KZG library             backend=gokzg
INFO [04-17|09:09:37.202] Allocated trie memory caches             clean=154.00MiB dirty=256.00MiB
INFO [04-17|09:09:37.202] Using pebble as the backing database
INFO [04-17|09:09:37.202] Allocated cache and file handles         database=/tmp/tmp1xsd44so/geth/chaindata cache=512.00MiB handles=5120
DEBUG[04-17|09:09:37.404] Chain freezer table opened               database=/tmp/tmp1xsd44so/geth/chaindata/ancient/chain table=receipts items=0 size=0.00B
DEBUG[04-17|09:09:37.405] Chain freezer table opened               database=/tmp/tmp1xsd44so/geth/chaindata/ancient/chain table=diffs items=0 size=0.00B
DEBUG[04-17|09:09:37.406] Chain freezer table opened               database=/tmp/tmp1xsd44so/geth/chaindata/ancient/chain table=headers items=0 size=0.00B
DEBUG[04-17|09:09:37.406] Chain freezer table opened               database=/tmp/tmp1xsd44so/geth/chaindata/ancient/chain table=hashes items=0 size=0.00B
DEBUG[04-17|09:09:37.406] Chain freezer table opened               database=/tmp/tmp1xsd44so/geth/chaindata/ancient/chain table=bodies items=0 size=0.00B
INFO [04-17|09:09:37.406] Opened ancient database                  database=/tmp/tmp1xsd44so/geth/chaindata/ancient/chain readonly=false
INFO [04-17|09:09:37.408] State scheme set to already existing     scheme=hash
DEBUG[04-17|09:09:37.408] Current full block not old enough to freeze number=0 hash=db8524..5b0379 delay=90000
INFO [04-17|09:09:37.411] Initialising Ethereum protocol           network=1337 dbversion=<nil>
INFO [04-17|09:09:37.433] 
INFO [04-17|09:09:37.433] ---------------------------------------------------------------------------------------------------------------------------------------------------------
INFO [04-17|09:09:37.433] Chain ID:  131277322940537 (unknown)
INFO [04-17|09:09:37.433] Consensus: Beacon (proof-of-stake), merged from Ethash (proof-of-work)
INFO [04-17|09:09:37.433] 
INFO [04-17|09:09:37.433] Pre-Merge hard forks (block based):
INFO [04-17|09:09:37.433]  - Homestead:                   #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/homestead.md)
INFO [04-17|09:09:37.433]  - Tangerine Whistle (EIP 150): #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/tangerine-whistle.md)
INFO [04-17|09:09:37.433]  - Spurious Dragon/1 (EIP 155): #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
INFO [04-17|09:09:37.433]  - Spurious Dragon/2 (EIP 158): #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
INFO [04-17|09:09:37.433]  - Byzantium:                   #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/byzantium.md)
INFO [04-17|09:09:37.433]  - Constantinople:              #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/constantinople.md)
INFO [04-17|09:09:37.433]  - Petersburg:                  #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/petersburg.md)
INFO [04-17|09:09:37.433]  - Istanbul:                    #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/istanbul.md)
INFO [04-17|09:09:37.433]  - Berlin:                      #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/berlin.md)
INFO [04-17|09:09:37.434]  - London:                      #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/london.md)
INFO [04-17|09:09:37.434]  - Arrow Glacier:               #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/arrow-glacier.md)
INFO [04-17|09:09:37.434]  - Gray Glacier:                #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/gray-glacier.md)
INFO [04-17|09:09:37.434] 
INFO [04-17|09:09:37.434] Merge configured:
INFO [04-17|09:09:37.434]  - Hard-fork specification:    https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/paris.md
INFO [04-17|09:09:37.434]  - Network known to be merged: true
INFO [04-17|09:09:37.434]  - Total terminal difficulty:  -1
INFO [04-17|09:09:37.434] 
INFO [04-17|09:09:37.435] Post-Merge hard forks (timestamp based):
INFO [04-17|09:09:37.435]  - Shanghai:                    @0          (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/shanghai.md)
INFO [04-17|09:09:37.435]  - Cancun:                      @0          (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/cancun.md)
INFO [04-17|09:09:37.435] 
INFO [04-17|09:09:37.435] ---------------------------------------------------------------------------------------------------------------------------------------------------------
INFO [04-17|09:09:37.435] 
INFO [04-17|09:09:37.435] Loaded most recent local block           number=0 hash=db8524..5b0379 td=0 age=55y1mo15h
WARN [04-17|09:09:37.435] Failed to load snapshot                  err="missing or corrupted snapshot"
INFO [04-17|09:09:37.479] Rebuilding state snapshot
DEBUG[04-17|09:09:37.479] Journalled generator progress            progress=empty
DEBUG[04-17|09:09:37.501] Start snapshot generation                root=149cc3..2373b3
INFO [04-17|09:09:37.501] Initialized transaction indexer          range="last 2350000 blocks"
INFO [04-17|09:09:37.501] Resuming state snapshot generation       root=149cc3..2373b3 accounts=0 slots=0 storage=0.00B dangling=0 elapsed=21.900ms
TRACE[04-17|09:09:37.501] Detected outdated state range            kind=account prefix=0x61 last=0x err="wrong root: have 0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421 want 0x149cc3a1f5296e04547fd60cd867c8c0a300862684af92520a656910f12373b3"
DEBUG[04-17|09:09:37.502] Regenerated state range                  kind=account prefix=0x61 root=149cc3..2373b3 last=0x count=9 created=9 updated=0 untouched=0 deleted=0
DEBUG[04-17|09:09:37.502] Journalled generator progress            progress=done
DEBUG[04-17|09:09:37.502] Regenerated local transaction journal    transactions=0 accounts=0
INFO [04-17|09:09:37.523] Generated state snapshot                 accounts=9 slots=0 storage=378.00B dangling=0 elapsed=43.849ms
DEBUG[04-17|09:09:38.201] Blobpool tip threshold updated           tip=1
INFO [04-17|09:09:38.202] Chain post-merge, sync via beacon client
INFO [04-17|09:09:38.202] Gasprice oracle is ignoring threshold set threshold=2
TRACE[04-17|09:09:38.202] Decrease miner recommit interval         from=2s to=2s
TRACE[04-17|09:09:38.224] Engine API request received              method=ForkchoiceUpdated head=db8524..5b0379 finalized=db8524..5b0379 safe=db8524..5b0379
INFO [04-17|09:09:38.246] Entered PoS stage
INFO [04-17|09:09:38.268] Starting peer-to-peer node               instance=Geth/v1.13.15-stable/darwin-amd64/go1.22.1
WARN [04-17|09:09:38.268] P2P server will be useless, neither dialing nor listening
INFO [04-17|09:09:38.360] New local node record                    seq=1,713,366,578,359 id=238274fb68b719e6 ip=127.0.0.1 udp=0 tcp=0
DEBUG[04-17|09:09:38.360] IPCs registered                          namespaces=admin,debug,web3,eth,txpool,miner,net,dev
INFO [04-17|09:09:38.360] Started P2P networking                   self=enode://39661e4edcb7527bb67a0c33459869f4c77574575c329345391252d470a53503dd3950f6c2deac4ee394a891be39199d713ef6fe26f16bb248e67466e3b08c7c@127.0.0.1:0
INFO [04-17|09:09:38.360] IPC endpoint opened                      url=/tmp/tmp1xsd44so/geth.ipc
TRACE[04-17|09:09:38.360] Engine API request received              method=ForkchoiceUpdated head=db8524..5b0379 finalized=db8524..5b0379 safe=db8524..5b0379
TRACE[04-17|09:09:38.368] Accepted RPC connection                  conn=
DEBUG[04-17|09:09:38.368] RPC connection read error                err=EOF
DEBUG[04-17|09:09:38.382] Reinjecting stale transactions           count=0
INFO [04-17|09:09:38.382] Starting work on payload                 id=0x02a949a8cf438792
TRACE[04-17|09:09:38.382] Engine API request received              method=GetPayload        id=0x02a949a8cf438792
INFO [04-17|09:09:38.382] Updated payload                          id=0x02a949a8cf438792 number=1 hash=7820a0..299db8 txs=0 withdrawals=0 gas=0 fees=0 root=149cc3..2373b3 elapsed="144.297µs"
INFO [04-17|09:09:38.382] Stopping work on payload                 id=0x02a949a8cf438792 reason=delivery
WARN [04-17|09:09:38.382] Error performing sealing work            err="Invalid parameters"
TRACE[04-17|09:09:38.386] Accepted RPC connection                  conn=
DEBUG[04-17|09:09:38.386] Served eth_getBlockByNumber              reqid=0 duration="254.279µs"
DEBUG[04-17|09:09:38.388] Served eth_chainId                       reqid=1 duration="64.904µs"
DEBUG[04-17|09:09:38.390] Served eth_chainId                       reqid=2 duration="61.759µs"
DEBUG[04-17|09:09:38.392] Served eth_estimateGas                   reqid=3 duration="545.784µs"
DEBUG[04-17|09:09:38.392] Served eth_chainId                       reqid=4 duration="30.44µs"
DEBUG[04-17|09:09:38.393] Served eth_blockNumber                   reqid=5 duration="31.58µs"
DEBUG[04-17|09:09:38.393] Served eth_getBlockByNumber              reqid=6 duration="144.219µs"
INFO [04-17|09:09:38.395] Setting new local account                address=0xdC544d1AA88Ff8bbd2F2AeC754B1F1e99e1812fd
TRACE[04-17|09:09:38.395] Pooled new future transaction            hash=ca171b..03e088 from=0xdC544d1AA88Ff8bbd2F2AeC754B1F1e99e1812fd to=<nil>
TRACE[04-17|09:09:38.395] Removed old queued transactions          count=0
TRACE[04-17|09:09:38.395] Removed unpayable queued transactions    count=0
TRACE[04-17|09:09:38.395] Promoted queued transactions             count=1
INFO [04-17|09:09:38.395] Submitted contract creation              hash=0xca171ba87b2eaf383b4afbede13e76d0ae43506c98d908b1fa7a181edf03e088 from=0xdC544d1AA88Ff8bbd2F2AeC754B1F1e99e1812fd nonce=0 contract=0xd794C821fCCFF5D96F5Db44af7e29977630A9dc2 value=0
DEBUG[04-17|09:09:38.395] Distributed transactions                 plaintxs=1 blobtxs=0 largetxs=0 bcastpeers=0 bcastcount=0 annpeers=0 anncount=0
DEBUG[04-17|09:09:38.395] Served eth_sendTransaction               reqid=7 duration=1.236518ms
DEBUG[04-17|09:09:38.396] Served eth_chainId                       reqid=8 duration="26.193µs"
MATH_CONTRACT_DEPLOY_HASH: b'\xca\x17\x1b\xa8{.\xaf8;J\xfb\xed\xe1>v\xd0\xaeCPl\x98\xd9\x08\xb1\xfaz\x18\x1e\xdf\x03\xe0\x88'
WARN [04-17|09:09:38.397] Served eth_getTransactionReceipt         reqid=9 duration="129.591µs" err="transaction indexing is in progress" errdata="transaction indexing is in progress"
WARN [04-17|09:09:38.508] Served eth_getTransactionReceipt         reqid=10 duration="147.688µs" err="transaction indexing is in progress" errdata="transaction indexing is in progress"
WARN [04-17|09:09:38.612] Served eth_getTransactionReceipt         reqid=11 duration="182.516µs" err="transaction indexing is in progress" errdata="transaction indexing is in progress"
WARN [04-17|09:09:38.716] Served eth_getTransactionReceipt         reqid=12 duration="151.324µs" err="transaction indexing is in progress" errdata="transaction indexing is in progress"
WARN [04-17|09:09:38.822] Served eth_getTransactionReceipt         reqid=13 duration="145.97µs"  err="transaction indexing is in progress" errdata="transaction indexing is in progress"
WARN [04-17|09:09:38.926] Served eth_getTransactionReceipt         reqid=14 duration="109.269µs" err="transaction indexing is in progress" errdata="transaction indexing is in progress"
WARN [04-17|09:09:39.036] Served eth_getTransactionReceipt         reqid=15 duration="139.673µs" err="transaction indexing is in progress" errdata="transaction indexing is in progress"
WARN [04-17|09:09:39.142] Served eth_getTransactionReceipt         reqid=16 duration="131.598µs" err="transaction indexing is in progress" errdata="transaction indexing is in progress"
WARN [04-17|09:09:39.248] Served eth_getTransactionReceipt         reqid=17 duration="139.038µs" err="transaction indexing is in progress" errdata="transaction indexing is in progress"
WARN [04-17|09:09:39.351] Served eth_getTransactionReceipt         reqid=18 duration="132.846µs" err="transaction indexing is in progress" errdata="transaction indexing is in progress"
WARN [04-17|09:09:39.459] Served eth_getTransactionReceipt         reqid=19 duration="139.372µs" err="transaction indexing is in progress" errdata="transaction indexing is in progress"
WARN [04-17|09:09:39.563] Served eth_getTransactionReceipt         reqid=20 duration="147.151µs" err="transaction indexing is in progress" errdata="transaction indexing is in progress"
...
...
...

... polling eth_getTransactionReceipt never leaves indexing status

[fselmo]

Looks like this is related to #29475 but the transaction indexing never actually resolves for me, as one user claims it does in that issue.

Also, due to issue #29404, I can't call eth_estimateGas for any of my transactions unless I set terminalTotalDifficulty = -1 and difficulty = 0 at genesis to trick the config into using the correct fork rules even though they are set in the config.

[fselmo]

Duplicate of #29475.

[jwasinger]

Difficulty is not a signed value, so it's definitely bad that we allow it to be negative in the config. I opened #29566 which attempts to fix this issue (and also adds validation to prevent TTD from being nonzero).. Looks like the issue with fork rules not being set properly in genesis block has been resolved in #29469 .

[fselmo]

@jwasinger yeah I figured that was not intended behavior. Just poked around enough to get it to "work" until it was resolved. Thanks for the attention here. So that fix was not a part of 1.13.15 then?

[rjl493456442]

You probably have to try the latest master branch. #29469 is not backported to 1.13.15

[fselmo]

Ok thanks. That makes a lot of sense then. I'm not at my computer but it did work on master for me. I'll check again to see if all my issues are resolved then and I'll close this out if so.

[karalabe]

I'll close this if the original issue seems solved. If there's some followup, feel free to open a new issue (or reopen this).