Base node sometimes sync slow

[RyanWang0811]

hi Team

I run my Base node on AWS and sometimes Base node will sync slower than behind mainnet. I want to know how to fix this issue.

from the node-geth logs below, the node was behind the mainnet for 3 minutes. And the gap will be increased.

INFO [04-29|02:20:01.745] Imported new potential chain segment     number=13,783,820 hash=542f08..8d8676 blocks=1 txs=48  mgas=29.638 elapsed=429.898ms    mgasps=68.943  age=3m34s   snapdiffs=4.22MiB triedirty=0.00B
INFO [04-29|02:20:01.747] Chain head was updated                   number=13,783,820 hash=542f08..8d8676 root=20062e..c715c1 elapsed=1.60497ms    age=3m34s
INFO [04-29|02:20:01.913] Imported new potential chain segment     number=13,783,821 hash=1db0ce..8a55ae blocks=1 txs=54  mgas=9.950  elapsed=158.301ms    mgasps=62.856  age=3m32s   snapdiffs=4.24MiB triedirty=0.00B
INFO [04-29|02:20:01.915] Chain head was updated                   number=13,783,821 hash=1db0ce..8a55ae root=eefc15..f2ee0e elapsed=1.29939ms    age=3m32s
INFO [04-29|02:20:01.956] Imported new potential chain segment     number=13,783,822 hash=35c578..191ae4 blocks=1 txs=12  mgas=2.136  elapsed=36.169ms     mgasps=59.065  age=3m30s   snapdiffs=4.25MiB triedirty=0.00B
INFO [04-29|02:20:01.957] Chain head was updated                   number=13,783,822 hash=35c578..191ae4 root=5b39bd..482d19 elapsed="412.056µs"  age=3m30s
INFO [04-29|02:20:11.168] Imported new potential chain segment     number=13,783,823 hash=b2ffee..caede0 blocks=1 txs=130 mgas=18.151 elapsed=9.202s       mgasps=1.972   age=3m38s   snapdiffs=4.30MiB triedirty=0.00B
WARN [04-29|02:20:11.170] Ignoring already known beacon payload    number=13,783,823 hash=b2ffee..caede0 age=3m38s
INFO [04-29|02:20:11.174] Chain head was updated                   number=13,783,823 hash=b2ffee..caede0 root=bb753c..0402cf elapsed=3.128885ms   age=3m38s
INFO [04-29|02:20:11.295] Imported new potential chain segment     number=13,783,824 hash=2d6893..f4d194 blocks=1 txs=41  mgas=10.329 elapsed=115.046ms    mgasps=89.777  age=3m36s   snapdiffs=4.32MiB triedirty=0.00B
INFO [04-29|02:20:11.297] Chain head was updated                   number=13,783,824 hash=2d6893..f4d194 root=638379..5b3f97 elapsed="949.019µs"  age=3m36s
INFO [04-29|02:20:11.375] Imported new potential chain segment     number=13,783,825 hash=4fce74..067e86 blocks=1 txs=25  mgas=2.882  elapsed=71.211ms     mgasps=40.475  age=3m34s   snapdiffs=4.32MiB triedirty=0.00B
INFO [04-29|02:20:11.376] Chain head was updated                   number=13,783,825 hash=4fce74..067e86 root=362e21..9d5201 elapsed="546.88µs"   age=3m34s
INFO [04-29|02:20:11.632] Imported new potential chain segment     number=13,783,826 hash=00f572..731bd1 blocks=1 txs=80  mgas=15.041 elapsed=249.096ms    mgasps=60.383  age=3m32s   snapdiffs=4.35MiB triedirty=0.00B
INFO [04-29|02:20:11.634] Chain head was updated                   number=13,783,826 hash=00f572..731bd1 root=821bf3..2096b2 elapsed=1.789292ms   age=3m32s

the node-node logs below when syncing slower than behind the mainnet.

t=2024-04-29T02:18:41+0000 lvl=eror msg="Payload execution failed" block_hash=0xe7f485a78be22393d9b2c16fb56f2b77fe0e450923c57f79e139d804676ea5bd err="context deadline exceeded"
t=2024-04-29T02:18:41+0000 lvl=warn msg="Derivation process temporary error" attempts=1 err="engine stage failed: temp: failed to update insert payload: failed to execute payload: context deadline exceeded"
t=2024-04-29T02:18:51+0000 lvl=eror msg="Payload execution failed" block_hash=0x633de0d78a0c601b1872b34a319fba4613fe152d445adcfa2bd7086742331988 err="context deadline exceeded"
t=2024-04-29T02:18:51+0000 lvl=warn msg="Derivation process temporary error" attempts=1 err="engine stage failed: temp: failed to update insert payload: failed to execute payload: context deadline exceeded"
t=2024-04-29T02:19:00+0000 lvl=eror msg="Payload execution failed" block_hash=0x70a51d44382b3f401c0e406d8989d5cb136e86d2f8214cd60f73c25f0c763d18 err="context deadline exceeded"
t=2024-04-29T02:19:00+0000 lvl=warn msg="Derivation process temporary error" attempts=1 err="engine stage failed: temp: failed to update insert payload: failed to execute payload: context deadline exceeded"
t=2024-04-29T02:19:10+0000 lvl=eror msg="Payload execution failed" block_hash=0x75b989ad0c4ac2e40d0201141326e71fb69b32a9c8e87ef339e379ab6a3bd374 err="context deadline exceeded"
t=2024-04-29T02:19:10+0000 lvl=warn msg="Derivation process temporary error" attempts=1 err="engine stage failed: temp: failed to update insert payload: failed to execute payload: context deadline exceeded"
t=2024-04-29T02:19:20+0000 lvl=eror msg="Payload execution failed" block_hash=0xe2de1b31a94573d95d05c1ecbf5c607749752ebe454d3718ba6213e1331667e8 err="context deadline exceeded"
t=2024-04-29T02:19:20+0000 lvl=warn msg="Derivation process temporary error" attempts=1 err="engine stage failed: temp: failed to update insert payload: failed to execute payload: context deadline exceeded"
t=2024-04-29T02:19:29+0000 lvl=eror msg="Payload execution failed" block_hash=0x782095b42fa6013f7bc16e9fd1a6d36867b6576ea253a60a9568314549656ec0 err="context deadline exceeded"
t=2024-04-29T02:19:29+0000 lvl=warn msg="Derivation process temporary error" attempts=1 err="engine stage failed: temp: failed to update insert payload: failed to execute payload: context deadline exceeded"
t=2024-04-29T02:19:38+0000 lvl=eror msg="Payload execution failed" block_hash=0x60685607692ee62ea7bec3c5821f19ab072b41ee279eb13fbae57223a181414f err="context deadline exceeded"
t=2024-04-29T02:19:38+0000 lvl=warn msg="Derivation process temporary error" attempts=1 err="engine stage failed: temp: failed to update insert payload: failed to execute payload: context deadline exceeded"

Usually, the node will follow up mainnet after 2~3 hous. But I want to know how to fix this issue.

[wbnns]

@RyanWang0811

Hi there! Can you please run this command and share the full output?

curl -d '{"id":0,"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["latest",false]}' \
  -H "Content-Type: application/json" http://localhost:8545

It's possible it could be related to this: #245 (comment)

(fix coming soon)

[wbnns]

@RyanWang0811 Heya! We just pushed up a commit that may help alleviate this issue:
c0b0349

Please pull down the change and if you're still encountering the issue after, please let us know! We'll reopen this if so! 👍 If you can also share the output of that previous command I mentioned, that will be helpful to know too.

[RyanWang0811]

Sorry for the late reply.

I found the doc mentioned that Amazon Elastic Block Store (EBS) has buffered disk reads issue.

https://docs.base.org/tutorials/run-a-base-node/#hardware-requirements

So I change to use AWS i family type with NVME Disk. After change it, everything seem to good to me.