P2P distribution gets stuck in the first attempt

[abhishek-das-gupta]

OS: RH9
libtorrent version: 1.2.0
boost version: 1.75.0

Hi all, we had a working libtorent python binding that works successfully on other Oses (excluding RHEL9) . The versions we use on other Oses i.e excluding RH9 are libtorrent 1.1.5 and boost 1.65.0.

But for RHEL9, I had to upgrade libtorrent (1.15 -> 1.2.0) and boost version from (1.65.0 -> 1.75.0) to get a successfull build/
I had to manually add the following changes in libtorrent files to get a successful build.

I am not sure why I had to do any of these things?

1.Changing get_io_service()
`obj.get_io_service() -> (boost::asio::io_context&)(obj).get_executor().context();`

which was found from boost mailing list and various online forums.

2.
Manually adding the libtorrent commit:
https://github.com/arvidn/libtorrent/commit/e4b812b50b0a

After getting a successful build, the torrent distribution b/w peers is not happening in one-go. Only when I do a systemctl restart peer the distribtuion completes and only after the restart I see in the logs that pieces are gettting downloaded.

Here are the logs for libtorrent.log before I do systemctl restart peer the logs get stuck here.

[29/May/2023 21:47:03 -0700] 12582 MainThread server       INFO     Adding torrent: file:///opt/cloudera/parcel-cache/CDH-7.1.9-1.cdh7.1.9.p0.40847337-el9.parcel.torrent
[29/May/2023 21:47:03 -0700] 12582 MainThread torrent      INFO     Save torrent: <libtorrent.add_torrent_params object at 0x7f88da475720>
[29/May/2023 21:47:03 -0700] 12582 Thread-2 server       INFO     add_torrent added torrent: CDH-7.1.9-1.cdh7.1.9.p0.40847337-el9.parcel
[29/May/2023 21:47:03 -0700] 12582 MainThread torrent      INFO     [CDH-7.1.9-1.cdh7.1.9.p0.40847337-el9.parcel] activating: d1f6a58f1f1ece1208d65110e7b91b87d1cf9873
[29/May/2023 21:47:03 -0700] 12582 MainThread torrent      INFO     connected to peers {('10.65.44.43', 7191), ('10.65.31.193', 7191), ('10.65.7.86', 7191)}
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     torrent_added CDH-7.1.9-1.cdh7.1.9.p0.40847337-el9.parcel added
[29/May/2023 21:47:04 -0700] 12582 MainThread server       INFO     Adding torrent seed: <class 'str'> 99 'http://rh-9-1.vpc.cloudera.com:7180/cmf/parcel/download/CDH-7.1.9-1.cdh7.1.9.p0.40847337-el9.parcel'
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     torrent_log CDH-7.1.9-1.cdh7.1.9.p0.40847337-el9.parcel: add_torrent add_peer() [ ] connect-candidates: -1
[29/May/2023 21:47:04 -0700] 12582 MainThread torrent      INFO     Adding seed url: http://rh-9-1.vpc.cloudera.com:7180/cmf/parcel/download/CDH-7.1.9-1.cdh7.1.9.p0.40847337-el9.parcel
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     torrent_log CDH-7.1.9-1.cdh7.1.9.p0.40847337-el9.parcel: creating torrent: CDH-7.1.9-1.cdh7.1.9.p0.40847337-el9.parcel max-uploads: 1 max-connections: 16777215 upload-limit: -1 download-limit: -1 flags: apply-ip-filter paused auto-managed update-subscribe  save-path: /opt/cloudera/parcels/.flood/CDH-7.1.9-1.cdh7.1.9.p0.40847337-el9.parcel/
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     torrent_log CDH-7.1.9-1.cdh7.1.9.p0.40847337-el9.parcel: *** UPDATE LIST [ torrent_want_scrape : 1 ]
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     torrent_log CDH-7.1.9-1.cdh7.1.9.p0.40847337-el9.parcel: init torrent: CDH-7.1.9-1.cdh7.1.9.p0.40847337-el9.parcel
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     torrent_log CDH-7.1.9-1.cdh7.1.9.p0.40847337-el9.parcel: init, async_check_files
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     dht_log DHT traversal: [195] NEW target: 0000000000000000000000000000000000000000 k: 8
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     dht_log DHT rpc_manager: [195] invoking get_peers -> 10.65.44.43:7191
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     dht_pkt ==> [10.65.44.43:7191] { 'a': { 'id': '\xd8\xd5\xc2\xd8\x81\x021\x03\xe9\xa8\x96\xd6Y~\xb2\xb5\xb8\x09\xbcN', 'info_hash': '\xd8\xd5\xd3\xe70{\xd6RI\x08\x0fR\x1427\x86\x9euvp' }, 'q': 'get_peers', 't': '\x01\x91', 'v': 'LT\x01\x02', 'y': 'q' }
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     dht_log DHT traversal: [196] NEW target: 0000000000000000000000000000000000000000 k: 8
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     dht_log DHT rpc_manager: [196] invoking get_peers -> 10.65.31.193:7191
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     dht_pkt ==> [10.65.31.193:7191] { 'a': { 'id': '\xd8\xd5\xc2\xd8\x81\x021\x03\xe9\xa8\x96\xd6Y~\xb2\xb5\xb8\x09\xbcN', 'info_hash': '\xd8\xd5\xd0\xd0\xe9Z&KC\x9b\xa0\xe4\xae\x023!h\xfd\xdf\x98' }, 'q': 'get_peers', 't': ':h', 'v': 'LT\x01\x02', 'y': 'q' }
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     dht_log DHT traversal: [197] NEW target: 0000000000000000000000000000000000000000 k: 8
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     dht_log DHT rpc_manager: [197] invoking get_peers -> 10.65.7.86:7191
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     dht_pkt ==> [10.65.7.86:7191] { 'a': { 'id': '\xd8\xd5\xc2\xd8\x81\x021\x03\xe9\xa8\x96\xd6Y~\xb2\xb5\xb8\x09\xbcN', 'info_hash': '\xd8\xd5\xdc4_\xb6\x97n\x0aL\x84\xea\x02Gh\xc39'\xf5\x1e' }, 'q': 'get_peers', 't': '^\x19', 'v': 'LT\x01\x02', 'y': 'q' }
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     dht_log DHT traversal: [198] NEW target: 0000000000000000000000000000000000000000 k: 8
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     dht_log DHT rpc_manager: [198] invoking get_peers -> 127.0.0.1:7191
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     dht_pkt ==> [127.0.0.1:7191] { 'a': { 'id': '\xd8\xd5\xc2\xd8\x81\x021\x03\xe9\xa8\x96\xd6Y~\xb2\xb5\xb8\x09\xbcN', 'info_hash': '\xd8\xd5\xcf\x16W\x18g}\xfb\x8e\x1e/Q\xf8\x17\xe6\x81\xd7\xe4W' }, 'q': 'get_peers', 't': '?\xbc', 'v': 'LT\x01\x02', 'y': 'q' }
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     dht_pkt <== [10.65.7.86:7191] { 'a': { 'id': '\xd8\xd5\xc2\xd8\x81\x021\x03\xe9\xa8\x96\xd6Y~\xb2\xb5\xb8\x09\xbcN', 'info_hash': '\xd8\xd5\xdc4_\xb6\x97n\x0aL\x84\xea\x02Gh\xc39'\xf5\x1e' }, 'q': 'get_peers', 't': '^\x19', 'v': 'LT\x01\x02', 'y': 'q' }
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     dht_pkt ==> [10.65.7.86:7191] { 'ip': '\x0aA\x07V\x1c\x17', 'r': { 'id': '\xd8\xd5\xc2\xd8\x81\x021\x03\xe9\xa8\x96\xd6Y~\xb2\xb5\xb8\x09\xbcN', 'nodes': '\xd8\xd5\xf7{\xc1\xb4\xde1?...q\xd4\x8f\xbe\x94\x9ee\xa7\xa0', 'p': 7191, 'token': 'rP\x9a\xfd' }, 't': '^\x19', 'v': 'LT\x01\x02', 'y': 'r' }
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     dht_pkt <== [127.0.0.1:7191] { 'a': { 'id': '\xd8\xd5\xc2\xd8\x81\x021\x03\xe9\xa8\x96\xd6Y~\xb2\xb5\xb8\x09\xbcN', 'info_hash': '\xd8\xd5\xcf\x16W\x18g}\xfb\x8e\x1e/Q\xf8\x17\xe6\x81\xd7\xe4W' }, 'q': 'get_peers', 't': '?\xbc', 'v': 'LT\x01\x02', 'y': 'q' }
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     dht_pkt ==> [127.0.0.1:7191] { 'ip': '\x7f\x00\x00\x01\x1c\x17', 'r': { 'id': '\xd8\xd5\xc2\xd8\x81\x021\x03\xe9\xa8\x96\xd6Y~\xb2\xb5\xb8\x09\xbcN', 'nodes': '\xd8\xd5\xf7{\xc1\xb4\xde1?...q\xd4\x8f\xbe\x94\x9ee\xa7\xa0', 'p': 7191, 'token': 'N\x10\xd3z' }, 't': '?\xbc', 'v': 'LT\x01\x02', 'y': 'r' }
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     dht_pkt <== [10.65.7.86:7191] { 'ip': '\x0aA\x07V\x1c\x17', 'r': { 'id': '\xd8\xd5\xc2\xd8\x81\x021\x03\xe9\xa8\x96\xd6Y~\xb2\xb5\xb8\x09\xbcN', 'nodes': '\xd8\xd5\xf7{\xc1\xb4\xde1?...q\xd4\x8f\xbe\x94\x9ee\xa7\xa0', 'p': 7191, 'token': 'rP\x9a\xfd' }, 't': '^\x19', 'v': 'LT\x01\x02', 'y': 'r' }
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     dht_log DHT rpc_manager: [197] round trip time(ms): 0 from 10.65.7.86:7191
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     dht_log DHT rpc_manager: [197] reply with transaction id: 2 from 10.65.7.86:7191
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     dht_pkt <== [127.0.0.1:7191] { 'ip': '\x7f\x00\x00\x01\x1c\x17', 'r': { 'id': '\xd8\xd5\xc2\xd8\x81\x021\x03\xe9\xa8\x96\xd6Y~\xb2\xb5\xb8\x09\xbcN', 'nodes': '\xd8\xd5\xf7{\xc1\xb4\xde1?...q\xd4\x8f\xbe\x94\x9ee\xa7\xa0', 'p': 7191, 'token': 'N\x10\xd3z' }, 't': '?\xbc', 'v': 'LT\x01\x02', 'y': 'r' }
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     dht_log DHT rpc_manager: [198] round trip time(ms): 0 from 127.0.0.1:7191
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     dht_log DHT rpc_manager: [198] reply with transaction id: 2 from 127.0.0.1:7191
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     dht_pkt <== [10.65.31.193:7191] { 'ip': '\x0aA\x07V\x1c\x17', 'r': { 'id': '\x8au\x0a\x80\xdal\xab\xb7\xd9\x12\x97\xb1\x06\xa1\x0b\xf3@\xca\x18"', 'nodes': '\xd8\xd5\xc2\xd8\x81\x021\x03\xe9...@Q\x0bu\xfduf8 ', 'p': 7191, 'token': '\xa5\x919\xf0' }, 't': ':h', 'v': 'LT\x01\x02', 'y': 'r' }
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     dht_log DHT rpc_manager: [196] round trip time(ms): 0 from 10.65.31.193:7191
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     dht_log DHT rpc_manager: [196] reply with transaction id: 2 from 10.65.31.193:7191
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     dht_log DHT routing_table: evicting node (changed ID): old: a4568e0fcb016bfdace5d31c1b5c9954ccd73bc5 new: 8a750a80da6cabb7d91297b106a10bf340ca1822 10.65.31.193
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     dht_pkt <== [10.65.44.43:7191] { 'ip': '\x0aA\x07V\x1c\x17', 'r': { 'id': '!\xd3U\xf2\x9c\xda\\x82\xc7\xe0\xbc\x9a5U\xe5$N\xca\x9fh', 'nodes': '\xb0n<6\xe4\xaa\xc6E\xa3...%\xbfQ\x1f\xdf\x0d\xa8:\x8b', 'p': 7191, 'token': 'II\xec\xb0' }, 't': '\x01\x91', 'v': 'LT\x01\x02', 'y': 'r' }
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     dht_log DHT rpc_manager: [195] round trip time(ms): 1 from 10.65.44.43:7191
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     dht_log DHT rpc_manager: [195] reply with transaction id: 2 from 10.65.44.43:7191
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     dht_log DHT traversal: [199] NEW target: 0000000000000000000000000000000000000000 k: 8
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     dht_log DHT rpc_manager: [199] invoking get_peers -> 10.65.44.43:7191
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     dht_pkt ==> [10.65.44.43:7191] { 'a': { 'id': '\xd8\xd5\xc2\xd8\x81\x021\x03\xe9\xa8\x96\xd6Y~\xb2\xb5\xb8\x09\xbcN', 'info_hash': '\xd8\xd5\xe2\xb9h\x87\xa5~,`\xf3\xbe~\xa3\x09\xc58\xe2\x19q' }, 'q': 'get_peers', 't': '`R', 'v': 'LT\x01\x02', 'y': 'q' }
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     dht_log DHT traversal: [200] NEW target: 0000000000000000000000000000000000000000 k: 8
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     dht_log DHT rpc_manager: [200] invoking get_peers -> 10.65.31.193:7191
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     dht_pkt ==> [10.65.31.193:7191] { 'a': { 'id': '\xd8\xd5\xc2\xd8\x81\x021\x03\xe9\xa8\x96\xd6Y~\xb2\xb5\xb8\x09\xbcN', 'info_hash': '\xd8\xd5\xc5\xe6\xe7\xb1\xcf36\xed\x12\xc3\x18\x92\xfaQ7@z\xea' }, 'q': 'get_peers', 't': '\x1e\x8b', 'v': 'LT\x01\x02', 'y': 'q' }
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     dht_log DHT traversal: [201] NEW target: 0000000000000000000000000000000000000000 k: 8
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     dht_log DHT rpc_manager: [201] invoking get_peers -> 10.65.7.86:7191
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     dht_pkt ==> [10.65.7.86:7191] { 'a': { 'id': '\xd8\xd5\xc2\xd8\x81\x021\x03\xe9\xa8\x96\xd6Y~\xb2\xb5\xb8\x09\xbcN', 'info_hash': '\xd8\xd5\xe6\xc9-\xbd\x9e\x830K\xf2\x81\xfd\xe8\x99-\xde< I' }, 'q': 'get_peers', 't': '-\xd7', 'v': 'LT\x01\x02', 'y': 'q' }
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     dht_pkt <== [10.65.7.86:7191] { 'a': { 'id': '\xd8\xd5\xc2\xd8\x81\x021\x03\xe9\xa8\x96\xd6Y~\xb2\xb5\xb8\x09\xbcN', 'info_hash': '\xd8\xd5\xe6\xc9-\xbd\x9e\x830K\xf2\x81\xfd\xe8\x99-\xde< I' }, 'q': 'get_peers', 't': '-\xd7', 'v': 'LT\x01\x02', 'y': 'q' }
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     dht_pkt ==> [10.65.7.86:7191] { 'ip': '\x0aA\x07V\x1c\x17', 'r': { 'id': '\xd8\xd5\xc2\xd8\x81\x021\x03\xe9\xa8\x96\xd6Y~\xb2\xb5\xb8\x09\xbcN', 'nodes': '\xd8\xd5\xf7{\xc1\xb4\xde1?...\x1c`u\xaeP\x054\x1a\xe1', 'p': 7191, 'token': '\xd4D.\xf4' }, 't': '-\xd7', 'v': 'LT\x01\x02', 'y': 'r' }
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     dht_pkt <== [10.65.7.86:7191] { 'ip': '\x0aA\x07V\x1c\x17', 'r': { 'id': '\xd8\xd5\xc2\xd8\x81\x021\x03\xe9\xa8\x96\xd6Y~\xb2\xb5\xb8\x09\xbcN', 'nodes': '\xd8\xd5\xf7{\xc1\xb4\xde1?...\x1c`u\xaeP\x054\x1a\xe1', 'p': 7191, 'token': '\xd4D.\xf4' }, 't': '-\xd7', 'v': 'LT\x01\x02', 'y': 'r' }
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     dht_log DHT rpc_manager: [201] round trip time(ms): 0 from 10.65.7.86:7191
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     dht_log DHT rpc_manager: [201] reply with transaction id: 2 from 10.65.7.86:7191
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     dht_pkt <== [10.65.31.193:7191] { 'ip': '\x0aA\x07V\x1c\x17', 'r': { 'id': '\x8au\x0a\x80\xdal\xab\xb7\xd9\x12\x97\xb1\x06\xa1\x0b\xf3@\xca\x18"', 'nodes': '\xd8\xd5\xc2\xd8\x81\x021\x03\xe9...@Q\x0bu\xfduf8 ', 'p': 7191, 'token': '\xf2\x00^\xaf' }, 't': '\x1e\x8b', 'v': 'LT\x01\x02', 'y': 'r' }
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     dht_log DHT rpc_manager: [200] round trip time(ms): 0 from 10.65.31.193:7191
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     dht_log DHT rpc_manager: [200] reply with transaction id: 2 from 10.65.31.193:7191
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     dht_pkt <== [10.65.44.43:7191] { 'ip': '\x0aA\x07V\x1c\x17', 'r': { 'id': '!\xd3U\xf2\x9c\xda\\x82\xc7\xe0\xbc\x9a5U\xe5$N\xca\x9fh', 'nodes': '\xb0n<6\xe4\xaa\xc6E\xa3...%\xbfQ\x1f\xdf\x0d\xa8:\x8b', 'p': 7191, 'token': 'w]|\x05' }, 't': '`R', 'v': 'LT\x01\x02', 'y': 'r' }
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     dht_log DHT rpc_manager: [199] round trip time(ms): 1 from 10.65.44.43:7191
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     dht_log DHT rpc_manager: [199] reply with transaction id: 2 from 10.65.44.43:7191
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     torrent_log CDH-7.1.9-1.cdh7.1.9.p0.40847337-el9.parcel: resume-checked add_peer() [ ] connect-candidates: 3
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     torrent_log CDH-7.1.9-1.cdh7.1.9.p0.40847337-el9.parcel: resume added peers (total peers: 3)
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     torrent_log CDH-7.1.9-1.cdh7.1.9.p0.40847337-el9.parcel: fastresume data accepted
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     state_changed CDH-7.1.9-1.cdh7.1.9.p0.40847337-el9.parcel: state changed to: downloading
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     torrent_log CDH-7.1.9-1.cdh7.1.9.p0.40847337-el9.parcel: set_state() 3
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     torrent_log CDH-7.1.9-1.cdh7.1.9.p0.40847337-el9.parcel: *** UPDATE LIST [ torrent_downloading_auto_managed : 1 ]
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     torrent_checked CDH-7.1.9-1.cdh7.1.9.p0.40847337-el9.parcel checked
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     torrent_log CDH-7.1.9-1.cdh7.1.9.p0.40847337-el9.parcel: start_announcing(), paused
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     dht_log DHT rpc_manager: [194] short-timing out transaction id: 19783 from: 212.164.65.35:11777
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     dht_log DHT rpc_manager: [193] short-timing out transaction id: 10294 from: 14.44.34.168:8137
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     dht_log DHT rpc_manager: [192] short-timing out transaction id: 23043 from: 84.64.218.114:50322
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     dht_log DHT routing_table: NODE FAILED id: d8d58d6c0d9d062e7c6d7b63665bee91825075a7 ip: 212.164.65.35:11777 fails: 0 pinged: 0 up-time: 28
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     dht_log DHT routing_table: NODE FAILED id: d8d55eb9d38c647af4fdd341f0e6ad80639dbefe ip: 14.44.34.168:8137 fails: 0 pinged: 0 up-time: 28
[29/May/2023 21:47:04 -0700] 12582 Thread-2 server       INFO     log <== INCOMING CONNECTION [ rejected, no active torrents ]

I am newbie to libtorrent and I am not versatile on how to read the logs here.

• Can anyone suggest what is happening in the logs?
• How to debug this?
• I tried to capture the core dump during when this distribution process gets stuck. I think some type of wait is happening which might be related why the above logs get stuck?

[Switching to thread 2 (Thread 0x7ff5ed306640 (LWP 12587))]
#0  0x00007ff5ef144abe in epoll_wait () from /lib64/libc.so.6
(gdb) bt full
#0  0x00007ff5ef144abe in epoll_wait () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007ff5edc89c76 in boost::asio::detail::epoll_reactor::run (this=0x5613dae86b80, usec=<optimized out>, ops=...)
    at /grid/0/jenkins/cmf/build/redhat9/boost//include/boost/asio/detail/impl/epoll_reactor.ipp:471
        timeout = <optimized out>
        events = {{events = 1, data = {ptr = 0x5613dae86be8, fd = -622302232, u32 = 3672665064, u64 = 94643277032424}}, {events = 9, data = {ptr = 0x284600000000, fd = 0, u32 = 0, 
              u64 = 44281112821760}}, {events = 475136, data = {ptr = 0x200004000, fd = 16384, u32 = 16384, u64 = 8589950976}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {
            events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 32757, data = {ptr = 0x7f0000000000, fd = 0, u32 = 0, u64 = 139637976727552}}, {events = 3994663040, 
            data = {ptr = 0x7ff5, fd = 32757, u32 = 32757, u64 = 32757}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 2, data = {ptr = 0x0, fd = 0, u32 = 0, 
              u64 = 0}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 0, data = {ptr = 0x7ff5, fd = 32757, u32 = 32757, u64 = 32757}}, {events = 32512, data = {
              ptr = 0x7ff5ee19b880 <boost::system::detail::system_cat_holder<void>::instance>, fd = -300304256, u32 = 3994663040, u64 = 140694238378112}}, {events = 0, data = {ptr = 0x0, 
              fd = 0, u32 = 0, u64 = 0}}, {events = 0, data = {ptr = 0x2, fd = 2, u32 = 2, u64 = 2}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 0, data = {
              ptr = 0x7ff500000000, fd = 0, u32 = 0, u64 = 140690243715072}}, {events = 0, data = {ptr = 0xee19b88000000000, fd = 0, u32 = 0, u64 = 17156947115339939840}}, {events = 32757, 
            data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 0, data = {ptr = 0x200000000, fd = 0, u32 = 0, u64 = 8589934592}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, 
              u64 = 0}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 32757, data = {ptr = 0x7f0000000000, fd = 0, u32 = 0, u64 = 139637976727552}}, {
            events = 3994663040, data = {ptr = 0x7ff5, fd = 32757, u32 = 32757, u64 = 32757}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 2, data = {ptr = 0x0, 
              fd = 0, u32 = 0, u64 = 0}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 0, data = {ptr = 0x5613, fd = 22035, u32 = 22035, u64 = 22035}}, {
            events = 22016, data = {ptr = 0x7ff5ee19b880 <boost::system::detail::system_cat_holder<void>::instance>, fd = -300304256, u32 = 3994663040, u64 = 140694238378112}}, {
            events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 0, data = {ptr = 0x2, fd = 2, u32 = 2, u64 = 2}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, 
              u64 = 0}}, {events = 0, data = {ptr = 0x7ff500000000, fd = 0, u32 = 0, u64 = 140690243715072}}, {events = 0, data = {ptr = 0xee19b88000007f00, fd = 32512, u32 = 32512, 
              u64 = 17156947115339972352}}, {events = 32757, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 0, data = {ptr = 0x200000000, fd = 0, u32 = 0, u64 = 8589934592}}, {
            events = 0, data = {ptr = 0x7ff5e9abce0d, fd = -374616563, u32 = 3920350733, u64 = 140694164065805}}, {events = 474, data = {ptr = 0xea294e6800000000, fd = 0, u32 = 0, 
              u64 = 16873103687503839232}}, {events = 32757, data = {ptr = 0x7, fd = 7, u32 = 7, u64 = 7}}, {events = 3928575592, data = {ptr = 0xea2947a000007ff5, fd = 32757, u32 = 32757, 
              u64 = 16873096231440646133}}, {events = 32757, data = {ptr = 0x7ff5edc997d3 <libtorrent::bt_peer_connection::dispatch_message(int)+259>, fd = -305555501, u32 = 3989411795, 
              u64 = 140694233126867}}, {events = 2, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 3990525661, data = {
              ptr = 0x500007ff5, fd = 32757, u32 = 32757, u64 = 21474869237}}, {events = 0, data = {ptr = 0x7ff5e8043870, fd = -402376592, u32 = 3892590704, u64 = 140694136305776}}, {
            events = 4, data = {ptr = 0x1da00000000, fd = 0, u32 = 0, u64 = 2035814498304}}, {events = 0, data = {ptr = 0x7ff5ea2947a0, fd = -366393440, u32 = 3928573856, 
              u64 = 140694172288928}}, {events = 475, data = {ptr = 0xe804388000000000, fd = 0, u32 = 0, u64 = 16718549839113093120}}, {events = 32757, data = {
              ptr = 0x7ff5ef1f8638 <__libc_single_threaded>, fd = -283146696, u32 = 4011820600, u64 = 140694255535672}}, {events = 3928575592, data = {ptr = 0xedc9c71a00007ff5, fd = 32757, 
              u32 = 32757, u64 = 17134445171790872565}}, {events = 32757, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 3892590704, data = {ptr = 0x200007ff5, fd = 32757, 
              u32 = 32757, u64 = 8589967349}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 3928573856, data = {ptr = 0xedd50ff200007ff5, fd = 32757, u32 = 32757, 
              u64 = 17137621489084825589}}, {events = 32757, data = {ptr = 0x1db, fd = 475, u32 = 475, u64 = 475}}, {events = 3920350733, data = {ptr = 0xed3059d800007ff5, fd = 32757, 
              u32 = 32757, u64 = 17091259370118873077}}, {events = 32757, data = {ptr = 0x7ff5ed305998, fd = -315598440, u32 = 3979368856, u64 = 140694223083928}}, {events = 3979368864, 
            data = {ptr = 0xed3059ac00007ff5, fd = 32757, u32 = 32757, u64 = 17091259181140312053}}, {events = 32757, data = {ptr = 0x7ff5ea294db4, fd = -366391884, u32 = 3928575412, 
              u64 = 140694172290484}}, {events = 3928574196, data = {ptr = 0xed3059b000007ff5, fd = 32757, u32 = 32757, u64 = 17091259198320181237}}, {events = 32757, data = {
              ptr = 0x5613db2fa638, fd = -617634248, u32 = 3677333048, u64 = 94643281700408}}, {events = 3677333048, data = {ptr = 0x100005613, fd = 22035, u32 = 22035, u64 = 4294989331}}, 
          {events = 0, data = {ptr = 0x2, fd = 2, u32 = 2, u64 = 2}}, {events = 3928573856, data = {ptr = 0xea29489800007ff5, fd = 32757, u32 = 32757, u64 = 16873097296592535541}}, {
            events = 32757, data = {ptr = 0x66cd, fd = 26317, u32 = 26317, u64 = 26317}}, {events = 3892590704, data = {ptr = 0xea29479000007ff5, fd = 32757, u32 = 32757, 
              u64 = 16873096162721169397}}, {events = 32757, data = {ptr = 0x7ff5ed305ac0, fd = -315598144, u32 = 3979369152, u64 = 140694223084224}}, {events = 3928573856, data = {
              ptr = 0xed305ac000007ff5, fd = 32757, u32 = 32757, u64 = 17091260366551285749}}, {events = 32757, data = {
              ptr = 0x7ff5edd3abac <libtorrent::peer_connection::peer_log(libtorrent::peer_log_alert::direction_t, char const*, char const*, ...) const+412>, fd = -304895060, 
              u32 = 3990072236, u64 = 140694233787308}}, {events = 3928575412, data = {ptr = 0xedf04e9100007ff5, fd = 32757, u32 = 32757, u64 = 17145290166076735477}}, {events = 32757, 
            data = {ptr = 0x7ff5edf04c5b, fd = -303018917, u32 = 3991948379, u64 = 140694235663451}}, {events = 3677333048, data = {ptr = 0xed305a3000000002, fd = 2, u32 = 2, 
              u64 = 17091259748075962370}}, {events = 32757, data = {ptr = 0x7ff5edcbeb58 <libtorrent::disk_job_pool::free_jobs(libtorrent::disk_io_job**, int)+376>, fd = -305403048, 
              u32 = 3989564248, u64 = 140694233279320}}, {events = 3892590720, data = {ptr = 0x7ff5, fd = 32757, u32 = 32757, u64 = 32757}}, {events = 32757, data = {ptr = 0x561300000000, 
              fd = 0, u32 = 0, u64 = 94639604367360}}, {events = 3979369016, data = {ptr = 0xed305ad000007ff5, fd = 32757, u32 = 32757, u64 = 17091260435270762485}}, {events = 32757, 
            data = {ptr = 0x7ff5ed3059f0, fd = -315598352, u32 = 3979368944, u64 = 140694223084016}}, {events = 1, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 0, data = {
              ptr = 0x7ff5e8eaa110, fd = -387276528, u32 = 3907690768, u64 = 140694151405840}}, {events = 1, data = {ptr = 0xdb2fa77800000000, fd = 0, u32 = 0, 
              u64 = 15794026552049532928}}, {events = 22035, data = {ptr = 0x7ff5ed305a30, fd = -315598288, u32 = 3979369008, u64 = 140694223084080}}, {events = 3489668944, data = {
              ptr = 0xedcaeddf00007ff5, fd = 32757, u32 = 32757, u64 = 17134769274317996021}}, {events = 32757, data = {ptr = 0x7ff5e8eaa110, fd = -387276528, u32 = 3907690768, 
              u64 = 140694151405840}}, {events = 3893290144, data = {ptr = 0xed305ae000007ff5, fd = 32757, u32 = 32757, u64 = 17091260503990239221}}, {events = 32757, data = {
              ptr = 0x7ff5edd505c6 <libtorrent::utp_stream::async_read_some<boost::asio::mutable_buffers_1, libtorrent::aux::allocating_handler<std::_Bind<void (libtorrent::peer_connection::*(std::shared_ptr<libtorrent::peer_connection>, std::_Placeholder<1>, std::_Placeholder<2>))(boost::system::error_code const&, unsigned long)>, 342ul> >(boost::asio::mutable_buffers_1 const&, libtorrent::aux::allocating_handler<std::_Bind<void (libtorrent::peer_connection::*(std::shared_ptr<libtorrent::peer_connection>, std::_Placeholder<1>, std::_Placeholder<2>))(boost::system::error_code const&, unsigned long)>, 342ul> const&)+150>, fd = -304806458, u32 = 3990160838, u64 = 140694233875910}}, {events = 3893157856, data = {ptr = 0x7ff5, fd = 32757, u32 = 32757, 
              u64 = 32757}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 0, data = {ptr = 0xe804388000000000, fd = 0, u32 = 0, u64 = 16718549839113093120}}, {
            events = 32757, data = {ptr = 0x7ff5e80ee4a0, fd = -401677152, u32 = 3893290144, u64 = 140694137005216}}, {events = 3893105488, data = {ptr = 0xea29479000007ff5, fd = 32757, 
--Type <RET> for more, q to quit, c to continue without paging--
              u32 = 32757, u64 = 16873096162721169397}}, {events = 32757, data = {ptr = 0x7ff5e80ee490, fd = -401677168, u32 = 3893290128, u64 = 140694137005200}}, {events = 3920334811, 
            data = {ptr = 0xea2947a000007ff5, fd = 32757, u32 = 32757, u64 = 16873096231440646133}}, {events = 32757, data = {
              ptr = 0x7ff5edeaf758 <libtorrent::utp_stream::issue_read()+72>, fd = -303368360, u32 = 3991598936, u64 = 140694235314008}}, {events = 3928573840, data = {
              ptr = 0x66cd00007ff5, fd = 32757, u32 = 32757, u64 = 113030654361589}}, {events = 0, data = {ptr = 0x7ff5ea294838, fd = -366393288, u32 = 3928574008, u64 = 140694172289080}}, 
          {events = 3990141203, data = {ptr = 0xe9ab8fdb00007ff5, fd = 32757, u32 = 32757, u64 = 16837709802615373813}}, {events = 32757, data = {ptr = 0x66cd, fd = 26317, u32 = 26317, 
              u64 = 26317}}, {events = 3990142800, data = {ptr = 0x7ff5, fd = 32757, u32 = 32757, u64 = 32757}}, {events = 0, data = {ptr = 0x7ff5ea2947a0, fd = -366393440, 
              u32 = 3928573856, u64 = 140694172288928}}, {events = 3928573840, data = {ptr = 0xea294a5000007ff5, fd = 32757, u32 = 32757, u64 = 16873099186378145781}}, {events = 32757, 
            data = {ptr = 0x7ff5ea294838, fd = -366393288, u32 = 3928574008, u64 = 140694172289080}}, {events = 3676723152, data = {ptr = 0x1da00005613, fd = 22035, u32 = 22035, 
              u64 = 2035814520339}}, {events = 0, data = {ptr = 0x7ff5ea2947a0, fd = -366393440, u32 = 3928573856, u64 = 140694172288928}}, {events = 0, data = {ptr = 0xed305c5800000000, 
              fd = 0, u32 = 0, u64 = 17091262118897909760}}, {events = 32757, data = {ptr = 0x1da, fd = 474, u32 = 474, u64 = 474}}, {events = 3928573856, data = {ptr = 0xedd4c0a500000001, 
              fd = 1, u32 = 1, u64 = 17137534296953716737}}, {events = 32757, data = {ptr = 0x7ff500002c00, fd = 11264, u32 = 11264, u64 = 140690243726336}}, {events = 0, data = {
              ptr = 0xea2947a000000000, fd = 0, u32 = 0, u64 = 16873096231440613376}}, {events = 32757, data = {ptr = 0x7ff5ea294790, fd = -366393456, u32 = 3928573840, 
              u64 = 140694172288912}}, {events = 3928573856, data = {ptr = 0x7ff5, fd = 32757, u32 = 32757, u64 = 32757}}, {events = 0, data = {ptr = 0x7f0000000000, fd = 0, u32 = 0, 
              u64 = 139637976727552}}, {events = 317466, data = {ptr = 0x400000000, fd = 0, u32 = 0, u64 = 17179869184}}, {events = 0, data = {ptr = 0x1ffffff80, fd = -128, 
              u32 = 4294967168, u64 = 8589934464}}, {events = 3677333600, data = {ptr = 0xdb2fa86000005613, fd = 22035, u32 = 22035, u64 = 15794027548481967635}}, {events = 22035, data = {
              ptr = 0x7ff5ed305cb0, fd = -315597648, u32 = 3979369648, u64 = 140694223084720}}, {events = 3677333368, data = {ptr = 0xdb2fa80000005613, fd = 22035, u32 = 22035, 
              u64 = 15794027136165107219}}, {events = 22035, data = {ptr = 0x5613db2fa778, fd = -617633928, u32 = 3677333368, u64 = 94643281700728}}, {events = 3677333504, data = {
              ptr = 0xdb2f968000005613, fd = 22035, u32 = 22035, u64 = 15794007894711621139}}, {events = 22035, data = {ptr = 0x7ff5ed305d10, fd = -315597552, u32 = 3979369744, 
              u64 = 140694223084816}}, {events = 2756888832, data = {ptr = 0xe8c86cc02ce84f92, fd = 753422226, u32 = 753422226, u64 = 16773776384784355218}}, {events = 32757, data = {
              ptr = 0xffffffffffffff80, fd = -128, u32 = 4294967168, u64 = 18446744073709551488}}, {events = 11, data = {ptr = 0xdb2fa77800000000, fd = 0, u32 = 0, 
              u64 = 15794026552049532928}}, {events = 22035, data = {ptr = 0x7ff5edcaecf0 <libtorrent::disk_io_thread::call_job_handlers()>, fd = -305468176, u32 = 3989499120, 
              u64 = 140694233214192}}, {events = 0, data = {ptr = 0xd0001f5000000000, fd = 0, u32 = 0, u64 = 14988013988346855424}}, {events = 32757, data = {
              ptr = 0x7ff5ef0a5955 <free+85>, fd = -284534443, u32 = 4010432853, u64 = 140694254147925}}, {events = 3676723152, data = {ptr = 0xd0001f5000005613, fd = 22035, u32 = 22035, 
              u64 = 14988013988346877459}}, {events = 32757, data = {ptr = 0x5613db2fa778, fd = -617633928, u32 = 3677333368, u64 = 94643281700728}}, {events = 3989499120, data = {
              ptr = 0x7ff5, fd = 32757, u32 = 32757, u64 = 32757}}, {events = 0, data = {
              ptr = 0x7ff5edcb8343 <boost::asio::detail::completion_handler<std::_Bind<void (libtorrent::disk_io_thread::*(libtorrent::disk_io_thread*))()>, boost::asio::io_context::basic_executor_type<std::allocator<void>, 0u> >::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long)+99>, fd = -305429693, 
              u32 = 3989537603, u64 = 140694233252675}}}
        num_events = <optimized out>
        check_timers = <optimized out>
#2  0x00007ff5edc8b41a in boost::asio::detail::scheduler::do_run_one (this=this@entry=0x5613db2657d0, lock=..., this_thread=..., ec=...)
    at /grid/0/jenkins/cmf/build/redhat9/boost//include/boost/asio/detail/impl/scheduler.ipp:465
        on_exit = {scheduler_ = 0x5613db2657d0, lock_ = 0x7ff5ed305d10, this_thread_ = 0x7ff5ed305d40}
        o = <optimized out>
        more_handlers = <optimized out>
#3  0x00007ff5edd8fae1 in boost::asio::detail::scheduler::run (this=0x5613db2657d0, ec=...) at /grid/0/jenkins/cmf/build/redhat9/boost//include/boost/asio/detail/impl/scheduler.ipp:204
        this_thread = {<boost::asio::detail::thread_info_base> = {<boost::asio::detail::noncopyable> = {<No data fields>}, reusable_memory_ = {0x7ff5e8c86cc0, 0x0, 0x0}, 
            has_pending_exception_ = 0, pending_exception_ = {_M_exception_object = 0x0}}, private_op_queue = {<boost::asio::detail::noncopyable> = {<No data fields>}, front_ = 0x0, 
            back_ = 0x0}, private_outstanding_work = 0}
        ctx = {<boost::asio::detail::noncopyable> = {<No data fields>}, key_ = 0x5613db2657d0, value_ = 0x7ff5ed305d40, next_ = 0x0}
        lock = {<boost::asio::detail::noncopyable> = {<No data fields>}, mutex_ = @0x5613db265800, locked_ = false}
        n = 3946903
#4  0x00007ff5edd8d1c6 in boost::asio::io_context::run (this=<optimized out>) at /grid/0/jenkins/cmf/build/redhat9/boost//include/boost/asio/impl/io_context.ipp:63
        ec = {val_ = 0, failed_ = false, cat_ = 0x7ff5ee19b880 <boost::system::detail::system_cat_holder<void>::instance>}
        s = <optimized out>
        ec = {val_ = <optimized out>, failed_ = <optimized out>, cat_ = <optimized out>}
        s = <optimized out>
#5  operator() (__closure=<optimized out>) at session.cpp:357
        this = <optimized out>
        this = <optimized out>
#6  std::__invoke_impl<void, libtorrent::session::start(libtorrent::session_params&&, libtorrent::io_service*)::<lambda()> > (__f=...) at /usr/include/c++/11/bits/invoke.h:61
No locals.
#7  std::__invoke<libtorrent::session::start(libtorrent::session_params&&, libtorrent::io_service*)::<lambda()> > (__fn=...) at /usr/include/c++/11/bits/invoke.h:96
No locals.
#8  std::thread::_Invoker<std::tuple<libtorrent::session::start(libtorrent::session_params&&, libtorrent::io_service*)::<lambda()> > >::_M_invoke<0> (this=<optimized out>)
    at /usr/include/c++/11/bits/std_thread.h:253

[arvidn]

if you're using libtorrent 1.2, I would highly recommend using the most recent version. i.e. https://github.com/arvidn/libtorrent/releases/tag/v1.2.19

I am not sure why I had to do any of these things?
obj.get_io_service() -> (boost::asio::io_context&)(obj).get_executor().context();

That looks super shady. That's a reinterpret cast. If you use a boost version that's compatible with the libtorrent version you're using, you shouldn't need to patch up anything like that. Perhaps just updating to libtorrent 1.2.9 would help.

What does "systemctl restart peer" do?

Can anyone suggest what is happening in the logs?
How to debug this?

You can take a look a this: https://libtorrent.org/troubleshooting.html

I tried to capture the core dump during when this distribution process gets stuck. I think some type of wait is happening which might be related why the above logs get stuck?

When you say the process is "stuck", do you mean that that the process stops entirely? like a deadlock, or do you just mean it stops making progress on the download?