Add rebalance.py as c-lightning plugin #1
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
pyln-client>=0.7.3 required as dependency @Kukks @rockstardev @pavlenex |
|
Do these need to be included at compile time? Maybe we can map a local directory to the plugins dir once in the docker setup |
|
I'm not familiar with c-lightning so can't really comment on this PR ;( |
NicolasDorier
force-pushed
the
btcpaymaster
branch
from
b54959a
to
53f0a73
Compare
Kukks
pushed a commit
that referenced
this pull request
Aug 10, 2021
e.g. in test_closing_id we can get a spend from the first (closed) channel in the same block as the open of the second. Half the time, we'll choose the wrong one as scid. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
NicolasDorier
pushed a commit
that referenced
this pull request
Nov 25, 2021
The variable `block` (instace of `struct block`) is
allocated on the stack without being initialized, i.e. its
member `prev` points to nowhere. This causes a segmentation
fault on my machine on the binding of "prev_hash" on running
`wallet_block_add`, as the following core-dump analysis
shows:
$ egdb ./wallet/test/run-wallet ./run-wallet.core
[...]
Core was generated by `run-wallet'.
Program terminated with signal SIGSEGV, Segmentation fault.
---Type <return> to continue, or q <return> to quit---
#0 0x000008f67a04b660 in memcpy (dst0=<optimized out>, src0=0x100007f8c, length=32) at /usr/src/lib/libc/string/memcpy.c:97
97 TLOOP1(*dst++ = *src++);
(gdb) bt
#0 0x000008f67a04b660 in memcpy (dst0=<optimized out>, src0=0x100007f8c, length=32) at /usr/src/lib/libc/string/memcpy.c:97
#1 0x000008f73e838f60 in sqlite3VdbeMemSetStr () from /usr/local/lib/libsqlite3.so.37.12
#2 0x000008f73e83cb11 in bindText () from /usr/local/lib/libsqlite3.so.37.12
#3 0x000008f44bc91345 in db_sqlite3_query (stmt=0x8f6845bf028) at wallet/db_sqlite3.c:77
#4 0x000008f44bc91122 in db_sqlite3_exec (stmt=0x8f6845bf028) at wallet/db_sqlite3.c:110
ElementsProject#5 0x000008f44bcbb3b2 in db_exec_prepared_v2 (stmt=0x8f6845bf028) at ./wallet/db.c:2055
ElementsProject#6 0x000008f44bcc6890 in wallet_block_add (w=0x8f688b5bba8, b=0x7f7ffffca788) at ./wallet/wallet.c:3556
ElementsProject#7 0x000008f44bce2607 in test_wallet_outputs (ld=0x8f6a35a7828, ctx=0x8f6a35c0268) at wallet/test/run-wallet.c:1104
ElementsProject#8 0x000008f44bcddec0 in main (argc=1, argv=0x7f7ffffcaaf8) at wallet/test/run-wallet.c:1930
Fix by explicitely setting the whole structure to zero.
[ Rebuilt generated files, too --RR ]
NicolasDorier
force-pushed
the
btcpaymaster
branch
2 times, most recently
from
45782ce
to
ff2a16c
Compare
NicolasDorier
pushed a commit
that referenced
this pull request
Aug 24, 2023
This will fix a crash that I caused on armv7 and by looking inside the coredump with gdb (by adding an assert on n that must be different from null) I get the following stacktrace ``` (gdb) bt \#0 0x00000000 in ?? () \#1 0x0043a038 in send_backtrace (why=0xbe9e3600 "FATAL SIGNAL 11") at common/daemon.c:36 \#2 0x0043a0ec in crashdump (sig=11) at common/daemon.c:46 \#3 <signal handler called> \#4 0x00406d04 in node_announcement (map=0x938ecc, nann_off=495146) at common/gossmap.c:586 \ElementsProject#5 0x00406fec in map_catchup (map=0x938ecc, num_rejected=0xbe9e3a40) at common/gossmap.c:643 \ElementsProject#6 0x004073a4 in load_gossip_store (map=0x938ecc, num_rejected=0xbe9e3a40) at common/gossmap.c:697 \ElementsProject#7 0x00408244 in gossmap_load (ctx=0x0, filename=0x4e16b8 "gossip_store", num_channel_updates_rejected=0xbe9e3a40) at common/gossmap.c:976 \ElementsProject#8 0x0041a548 in init (p=0x93831c, buf=0x9399d4 "\n\n{\"jsonrpc\":\"2.0\",\"id\":\"cln:init#25\",\"method\":\"init\",\"params\":{\"options\":{},\"configuration\":{\"lightning-dir\":\"/home/vincent/.lightning/testnet\",\"rpc-file\":\"lightning-rpc\",\"startup\":true,\"network\":\"te"..., config=0x939cdc) at plugins/topology.c:622 \ElementsProject#9 0x0041e5d0 in handle_init (cmd=0x938934, buf=0x9399d4 "\n\n{\"jsonrpc\":\"2.0\",\"id\":\"cln:init#25\",\"method\":\"init\",\"params\":{\"options\":{},\"configuration\":{\"lightning-dir\":\"/home/vincent/.lightning/testnet\",\"rpc-file\":\"lightning-rpc\",\"startup\":true,\"network\":\"te"..., params=0x939c8c) at plugins/libplugin.c:1208 \ElementsProject#10 0x0041fc04 in ld_command_handle (plugin=0x93831c, toks=0x939bec) at plugins/libplugin.c:1572 \ElementsProject#11 0x00420050 in ld_read_json_one (plugin=0x93831c) at plugins/libplugin.c:1667 \ElementsProject#12 0x004201bc in ld_read_json (conn=0x9391c4, plugin=0x93831c) at plugins/libplugin.c:1687 \ElementsProject#13 0x004cb82c in next_plan (conn=0x9391c4, plan=0x9391d8) at ccan/ccan/io/io.c:59 \ElementsProject#14 0x004cc67c in do_plan (conn=0x9391c4, plan=0x9391d8, idle_on_epipe=false) at ccan/ccan/io/io.c:407 \ElementsProject#15 0x004cc6dc in io_ready (conn=0x9391c4, pollflags=1) at ccan/ccan/io/io.c:417 \ElementsProject#16 0x004cf8cc in io_loop (timers=0x9383c4, expired=0xbe9e3ce4) at ccan/ccan/io/poll.c:453 \ElementsProject#17 0x00420af4 in plugin_main (argv=0xbe9e3eb4, init=0x41a46c <init>, restartability=PLUGIN_STATIC, init_rpc=true, features=0x0, commands=0x6167e8 <commands>, num_commands=4, notif_subs=0x0, num_notif_subs=0, hook_subs=0x0, num_hook_subs=0, notif_topics=0x0, num_notif_topics=0) at plugins/libplugin.c:1891 \ElementsProject#18 0x0041a6f8 in main (argc=1, argv=0xbe9e3eb4) at plugins/topology.c:679 ``` I do not know if this is a solution because I do not know when I can parse a node announcement for a node that it is not longer in the gossip map. So, I hope this is just usefult for @rustyrussell Changelog-Fixed: fixes `FATAL SIGNAL 11` on gossmap node announcement parsing. Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
NicolasDorier
pushed a commit
that referenced
this pull request
Aug 24, 2023
The issue is that common_setup() wasn't called by the fuzz target,
leaving secp256k1_ctx as NULL.
UBSan error:
$ UBSAN_OPTIONS="print_stacktrace=1:halt_on_error=1" \
./fuzz-channel_id crash-1575b41ef09e62e4c09c165e6dc037a110b113f2
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 1153355603
INFO: Loaded 1 modules (25915 inline 8-bit counters): 25915 [0x563bae7ac3a8, 0x563bae7b28e3),
INFO: Loaded 1 PC tables (25915 PCs): 25915 [0x563bae7b28e8,0x563bae817c98),
./fuzz-channel_id: Running 1 inputs 1 time(s) each.
Running: crash-1575b41ef09e62e4c09c165e6dc037a110b113f2
bitcoin/pubkey.c:22:33: runtime error: null pointer passed as argument 1, which is declared to never be null
external/libwally-core/src/secp256k1/include/secp256k1.h:373:3: note: nonnull attribute specified here
#0 0x563bae41e3db in pubkey_from_der bitcoin/pubkey.c:19:7
#1 0x563bae4205e0 in fromwire_pubkey bitcoin/pubkey.c:111:7
#2 0x563bae46437c in run tests/fuzz/fuzz-channel_id.c:42:3
#3 0x563bae2f6016 in LLVMFuzzerTestOneInput tests/fuzz/libfuzz.c:23:2
#4 0x563bae20a450 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long)
ElementsProject#5 0x563bae1f4c3f in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long)
ElementsProject#6 0x563bae1fa6e6 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long))
ElementsProject#7 0x563bae223052 in main (tests/fuzz/fuzz-channel_id+0x181052) (BuildId: f7f56e14ffc06df54ab732d79ea922e773de1f25)
ElementsProject#8 0x7fa7fa113082 in __libc_start_main
ElementsProject#9 0x563bae1efbdd in _start
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior bitcoin/pubkey.c:22:33 in
NicolasDorier
pushed a commit
that referenced
this pull request
Aug 24, 2023
Detected by UBSan:
$ UBSAN_OPTIONS=print_stacktrace=1 ./wallet/test/run-psbt_fixup
bitcoin/psbt.c:733:2: runtime error: applying zero offset to null pointer
#0 0x53c829 in psbt_from_bytes lightning/bitcoin/psbt.c:733:2
#1 0x5adcb0 in main lightning/wallet/test/run-psbt_fixup.c:174:10
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior bitcoin/psbt.c:733:2
NicolasDorier
pushed a commit
that referenced
this pull request
Aug 24, 2023
The function is tiny and was only used in one location. And that one
location was leaking memory.
Detected by ASan:
==2637667==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 7 byte(s) in 1 object(s) allocated from:
#0 0x4cd758 in __interceptor_strdup
#1 0x64c70c in json_stream_log_suppress_for_cmd lightning/lightningd/jsonrpc.c:597:31
#2 0x68a630 in json_getlog lightning/lightningd/log.c:974:2
...
SUMMARY: AddressSanitizer: 7 byte(s) leaked in 1 allocation(s).
NicolasDorier
pushed a commit
that referenced
this pull request
Aug 24, 2023
It is possible for db_column_bytes() to return 0 and for db_column_blob() to return NULL even when db_column_is_null() returns false. We need to short circuit in this case. Detected by UBSan: db/bindings.c:479:12: runtime error: null pointer passed as argument 2, which is declared to never be null /usr/include/string.h:44:28: note: nonnull attribute specified here #0 0x95f117 in db_col_arr_ db/bindings.c:479:2 #1 0x95ef85 in db_col_channel_type db/bindings.c:459:32 #2 0x852c03 in wallet_stmt2channel wallet/wallet.c:1483:9 #3 0x81f396 in wallet_channels_load_active wallet/wallet.c:1749:23 #4 0x81f03d in wallet_init_channels wallet/wallet.c:1765:9 ElementsProject#5 0x72f1f9 in load_channels_from_wallet lightningd/peer_control.c:2257:7 ElementsProject#6 0x672856 in main lightningd/lightningd.c:1121:25
NicolasDorier
pushed a commit
that referenced
this pull request
Aug 24, 2023
Fixes nullability errors detected by UBSan:
wire/fromwire.c:173:46: runtime error: null pointer passed as argument 1, which is declared to never be null
external/libwally-core/src/secp256k1/include/secp256k1.h:432:3: note: nonnull attribute specified here
#0 0x65214a in fromwire_secp256k1_ecdsa_signature wire/fromwire.c:173:6
#1 0x659500 in printwire_secp256k1_ecdsa_signature devtools/print_wire.c:331:1
#2 0x646ba2 in printwire_channel_update wire/peer_printgen.c:1900:7
#3 0x637182 in printpeer_wire_message wire/peer_printgen.c:128:11
#4 0x65a097 in main devtools/decodemsg.c:85:10
NicolasDorier
pushed a commit
that referenced
this pull request
Aug 24, 2023
Memory leak detected by ASan:
==880002==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 32816 byte(s) in 1 object(s) allocated from:
#0 0x5039e7 in malloc (lightningd/lightningd+0x5039e7)
#1 0x7f2e8c203884 in __alloc_dir (/lib64/libc.so.6+0xd2884)
NicolasDorier
pushed a commit
that referenced
this pull request
Aug 24, 2023
Detected by ASan in test_hsmtool_generatehsm:
==58698==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 120 byte(s) in 1 object(s) allocated from:
#0 0x4e6247 in malloc
#1 0x7f078452d672 in getdelim
SUMMARY: AddressSanitizer: 120 byte(s) leaked in 1 allocation(s).
NicolasDorier
pushed a commit
that referenced
this pull request
Aug 24, 2023
These show that we should clean up our notes. Here's the result from test_hardmpp: # we have computed a set of 1 flows with probability 0.328, fees 0msat and delay 23 # No MPP, so added 0msat shadow fee # Shadow route on flow 0/1 added 0 block delay. now 5 # sendpay flow groupid=1, partid=1, delivering=1800000000msat, probability=0.328 # Update chan knowledge scid=103x2x0, dir=0: [0msat,1799999999msat] # onion error WIRE_TEMPORARY_CHANNEL_FAILURE from node #1 103x2x0: failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote) # we have computed a set of 2 flows with probability 0.115, fees 0msat and delay 23 # Shadow route on flow 0/2 added 0 block delay. now 5 # Shadow route on flow 1/2 added 0 block delay. now 5 # sendpay flow groupid=1, partid=3, delivering=500000000msat, probability=0.475 # sendpay flow groupid=1, partid=2, delivering=1300000000msat, probability=0.242 # Update chan knowledge scid=103x2x0, dir=0: [0msat,1299999999msat] # onion error WIRE_TEMPORARY_CHANNEL_FAILURE from node #1 103x2x0: failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote) # we have computed a set of 2 flows with probability 0.084, fees 0msat and delay 23 # Shadow route on flow 0/2 added 0 block delay. now 5 # Shadow route on flow 1/2 added 0 block delay. now 5 # sendpay flow groupid=1, partid=5, delivering=260000000msat, probability=0.467 # sendpay flow groupid=1, partid=4, delivering=1040000000msat, probability=0.179 # Update chan knowledge scid=103x2x0, dir=0: [0msat,1039999999msat] # onion error WIRE_TEMPORARY_CHANNEL_FAILURE from node #1 103x2x0: failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote) # we have computed a set of 2 flows with probability 0.052, fees 0msat and delay 23 # Shadow route on flow 0/2 added 0 block delay. now 5 # Shadow route on flow 1/2 added 0 block delay. now 5 # sendpay flow groupid=1, partid=7, delivering=120000000msat, probability=0.494 # sendpay flow groupid=1, partid=6, delivering=920000000msat, probability=0.105 Ideally it would look something like: # Computed 1 flows, probability=0.328: # Flow 1: 103x2x0 1800000000msat fee=0msat probability=0.328 shadow=+0msat/0blocks # Flow 1: FAIL: TEMPORARY_CHANNEL_FAILURE for 103x2x0. # Computed 2 flows, probability=0.115: # Flow 2: XXX->XXX 1300000000msat fee=XXX, probability=0.475 shadow=+0msat/0blocks # Flow 3: XXX->XXX 500000000msat fee=XXX, probability=0.475 shadow=+0msat/0blocks # Flow 2: FAIL: TEMPORARY_CHANNEL_FAILURE from node #1 103x2x0 # Computed 2 flows (3 total), probability=0.084, fee=0msat, delay=23 ... # Flow 4: SUCCESS, 2 in progress should succeed soon. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
NicolasDorier
pushed a commit
that referenced
this pull request
Aug 24, 2023
It now looks like (for test_hardmpp): ``` # we have computed a set of 1 flows with probability 0.328, fees 0msat and delay 23 # Flow 1: amount=1800000000msat prob=0.328 fees=0msat delay=12 path=-103x2x0/1(min=max=4294967295msat)->-103x5x0/0->-103x3x0/1-> # Flow 1: Failed at node #1 (WIRE_TEMPORARY_CHANNEL_FAILURE): failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote) # Flow 1: Failure of 1800000000msat for 103x5x0/0 capacity [0msat,3000000000msat] -> [0msat,1799999999msat] # we have computed a set of 2 flows with probability 0.115, fees 0msat and delay 23 # Flow 2: amount=500000000msat prob=0.475 fees=0msat delay=12 path=-103x6x0/0(min=max=4294967295msat)->-103x1x0/1->-103x4x0/1-> # Flow 3: amount=1300000000msat prob=0.242 fees=0msat delay=12 path=-103x2x0/1(min=max=4294967295msat)->-103x5x0/0(max=1799999999msat)->-103x3x0/1-> # Flow 3: Failed at node #1 (WIRE_TEMPORARY_CHANNEL_FAILURE): failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote) # Flow 3: Failure of 1300000000msat for 103x5x0/0 capacity [0msat,1799999999msat] -> [0msat,1299999999msat] # we have computed a set of 2 flows with probability 0.084, fees 0msat and delay 23 # Flow 4: amount=260000000msat prob=0.467 fees=0msat delay=12 path=-103x6x0/0(500000000msat in 1 htlcs,min=max=4294967295msat)->-103x1x0/1(500000000msat in 1 htlcs)->-103x4x0/1(500000000msat in 1 htlcs)-> # Flow 5: amount=1040000000msat prob=0.179 fees=0msat delay=12 path=-103x2x0/1(min=max=4294967295msat)->-103x5x0/0(max=1299999999msat)->-103x3x0/1-> # Flow 5: Failed at node #1 (WIRE_TEMPORARY_CHANNEL_FAILURE): failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote) # Flow 5: Failure of 1040000000msat for 103x5x0/0 capacity [0msat,1299999999msat] -> [0msat,1039999999msat] # we have computed a set of 2 flows with probability 0.052, fees 0msat and delay 23 # Flow 6: amount=120000000msat prob=0.494 fees=0msat delay=12 path=-103x6x0/0(760000000msat in 2 htlcs,min=max=4294967295msat)->-103x1x0/1(760000000msat in 2 htlcs)->-103x4x0/1(760000000msat in 2 htlcs)-> # Flow 7: amount=920000000msat prob=0.105 fees=0msat delay=12 path=-103x2x0/1(min=max=4294967295msat)->-103x5x0/0(max=1039999999msat)->-103x3x0/1-> # Flow 7: Success ``` Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
NicolasDorier
force-pushed
the
btcpaymaster
branch
from
ff2a16c
to
02a2716
Compare
NicolasDorier
pushed a commit
that referenced
this pull request
Apr 9, 2024
Adding an index means: 1. Add the new subsystem, and new updated_index field to the db, and create xxx_index_deleted/created/updated APIs. 2. Hook up these functions to the points they need to be called. 3. Add index, start and limit fields to the list command. 4. Add created_index and updated_index into the list command. This does #1. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
NicolasDorier
force-pushed
the
btcpaymaster
branch
from
994aa55
to
3c3ba81
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
https://github.com/lightningd/plugins/blob/master/rebalance/rebalance.py