Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Index out of bounds in ProposalSender::send #2322

Closed
jsdanielh opened this issue Mar 18, 2024 · 2 comments · Fixed by #2344
Closed

Index out of bounds in ProposalSender::send #2322

jsdanielh opened this issue Mar 18, 2024 · 2 comments · Fixed by #2344
Assignees
@nibhar
Milestone
Nimiq PoS Mainnet

Comments

@jsdanielh
Copy link
Contributor

In the latest running testnet a node reported a panic in the ProposalSender::send code:

Mar 07 12:48:42 validator5 nimiq-client[2127553]: 2024-03-07T12:48:42.791489448Z ERROR panic                | thread 'tokio-runtime-worker' panicked at 'index out of bounds: the len is 8 but the index is 9': primitives/src/slots_allocation.rs:160
Mar 07 12:48:42 validator5 nimiq-client[2127553]:    0: <backtrace::capture::Backtrace as core::default::Default>::default
Mar 07 12:48:42 validator5 nimiq-client[2127553]:    1: log_panics::Config::install_panic_hook::{{closure}}
Mar 07 12:48:42 validator5 nimiq-client[2127553]:    2: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
Mar 07 12:48:42 validator5 nimiq-client[2127553]:              at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/alloc/src/boxed.rs:2021:9
Mar 07 12:48:42 validator5 nimiq-client[2127553]:       std::panicking::rust_panic_with_hook
Mar 07 12:48:42 validator5 nimiq-client[2127553]:              at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/std/src/panicking.rs:735:13
Mar 07 12:48:42 validator5 nimiq-client[2127553]:    3: std::panicking::begin_panic_handler::{{closure}}
Mar 07 12:48:42 validator5 nimiq-client[2127553]:              at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/std/src/panicking.rs:609:13
Mar 07 12:48:42 validator5 nimiq-client[2127553]:    4: std::sys_common::backtrace::__rust_end_short_backtrace
Mar 07 12:48:42 validator5 nimiq-client[2127553]:              at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/std/src/sys_common/backtrace.rs:170:18
Mar 07 12:48:42 validator5 nimiq-client[2127553]:    5: rust_begin_unwind
Mar 07 12:48:42 validator5 nimiq-client[2127553]:              at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/std/src/panicking.rs:597:5
Mar 07 12:48:42 validator5 nimiq-client[2127553]:    6: core::panicking::panic_fmt
Mar 07 12:48:42 validator5 nimiq-client[2127553]:              at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/core/src/panicking.rs:72:14
Mar 07 12:48:42 validator5 nimiq-client[2127553]:    7: core::panicking::panic_bounds_check
Mar 07 12:48:42 validator5 nimiq-client[2127553]:              at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/core/src/panicking.rs:180:5
Mar 07 12:48:42 validator5 nimiq-client[2127553]:    8: nimiq_validator::proposal_buffer::ProposalSender<TValidatorNetwork>::send
Mar 07 12:48:42 validator5 nimiq-client[2127553]:    9: <futures_util::stream::stream::for_each::ForEach<St,Fut,F> as core::future::future::Future>::poll
Mar 07 12:48:42 validator5 nimiq-client[2127553]:   10: nimiq_validator::validator::Validator<TValidatorNetwork>::new::{{closure}}
Mar 07 12:48:42 validator5 nimiq-client[2127553]:   11: tokio::runtime::task::core::Core<T,S>::poll
Mar 07 12:48:42 validator5 nimiq-client[2127553]:   12: tokio::runtime::task::harness::Harness<T,S>::poll
Mar 07 12:48:42 validator5 nimiq-client[2127553]:   13: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
Mar 07 12:48:42 validator5 nimiq-client[2127553]:   14: tokio::runtime::scheduler::multi_thread::worker::Context::run
Mar 07 12:48:42 validator5 nimiq-client[2127553]:   15: tokio::runtime::context::scoped::Scoped<T>::set
Mar 07 12:48:42 validator5 nimiq-client[2127553]:   16: tokio::runtime::context::runtime::enter_runtime
Mar 07 12:48:42 validator5 nimiq-client[2127553]:   17: tokio::runtime::scheduler::multi_thread::worker::run
Mar 07 12:48:42 validator5 nimiq-client[2127553]:   18: tokio::runtime::task::core::Core<T,S>::poll
Mar 07 12:48:42 validator5 nimiq-client[2127553]:   19: tokio::runtime::task::harness::Harness<T,S>::poll
Mar 07 12:48:42 validator5 nimiq-client[2127553]:   20: tokio::runtime::blocking::pool::Inner::run
Mar 07 12:48:42 validator5 nimiq-client[2127553]:   21: std::sys_common::backtrace::__rust_begin_short_backtrace
Mar 07 12:48:42 validator5 nimiq-client[2127553]:   22: core::ops::function::FnOnce::call_once{{vtable.shim}}
Mar 07 12:48:42 validator5 nimiq-client[2127553]:   23: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
Mar 07 12:48:42 validator5 nimiq-client[2127553]:              at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/alloc/src/boxed.rs:2007:9
Mar 07 12:48:42 validator5 nimiq-client[2127553]:       <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
Mar 07 12:48:42 validator5 nimiq-client[2127553]:              at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/alloc/src/boxed.rs:2007:9
Mar 07 12:48:42 validator5 nimiq-client[2127553]:       std::sys::unix::thread::Thread::new::thread_start
Mar 07 12:48:42 validator5 nimiq-client[2127553]:              at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/std/src/sys/unix/thread.rs:108:17
Mar 07 12:48:42 validator5 nimiq-client[2127553]:   24: <unknown>
Mar 07 12:48:42 validator5 nimiq-client[2127553]:   25: <unknown>
Mar 07 12:48:42 validator5 nimiq-client[2127553]:

It looks like the panic originates from this call:

core-rs-albatross/validator/src/proposal_buffer.rs

Line 377 in 2524dee

let stated_proposer = validators.get_validator_by_slot_band(proposal.0.signer);

@redmaner
Copy link
Contributor

Another trace

Mar 25 18:04:36 vps-a60b17b1 nimiq-client[2603706]: 2024-03-25T18:04:36.557497390Z ERROR panic                | thread 'tokio-runtime-worker' panicked at 'index out of bounds: the len is 11 but the index is 11': primitives/src/slots_allocation.rs:160
Mar 25 18:04:36 vps-a60b17b1 nimiq-client[2603706]:    0: <backtrace::capture::Backtrace as core::default::Default>::default
Mar 25 18:04:36 vps-a60b17b1 nimiq-client[2603706]:    1: log_panics::Config::install_panic_hook::{{closure}}
Mar 25 18:04:36 vps-a60b17b1 nimiq-client[2603706]:    2: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
Mar 25 18:04:36 vps-a60b17b1 nimiq-client[2603706]:              at rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/alloc/src/boxed.rs:2021:9
Mar 25 18:04:36 vps-a60b17b1 nimiq-client[2603706]:       std::panicking::rust_panic_with_hook
Mar 25 18:04:36 vps-a60b17b1 nimiq-client[2603706]:              at rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/std/src/panicking.rs:735:13
Mar 25 18:04:36 vps-a60b17b1 nimiq-client[2603706]:    3: std::panicking::begin_panic_handler::{{closure}}
Mar 25 18:04:36 vps-a60b17b1 nimiq-client[2603706]:              at rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/std/src/panicking.rs:609:13
Mar 25 18:04:36 vps-a60b17b1 nimiq-client[2603706]:    4: std::sys_common::backtrace::__rust_end_short_backtrace
Mar 25 18:04:36 vps-a60b17b1 nimiq-client[2603706]:              at rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/std/src/sys_common/backtrace.rs:170:18
Mar 25 18:04:36 vps-a60b17b1 nimiq-client[2603706]:    5: rust_begin_unwind
Mar 25 18:04:36 vps-a60b17b1 nimiq-client[2603706]:              at rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/std/src/panicking.rs:597:5
Mar 25 18:04:36 vps-a60b17b1 nimiq-client[2603706]:    6: core::panicking::panic_fmt
Mar 25 18:04:36 vps-a60b17b1 nimiq-client[2603706]:              at rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/core/src/panicking.rs:72:14
Mar 25 18:04:36 vps-a60b17b1 nimiq-client[2603706]:    7: core::panicking::panic_bounds_check
Mar 25 18:04:36 vps-a60b17b1 nimiq-client[2603706]:              at rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/core/src/panicking.rs:180:5
Mar 25 18:04:36 vps-a60b17b1 nimiq-client[2603706]:    8: nimiq_validator::proposal_buffer::ProposalSender<TValidatorNetwork>::send
Mar 25 18:04:36 vps-a60b17b1 nimiq-client[2603706]:    9: <futures_util::stream::stream::for_each::ForEach<St,Fut,F> as core::future::future::Future>::poll
Mar 25 18:04:36 vps-a60b17b1 nimiq-client[2603706]:   10: nimiq_validator::validator::Validator<TValidatorNetwork>::new::{{closure}}
Mar 25 18:04:36 vps-a60b17b1 nimiq-client[2603706]:   11: tokio::runtime::task::core::Core<T,S>::poll
Mar 25 18:04:36 vps-a60b17b1 nimiq-client[2603706]:   12: tokio::runtime::task::harness::Harness<T,S>::poll
Mar 25 18:04:36 vps-a60b17b1 nimiq-client[2603706]:   13: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
Mar 25 18:04:36 vps-a60b17b1 nimiq-client[2603706]:   14: tokio::runtime::scheduler::multi_thread::worker::Context::run
Mar 25 18:04:36 vps-a60b17b1 nimiq-client[2603706]:   15: tokio::runtime::context::scoped::Scoped<T>::set
Mar 25 18:04:36 vps-a60b17b1 nimiq-client[2603706]:   16: tokio::runtime::context::runtime::enter_runtime
Mar 25 18:04:36 vps-a60b17b1 nimiq-client[2603706]:   17: tokio::runtime::scheduler::multi_thread::worker::run
Mar 25 18:04:36 vps-a60b17b1 nimiq-client[2603706]:   18: tokio::runtime::task::core::Core<T,S>::poll
Mar 25 18:04:36 vps-a60b17b1 nimiq-client[2603706]:   19: tokio::runtime::task::harness::Harness<T,S>::poll
Mar 25 18:04:36 vps-a60b17b1 nimiq-client[2603706]:   20: tokio::runtime::blocking::pool::Inner::run
Mar 25 18:04:36 vps-a60b17b1 nimiq-client[2603706]:   21: std::sys_common::backtrace::__rust_begin_short_backtrace
Mar 25 18:04:36 vps-a60b17b1 nimiq-client[2603706]:   22: core::ops::function::FnOnce::call_once{{vtable.shim}}
Mar 25 18:04:36 vps-a60b17b1 nimiq-client[2603706]:   23: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
Mar 25 18:04:36 vps-a60b17b1 nimiq-client[2603706]:              at rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/alloc/src/boxed.rs:2007:9
Mar 25 18:04:36 vps-a60b17b1 nimiq-client[2603706]:       <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
Mar 25 18:04:36 vps-a60b17b1 nimiq-client[2603706]:              at rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/alloc/src/boxed.rs:2007:9
Mar 25 18:04:36 vps-a60b17b1 nimiq-client[2603706]:       std::sys::unix::thread::Thread::new::thread_start
Mar 25 18:04:36 vps-a60b17b1 nimiq-client[2603706]:              at rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/std/src/sys/unix/thread.rs:108:17
Mar 25 18:04:36 vps-a60b17b1 nimiq-client[2603706]:   24: <unknown>
Mar 25 18:04:36 vps-a60b17b1 nimiq-client[2603706]:   25: <unknown>

@nibhar
Copy link
Member

nibhar commented Mar 25, 2024

I do have a fix for this in the making. Just checking with a bit more scrutiny if it actually fixes things.

@nibhar nibhar mentioned this issue Apr 3, 2024
Merged
@hrxi hrxi closed this as completed in #2344 May 1, 2024
@jsdanielh jsdanielh added this to the Nimiq PoS Mainnet milestone May 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nibhar nibhar

Labels
None yet
Projects
None yet
Milestone
Nimiq PoS Mainnet
Development

Successfully merging a pull request may close this issue.

Sanity Check proposal sender from proposal messages nimiq/core-rs-albatross
3 participants
@redmaner @jsdanielh @nibhar