Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

tikv error in log: Raft raft: cannot step raft local message #11371

Open
dbsid opened this issue Nov 14, 2021 · 8 comments · May be fixed by tikv/raft-rs#475 or #12443
Open

tikv error in log: Raft raft: cannot step raft local message #11371

dbsid opened this issue Nov 14, 2021 · 8 comments · May be fixed by tikv/raft-rs#475 or #12443
Assignees
@tier-cap
Labels
severity/moderate type/bug Type: Issue - Confirmed a bug

Comments

@dbsid
Copy link
Contributor

dbsid commented Nov 14, 2021

Bug Report

What version of TiKV are you using?

v5.2.2 + pr #11304

What operating system and CPU are you using?

Steps to reproduce

Running sysbench oltp_insert for 24 hours

What did you expect?

no tikv error

What did happened?

[2021/11/13 23:12:14.915 +08:00] [ERROR] [peer.rs:553] ["handle raft message err"] [err_code=KV:Raft:StepLocalMsg] [err="Raft raft: cannot step raft local message"] [peer_id=15443] [region_id=15440]
[2021/11/13 23:12:14.915 +08:00] [ERROR] [peer.rs:553] ["handle raft message err"] [err_code=KV:Raft:StepLocalMsg] [err="Raft raft: cannot step raft local message"] [peer_id=10307] [region_id=10304]
[2021/11/13 23:12:14.915 +08:00] [ERROR] [peer.rs:553] ["handle raft message err"] [err_code=KV:Raft:StepLocalMsg] [err="Raft raft: cannot step raft local message"] [peer_id=17923] [region_id=17920]
[2021/11/13 23:12:14.915 +08:00] [ERROR] [peer.rs:553] ["handle raft message err"] [err_code=KV:Raft:StepLocalMsg] [err="Raft raft: cannot step raft local message"] [peer_id=12343] [region_id=12340]
[2021/11/13 23:12:14.917 +08:00] [ERROR] [peer.rs:553] ["handle raft message err"] [err_code=KV:Raft:StepLocalMsg] [err="Raft raft: cannot step raft local message"] [peer_id=18683] [region_id=18680]
[2021/11/13 23:12:14.923 +08:00] [ERROR] [peer.rs:553] ["handle raft message err"] [err_code=KV:Raft:StepLocalMsg] [err="Raft raft: cannot step raft local message"] [peer_id=16823] [region_id=16820]
[2021/11/13 23:12:14.923 +08:00] [ERROR] [peer.rs:553] ["handle raft message err"] [err_code=KV:Raft:StepLocalMsg] [err="Raft raft: cannot step raft local message"] [peer_id=3459] [region_id=3456]
[2021/11/13 23:12:14.923 +08:00] [ERROR] [peer.rs:553] ["handle raft message err"] [err_code=KV:Raft:StepLocalMsg] [err="Raft raft: cannot step raft local message"] [peer_id=1935] [region_id=1932]
[2021/11/13 23:12:14.923 +08:00] [ERROR] [peer.rs:553] ["handle raft message err"] [err_code=KV:Raft:StepLocalMsg] [err="Raft raft: cannot step raft local message"] [peer_id=15443] [region_id=15440]
[2021/11/13 23:12:14.923 +08:00] [ERROR] [peer.rs:553] ["handle raft message err"] [err_code=KV:Raft:StepLocalMsg] [err="Raft raft: cannot step raft local message"] [peer_id=22299] [region_id=22296]
[2021/11/13 23:12:14.923 +08:00] [ERROR] [peer.rs:553] ["handle raft message err"] [err_code=KV:Raft:StepLocalMsg] [err="Raft raft: cannot step raft local message"] [peer_id=14195] [region_id=14192]```
@dbsid
Copy link
Contributor Author

dbsid commented Nov 14, 2021

/assign gengliqi

@BusyJay
Copy link
Member

BusyJay commented Nov 24, 2021

Any updates?

@gengliqi
Copy link
Member

gengliqi commented Nov 24, 2021 •
edited

Seems it is caused by

tikv/components/raftstore/src/store/fsm/peer.rs

Lines 903 to 910 in 6a0792a

CasualMessage::RejectRaftAppend { peer_id } => {
let mut msg = raft::eraftpb::Message::new();
msg.msg_type = MessageType::MsgUnreachable;
msg.to = peer_id;
msg.from = self.fsm.peer.peer_id();
let raft_msg = self.fsm.peer.build_raft_messages(self.ctx, vec![msg]);
self.fsm.peer.send_raft_messages(&mut self.ctx, raft_msg);
.
The MsgUnreachable is a local msg in raft-rs. So it should not be sent to other peers. /cc @hicqu

@BusyJay BusyJay added the type/bug Type: Issue - Confirmed a bug label Nov 24, 2021
@BusyJay
Copy link
Member

BusyJay commented Nov 24, 2021

@tier-cap PTAL

@gengliqi gengliqi assigned tier-cap and unassigned gengliqi Dec 2, 2021
@Lily2025
Copy link

Lily2025 commented Dec 6, 2021

/severity Moderate

@hicqu
Copy link
Contributor

hicqu commented Dec 7, 2021

@gengliqi do you have any suggestions for this? Here I want a mechanism that a raft follower can tell its leader something, and then its leader can change the follower's replication state from Replicate to Probe.

@gengliqi
Copy link
Member

gengliqi commented Dec 8, 2021

How about making MsgUnreachable as a remote message?

@BusyJay
Copy link
Member

BusyJay commented Dec 8, 2021

You should check the message and using report_unreachable API instead.

tier-cap added a commit to tier-cap/tikv that referenced this issue Apr 21, 2022
@tier-cap
fix one bug about log append reject msg MsgUnreachable
b8a9226 
close tikv#11371

Signed-off-by: tier-cap <zhengxiaojin@pingcap.com>
@tier-cap tier-cap mentioned this issue Apr 21, 2022
Closed
tier-cap added a commit to tier-cap/raft-rs that referenced this issue Apr 21, 2022
@tier-cap
fix one bug about MsgUnreachable local type
1f58409 
close tikv/tikv#11371
Signed-off-by: tier-cap <zhengxiaojin@pingcap.com>
tier-cap added a commit to tier-cap/raft-rs that referenced this issue Apr 21, 2022
@tier-cap
fix one bug about MsgUnreachable local type
3dc6ec5 
close tikv/tikv#11371
Signed-off-by: tier-cap <zhengxiaojin@pingcap.com>

Signed-off-by: tier-cap <zhengxiaojin@pingcap.com>
@tier-cap tier-cap linked a pull request Apr 21, 2022 that will close this issue
Open
@tier-cap tier-cap linked a pull request Jun 7, 2022 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tier-cap tier-cap

Labels
severity/moderate type/bug Type: Issue - Confirmed a bug
Projects
None yet
Milestone
No milestone
10 participants
@innerr @BusyJay @dbsid @hicqu @Connor1996 @5kbpers @gengliqi @ti-chi-bot @tier-cap @Lily2025