Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

raftstore: accumulated read index requests may result in a large number of re-propose actions, increasing pressure on the store loop #16960

Open
cfzjywxk opened this issue May 8, 2024 · 8 comments
Open

raftstore: accumulated read index requests may result in a large number of re-propose actions, increasing pressure on the store loop #16960

cfzjywxk opened this issue May 8, 2024 · 8 comments
Labels
type/enhancement Type: Issue - Enhancement

Comments

@cfzjywxk
Copy link
Collaborator

cfzjywxk commented May 8, 2024 •
edited

Development Task

After #8926 and #9721, an addition_request is introduced to indicate whether the memory lock check is done for the follower or replica read requests.

When a ReadIndexResp is recived and apply_reads is called, the read index request would be re-proposed if lock check is not finished.

Consider a situation, when the number of follower read requests spike on a tikv node, and the read index response is not processed and returned in time from the leader peer, for example the leader peer is busy or there is something with the raft connections, these requests would be pending in the ReadIndexQueue.

When the ReadIndexResp is received after some time, MsgReadIndex message would be re-sent for each pending request to the leader peer, resulting spike of this type of raft messages, increasing the pressure on the raft store loop.
The logs are like

 "[2024/03/06 09:51:57.450 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.449 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.449 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.449 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.449 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.449 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.449 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.449 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.449 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.449 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.449 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.449 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.449 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.449 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.449 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.449 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.449 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.449 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.449 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.449 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.449 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.449 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.449 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.449 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.449 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.449 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.449 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.449 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.449 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.449 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.449 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.449 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.449 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.449 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.449 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.449 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.449 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.449 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.449 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.449 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.449 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.449 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.449 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.449 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.449 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.449 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.449 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.449 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.449 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.449 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.449 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.449 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.449 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.449 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.449 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.449 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.448 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.448 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.448 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.448 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.448 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.448 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.448 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.448 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.448 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.448 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.448 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.448 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.448 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.448 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.448 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.448 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.448 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.448 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.448 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.448 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.448 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.448 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.448 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.448 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.448 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.448 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.448 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.448 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.448 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.448 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.448 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.448 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.448 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.448 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.448 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.448 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.448 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.448 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.448 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.448 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.448 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.448 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.448 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.448 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.448 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.448 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.448 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.448 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.448 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.448 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.448 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.448 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.448 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.448 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.447 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.447 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.447 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.447 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.447 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.447 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.447 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.447 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.447 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.447 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.447 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.447 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.447 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.447 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.447 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.447 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.447 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.447 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.447 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.447 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.447 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.447 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.447 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.447 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.447 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.447 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.447 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.447 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.447 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.447 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.447 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.447 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.447 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.447 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.447 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.447 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.447 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.447 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.447 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.447 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.447 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.447 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.447 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.447 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.447 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.447 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.447 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.447 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.447 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.447 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.447 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.447 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.446 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.445 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]",
  "[2024/03/06 09:51:57.445 +00:00] [INFO] [peer.rs:3401] [\"re-propose read index request because the response is lost\"] [peer_id=1210685518] [region_id=1210685515]"

A lot of re-propose requests are sent with a few miliseconds.

It's better to find a strategy to avoid such spikes to avoid tail latency caused by pressure on the raft store loop.
@cfzjywxk cfzjywxk added the type/enhancement Type: Issue - Enhancement label May 8, 2024
@mittalrishabh
Copy link

why do we want to send it for each request and we batch them instead.

@cfzjywxk
Copy link
Collaborator Author

why do we want to send it for each request and we batch them instead.

@mittalrishabh
After introcing the async commit protocol, it's needed for the read index requests to check the memory locks. For different read requests the key ranges could be different, the batch mode could not be used then.

@mittalrishabh
Copy link

  1. Can we batch the read index if async commit is disabled ?
  2. Can we send memory locks along with the writes through raft to the follower.
    I strongly think that we need batching. We can not send read index for each request to the leader.

@cfzjywxk
Copy link
Collaborator Author

  1. Can we batch the read index if async commit is disabled ?
  2. Can we send memory locks along with the writes through raft to the follower.
    I strongly think that we need batching. We can not send read index for each request to the leader.
  1. I think so. But this could only work when async-commit is totally disabled on the whole cluster, otherwise there could be risk the linearzibility would be broken.
  2. It's checking memory locks on the leader peer when receiving ReadIndexRequest to make sure the snapshot isolation is guaranteed, but not sending memory locks. There is improvement space not to re-propose for each request.

@mittalrishabh
Copy link

mittalrishabh commented May 11, 2024 •
edited

It's checking memory locks on the leader peer when receiving ReadIndexRequest to make sure the snapshot isolation >>is guaranteed, but not sending memory locks.

Can't we maintain another copy of memory locks in follower so that read index doesn't need to check for memory locks.

@cfzjywxk
Copy link
Collaborator Author

It will introduce greate impact on concurrency control complexity and write performance.
There are some dicussions about it in the https://tikv.github.io/sig-transaction/design/async-commit/parallel-commit-known-issues-and-solutions.html document.

@mittalrishabh
Copy link

Thanks. I will go through the above link. Does it make sense to send batch read index to reduce the number of RPC calls.

@cfzjywxk
Copy link
Collaborator Author

Thanks. I will go through the above link. Does it make sense to send batch read index to reduce the number of RPC calls.

It could be an improvement I think, like re-propose the pending requests whose responses are regarded as lost within one message or RPC call.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/enhancement Type: Issue - Enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@cfzjywxk @mittalrishabh