[EPIC] v1 data path performance enhancement

[derekbit]

Is your feature request related to a problem? Please describe (👍 if you like this request)

This epic is used to track improvements of the v1 data path.

• Frontend
  • [POC] Investigate Longhorn with userspace block device (ublk) #5159
• Backend
  • [IMPROVEMENT] investigate v1 volume backend improvement #6590
  • [IMPROVEMENT] Improve the replica rebuilding using different data transmission protocol #5002 (replica rebuild)
• Investigate performance bottleneck
  • [IMPROVEMENT] Investigate performance bottleneck in v1 data path  #8436

Describe the solution you'd like

Describe alternatives you've considered

Additional context

cc @longhorn/dev-data-plane

[innobead]

The primary improvement would be:

• frontend <-> engine: introduce ublk (relying on kernel support, at least 6.1)
• engine/replica <-> replica: introduce a new protocol (like quic) to replace the in-house TCP-based protocol.

The purpose is to make v1 appropriate for running under constrained resource environments like Edge.

[derekbit]

@Kampadais improved the v1 data path based on my previous PoC.
longhorn/longhorn-engine#1067

I think we can work on the improvement in v1.8.0 together and replace the iSCSI frontend with the new one.
WDYT @PhanLe1010 @Kampadais

cc @innobead

[PhanLe1010]

Agree that we can implement it in 1.8.0. Because I am working on v1 performance related topics, I can help to drive this ticket if it is ok Derek

[derekbit]

Agree that we can implement it in 1.8.0. Because I am working on v1 performance related topics, I can help to drive this ticket if it is ok Derek

Sure! ;) BTW, the author of ubdsrv, Ming, is friendly and interested in Longhorn. Feel free to discuss with him if we encounter any issues.

[Kampadais]

I would like to work on it more. I am currently experimenting and investigating why with my multiple connections approach can't achieve ublk's performance.

[PhanLe1010]

Surely @Kampadais . I am happy to follow your lead!

[shuo-wu]

Recently Phan and me tried to make some improvements to the v1 backend data path:

1. Avoid unnecessary memory allocations during revision counter increment (for Write operation): Fix some v1 data path bottlenecks longhorn-engine#1085
2. Increase the connection between engine and replicas: Fix some v1 data path bottlenecks longhorn-engine#1085
3. Cache the disk size so that there is no need to invoke system call fstat for each Read operation: [IMPROVEMENT] Investigate performance bottleneck in v1 data path  #8436 (comment)
4. Use channels rather than mutex lock for revision counter increment
5. Increase the revision counter and write actual data simultaneously

Notice that only the first two ideas work as expected. And the remaining 3 ideas cannot get a better result in the fio test, regardless that the CPU profiling shows a CPU usage percentage decrement for the R/W function. We suspect that iSCSI tgt may be the main bottleneck. It may waste all CPU resources saved by the last 3 ideas, which finally leads to 0 improvements in the fio test.

If we plan to introduce ublk or other kinds of frontend, which has better improvement than iSCSI tgt. We can re-do the test for the last 3 ideas. For example:

               ublk + idea 1,2            ublk + idea 1,2,3            ublk + idea 1,2,3,4,5
IOPS
CPU

cc @PhanLe1010

[DamiaSan]

Maybe the bottleneck can be the revision counter itself? I mean the writing of the revision counter on the disk.