Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement the splice syscall #3322

Open
cohosh opened this issue Apr 3, 2024 · 1 comment
Open

Implement the splice syscall #3322

cohosh opened this issue Apr 3, 2024 · 1 comment
Labels
Status: Help Wanted Well suited for community contributions Type: Enhancement New functionality or improved design

Comments

@cohosh
Copy link
Contributor

cohosh commented Apr 3, 2024

Describe the issue
splice is a system call that allows for the efficient movement of data between file descriptors if one of the descriptors is a pipe.

The Go networking library switched to using splice for the read/write methods of net.TCPConn to improve performance. This means that read/write calls between net.TCPConn's in Shadow fail with the following Go logs:

2000/01/01 00:00:10 error copying to ORPort writeto tcp 0.0.0.0:8081->11.0.0.1:29902: readfrom tcp 127.0.0.1:20027->127.0.0.1:8080: splice: function not implemented

To Reproduce
I've created a minimal example of a simple TCP proxy in Go: https://github.com/cohosh/go-tcp-shadow-minimal

Operating System (please complete the following information):

  • OS and version: Debian GNU/Linux trixie/sid
  • Kernel version: Linux 6.7.9-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.7.9-2 (2024-03-13) x86_64 GNU/Linux

Shadow (please complete the following information):

  • Version and build information: post the output of shadow --show-build-info
Shadow 3.1.0 — v3.1.0-205-g1ca8075dd 2024-04-01--16:54:23
GLib 2.78.4
Built on 2024-04-02--15:00:39
Built from git branch main
Shadow was built with PROFILE=release, OPT_LEVEL=3, DEBUG=false, RUSTFLAGS="-C force-frame-pointers=y", CFLAGS="-std=gnu11 -O3 -ggdb -fno-omit-frame-pointer -Wreturn-type -Wswitch -DNDEBUG"
For more information, visit https://shadow.github.io or https://github.com/shadow
  • Which processes you are trying to run inside the Shadow simulation:
    minimal tcp proxy, tgen
@cohosh cohosh added the Type: Bug Error or flaw producing unexpected results label Apr 3, 2024
@stevenengler
Copy link
Contributor

stevenengler commented Apr 6, 2024

A couple of notes about implementing this while I'm thinking about it:

  1. splice has a few similar syscalls like tee, sendfile, and copy_file_range. When designing the interface for splice, it might be worth considering these other syscalls to see if it's possible to support more than just splice with the same interface.
  2. Currently to get bytes in or out of a file, you would use the file's readv and writev methods. But these are meant for copying bytes to/from the plugin's memory so they take an iovec of plugin pointers, which isn't useful for moving bytes between two files within shadow. It's probably best to add new methods to the file (maybe named something like splice_in/splice_out), and implement these for each file type (RegularFile, TcpSocket, EventFd, etc). The tricky part is that you need to move data from one file to another without dropping any bytes (the move must be infallible). If you remove bytes from one file you must add it to the other file, which means the receiving file must have enough space for it. Once you remove data from one file, you can't re-add it back to the original file if the receiving file doesn't have enough space. This could be especially tricky when copying bytes from a pipe to a regular file since we don't know if the Linux kernel "write-to-file" will succeed or not. It might be useful to have some sort of pattern where a file gives you a handle to some bytes which allows you to copy them, and then only once the copy is successful, you "commit" the changes on the handle causing the original file to remove those bytes.

@stevenengler stevenengler added Type: Enhancement New functionality or improved design Status: Help Wanted Well suited for community contributions and removed Type: Bug Error or flaw producing unexpected results labels Apr 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Status: Help Wanted Well suited for community contributions Type: Enhancement New functionality or improved design
Projects
None yet
Development

No branches or pull requests

2 participants