Package throughputbuffer provides a high throughput indefinitely growing io.ReadWriter, like bytes.Buffer, but optimized for minimal copies. It does the minimum amount of copies (1 per read + 1 per write) (or fewer through ReadFrom and WriteTo) and never has to move bytes in the buffer.
Byte slices are shared in a BufferPool. All byte slices are of the same length: the size given to New().
Buffers automatically shrink as data is read from them (they return their data to the pool).
package main
import (
"crypto/rand"
"github.com/Jille/throughputbuffer"
)
func main() {
pool := throughputbuffer.New(64 * 1024 * 1024)
buf1 := pool.Get()
io.CopyN(buf1, rand.Reader, 1024 * 1024 * 1024)
buf2 := pool.Get()
io.Copy(buf2, buf1) // As data is read from buf, the byteslices are freed and reused by buf2.
}