Skip to content

This package helps to work with huge amount of data, which cannot be stored in RAM

License

Notifications You must be signed in to change notification settings

ShoshinNikita/go-disk-buffer

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

56 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Go Disk Buffer

Package buffer helps to work with huge amount of data, which cannot be stored in RAM. Instead of keeping all data in RAM buffer.Buffer can store the data on a disk in a temporary file.

Features:

  • buffer.Buffer is compatible with io.Reader and io.Writer interfaces
  • buffer.Buffer can replace bytes.Buffer (except some methods – check Unavailable methods)
  • You can encrypt data on a disk. Just use Buffer.EnableEncryption method

Notes:

  • It is not recommended to use zero value of buffer.Buffer. Use buffer.NewBuffer() or buffer.NewBufferWithMaxMemorySize() instead
  • buffer.Buffer is not thread-safe!
  • buffer.Buffer uses a directory returned by os.TempDir() to store temp files. You can change the directory with Buffer.ChangeTempDir method

Example

With bytes.Buffer

package main

import (
    "bytes"
    "fmt"
)

func main() {
    b := bytes.Buffer{}
    // b := bytes.NewBuffer(nil)
    // b := bytes.NewBufferString("")

    b.Write([]byte("Hello,"))
    b.WriteByte(' ')
    b.Write([]byte("World!"))

    data := b.Next(13)
    fmt.Println(string(data)) // "Hello, World!"
}

With github.com/ShoshinNikita/go-disk-buffer.Buffer

package main

import (
    "fmt"

    buffer "github.com/ShoshinNikita/go-disk-buffer"
)

func main() {
    b := buffer.NewBufferWithMaxMemorySize(7) // store only 7 bytes in RAM
    // b := buffer.NewBuffer(nil)
    // b := buffer.NewBufferString("")

    b.Write([]byte("Hello,"))
    b.WriteByte(' ')
    b.Write([]byte("World!")) // will be stored on a disk in a temporary file

    data := b.Next(13)
    fmt.Println(string(data)) // "Hello, World!"
}

Benchmark

CPU: Intel Core i7-3630QM
RAM: 8 GB
Disk: HDD, 5400 rpm

Buffer_size_is_greater_than_data/bytes.Buffer-8     1000       1591091 ns/op      10043209 B/op     36 allocs/op
Buffer_size_is_greater_than_data/utils.Buffer-8     1000       1346077 ns/op       6901679 B/op     26 allocs/op

Buffer_size_is_equal_to_data/bytes.Buffer-8         1000       1760100 ns/op      10043195 B/op     36 allocs/op
Buffer_size_is_equal_to_data/utils.Buffer-8         2000       1357077 ns/op       7434159 B/op     27 allocs/op

Buffer_size_is_less_than_data/bytes.Buffer-8          50      36522090 ns/op     177848123 B/op     53 allocs/op
Buffer_size_is_less_than_data/utils.Buffer-8          10     110406320 ns/op     112327659 B/op     62 allocs/op

Available methods

Read

  • Read(p []byte) (n int, err error)
  • ReadByte() (byte, error)
  • Next(n int) []byte
  • WriteTo(w io.Writer) (n int64, err error)

Write

  • Write(p []byte) (n int, err error)
  • WriteByte(c byte) error
  • WriteRune(r rune) (n int, err error)
  • WriteString(s string) (n int, err error)
  • ReadFrom(r io.Reader) (n int64, err error)

Other

  • Len() int
  • Cap() int – equal to Len() method
  • Reset()

Unavailable methods

Can be added

  • ReadBytes(delim byte) (line []byte, err error)
  • ReadString(delim byte) (line string, err error)
  • ReadRune() (r rune, size int, err error)help wanted (check Buffer.readRune() method)

Won't be adeed

  • Bytes() []byte

    Reason: go-disk-buffer was created to store a huge amount of data. If your data can fit in RAM, you should use bytes.Buffer

  • String() string

    Reason: see the previous reason

  • Grow(n int)

    Reason: we can allocate the memory only in RAM. It doesn't make sense to allocate space on a disk

  • Truncate(n int)

  • UnreadByte() error

  • UnreadRune() error

About

This package helps to work with huge amount of data, which cannot be stored in RAM

Topics

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages