New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add QueryBinary, an alloc-free way to read all rows into a buffer #58
base: main
Are you sure you want to change the base?
Conversation
@@ -401,6 +403,24 @@ func (stmt *Stmt) ColumnDeclType(col int) string { | |||
return res | |||
} | |||
|
|||
func (stmt *Stmt) StepAllBinary(dstBuf []byte) (n int, err error) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
:ohno:
you could, now you're going this route... make this AppendAllBinary(dst []byte) ([]byte, int, error)
and it could loop over the C function, with it returning every time a row doesn't fit for AppendAllBinary to grow dst, then pass it back in to read more rows.
that way you get a clean Go api, an amortized number of C calls over the growing size of dst, and only a single pass through the query
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Look at the caller of this. That's where clean Go API will go. This is the lower level wrapper.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was suggesting that if you move that logic in here, instead of calling ResetAndClear every time you need to grow the array, you just have to start encoding the current row again. That way you only have to step through a query once. (Which is nice if someone throws in a huge complex join that requires sqlite go off and do some huge amount of pre-work.)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't want to attempt to bail out of step iteration in C, go back to Go, and then try to get back into C where I left off. Just not worth the complexity.
All or nothing.
Callers can arrange good enough buffer sizes in pools.
7cd933b
to
a6c43b3
Compare
6af984e
to
d10e7f3
Compare
WIP; goal is alloc-free reads of a query into a Go-provided buffer. Go code can then parse the simple binary format and alloc if needed (doing its own cache lookups, including alloc-free m[string([]byte)] lookups, and returning existing Views if data is unmodified) Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
d10e7f3
to
1c6bbb2
Compare
WIP; goal is alloc-free reads of a query into a Go-provided buffer. Go code can then parse the simple binary format and alloc if needed (doing its own cache lookups, including alloc-free m[string([]byte)] lookups, and returning existing Views if data is unmodified)
(Counterproposal to #43)