New Segmenter
Introduces a new Segmenter
type to tokenize []byte
. Analogous to the existing Scanner
, but does not require a io.Reader
. Segmenter
is maybe 10% faster, assuming you are operating on an existing []byte
.
Also adds a SegmentAll([]byte)
convenience method, if you’d like to do this as a one-liner and are not too concerned about allocations.