Skip to content

New Segmenter

Compare
Choose a tag to compare
@clipperhouse clipperhouse released this 18 May 17:07
· 32 commits to master since this release

Introduces a new Segmenter type to tokenize []byte. Analogous to the existing Scanner, but does not require a io.Reader. Segmenter is maybe 10% faster, assuming you are operating on an existing []byte.

Also adds a SegmentAll([]byte) convenience method, if you’d like to do this as a one-liner and are not too concerned about allocations.