A small library for converting tokenized PHP source code into XML (and potentially other formats)

Parser Building Toolkit for JavaScript

Solves basic Russian NLP tasks, API for lower level Natasha projects

Persian NLP Toolkit

한국어 자연어처리를 위한 파이썬 라이브러리입니다. 단어 추출/ 토크나이저 / 품사판별/ 전처리의 기능을 제공합니다.

Optimised tokenizer/lexer generator! 🐄 Uses /y for performance. Moo.

Self-contained Japanese Morphological Analyzer written in pure Go

🌭 Mustard is a Swift library for tokenizing strings when splitting by whitespace doesn't cut it.

An Integrated Corpus Tool With Multilingual Support for the Study of Language, Literature, and Translation

Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).

专注于可解释的NLP技术 An NLP Toolset With A Focus on Explainable Inference

Open Korean Text Processor - An Open-source Korean Text Processor

The fast scanner generator for Java™ with full Unicode support

Online playground for OpenAPI tokenizers

🌿 NodeJS PHP Parser - extract AST or tokens

Ungreedy subword tokenizer and vocabulary trainer for Python, Go & Javascript

支持中文和拼音的 SQLite fts5 全文搜索扩展 ｜ A SQLite3 fts5 tokenizer which supports Chinese and PinYin

Python port of Moses tokenizer, truecaser and normalizer

Tiny JavaScript tokenizer.

High performance Chinese tokenizer with both GBK and UTF-8 charset support based on MMSEG algorithm developed by ANSI C. Completely based on modular implementation and can be easily embedded in other programs, like: MySQL, PostgreSQL, PHP, etc.