A Turkish Tokenizer

The goal of this project is to develop a machine learning model that can classify movie reviews as positive or negative based on the sentiment expressed in the text.

An OCaml-based lexical analyzer that identifies and classifies tokens such as identifiers, operators, punctuation symbols, integer literals, and keywords. The project involves tokenizing input text, categorizing tokens, and printing them with their respective categories. Key functions include tokenize, is_alnum, is_punctuation, and print_tokens.

Implementation of the LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens Paper

💫 Industrial-strength Natural Language Processing (NLP) in Python

Tools and resources for the computational processing of Nheengatu (Modern Tupi)

Simple multilingual lemmatizer for Python, especially useful for speed and efficiency

Text Tokenizer Playground ( Transformers.js ) SDK in Hugginface.

Repo Related to Natural Language Processing and Social Media Analytics.

Data Pre-processing Application/UI is a simple UI which can automate repitive tasks, while ensuring consistency and efficiency in NLP data preprocessing.

A Python library for interacting with TI-(e)z80 (82/83/84 series) calculator files

retro style tokenization for language models

Slides, exercises, and exams for my course "Natural Language Processing" (École Pour l'Informatique et les Techniques Avancées, 2024)

🛥 Vaporetto: Very accelerated pointwise prediction based tokenizer

🎤 vibrato: Viterbi-based accelerated tokenizer

Sudachi in Rust 🦀 and new generation of SudachiPy

Basis Theory Developer Documentation

A modular resource tokenization service.

(py package) tokenizer based on BPE algorithm for the LLMs (supports the regex pattern and special tokens)

Public code samples and resources for the Thales CipherTrust Application Protection products of the CipherTrust Data Security Platform