Customizable tiktoken-compatible and SentencePiece BPE tokenization for language models.

Latest Versions

3 versions โ†’
Version โ–ผVulnerabilitiesUsagesDate
0.3.x
0.3.1
18
Oct 09, 2026
0.2.x
0.2.0
18
Sep 16, 2026
0.1.x
0.1.0
2
May 06, 2026
3 versions โ†’