Biptoken
JustVugg/Biptoken
Fast BPE tokenizer with guaranteed perfect text reconstruction. 2x faster than tiktoken at encoding/decoding while preserving all whitespace and formatting.
2stars
Forks
0
Open issues
0
Watchers
2
Size
0.0 MB
PythonMIT License
bpe-tokenizerfast-tokenizerllm-toolingnlpopen-sourceperfect-reconstructionpythonpython-librarytext-tokenizationtokenizerwhitespace-preservation
Created: Jun 13, 2025
Updated: Jul 19, 2026
Last push: Jun 13, 2025