tokenizers
huggingface/tokenizers
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
10,772stars
Forks
1,106
Open issues
185
Watchers
10,772
Size
17.5 MB
RustApache License 2.0
bertgptlanguage-modelnatural-language-processingnatural-language-understandingnlptransformers
Created: Nov 1, 2019
Updated: May 27, 2026
Last push: May 26, 2026