tokenizers
huggingface/tokenizers
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
11,158stars
Forks
1,213
Open issues
195
Watchers
11,158
Size
22.4 MB
RustApache License 2.0
bertgptlanguage-modelnatural-language-processingnatural-language-understandingnlptransformers
Created: Nov 1, 2019
Updated: Oct 7, 2026
Last push: Oct 7, 2026