tokenizers
huggingface/tokenizers
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
10,997stars
Forks
1,177
Open issues
238
Watchers
10,997
Size
22.4 MB
RustApache License 2.0
bertgptlanguage-modelnatural-language-processingnatural-language-understandingnlptransformers
Created: Nov 1, 2019
Updated: Aug 26, 2026
Last push: Aug 26, 2026