tokenizers
huggingface/tokenizers
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
10,626stars
Forks
1,070
Open issues
152
Watchers
10,626
Size
16.8 MB
RustApache License 2.0
bertgptlanguage-modelnatural-language-processingnatural-language-understandingnlptransformers
Created: Nov 1, 2019
Updated: Apr 14, 2026
Last push: Apr 13, 2026