tokenizers
huggingface/tokenizers
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
10,878stars
Forks
1,141
Open issues
228
Watchers
10,878
Size
19.0 MB
RustApache License 2.0
bertgptlanguage-modelnatural-language-processingnatural-language-understandingnlptransformers
Created: Nov 1, 2019
Updated: Jul 12, 2026
Last push: Jul 12, 2026