tokenizers

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

10,490stars

Forks

1,046

Open issues

143

Watchers

10,490

Size

14.1 MB

RustApache License 2.0

bertgptlanguage-modelnatural-language-processingnatural-language-understandingnlptransformers

Created: Nov 1, 2019

Updated: Feb 27, 2026

Last push: Feb 28, 2026