Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken

Image from Unsplash

TokenDagger is a drop-in replacement for OpenAI’s Tiktoken (the tokenizer behind Llama 3, Mistral, GPT-3.*, etc.). It’s written in C++ 17 with thin Python bindings, keeps the exact same BPE vocab/special-token rules, and focuses on raw speed.I’m...

This snippet was provided by a publicly available feed. Read the full article at github.com