Screenshot of platform.openai.com/tokenizer showing the sentence "Does ChatGPT,
Claude, Anthropic, Llama, Mistral, Gemini, and Perplexity all use Byte-Pair
Encoding (BPE)?" tokenizing to 32 tokens / 105 characters. Visible tabs:
GPT-5.x & O1/3, GPT-4 & GPT-3.5 (legacy), GPT-3 (legacy) — illustrates that
different model generations use different tokenizers.
Co-Authored-By: Claude <noreply@anthropic.com>
Shayan Rais
