add tokens.jpg — OpenAI tokenizer screenshot for tokenization slide

Browse Source

Screenshot of platform.openai.com/tokenizer showing the sentence "Does ChatGPT,
Claude, Anthropic, Llama, Mistral, Gemini, and Perplexity all use Byte-Pair
Encoding (BPE)?" tokenizing to 32 tokens / 105 characters. Visible tabs:
GPT-5.x & O1/3, GPT-4 & GPT-3.5 (legacy), GPT-3 (legacy) — illustrates that
different model generations use different tokenizers.

Co-Authored-By: Claude <noreply@anthropic.com>

...

This commit is contained in:

Shayan Rais

2026-05-07 11:45:09 +05:00

parent e53739367a

commit faa82716b0

1 changed files with 0 additions and 0 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

presentation/assets/llm/tokens.jpg

BIN

Binary file not shown.

Side by Side After Width:  |  Height:  |  Size: 351 KiB