diff --git a/presentation/claude-code-best-practice/index.html b/presentation/claude-code-best-practice/index.html index 521c0b2..3ce3142 100644 --- a/presentation/claude-code-best-practice/index.html +++ b/presentation/claude-code-best-practice/index.html @@ -522,50 +522,34 @@
-
- - -

Tokens in, tokens out

- - -
- Animated diagram combining tokenization and autoregressive generation: the BPE-tokenized prompt feeds into the LLM, which generates the answer token-by-token using the same shared vocabulary. -
- Input and output share the same vocabulary — tokenization shapes what the model even “sees”.
- “Anthropic” becomes “Anth” + “ropic” because that’s how it appears most often in training data. -
-
- -
+

How an LLM tokenizes input

+
+ Animated diagram combining tokenization and autoregressive generation: the BPE-tokenized prompt feeds into the LLM, which generates the answer token-by-token using the same shared vocabulary. +
+ BPE chops words into subword tokens — same color = same word, gray = punctuation. +
+
-
- - -

What the model actually sees

- - -
- Animated diagram showing the 32 integer token IDs the model receives: e.g. 28133 for 'Does', 17554 for ' Chat', 162016 for 'GPT', 97481 for ' Claude'. Generated tokens are also shown as IDs. Vocab size V ≈ 200,000. -
- The model never reads text — it reads a sequence of integers, each one an index into a vocabulary of ~200,000 entries.
- Notice the comma is always ID 11 — the same punctuation mark maps to the same integer, everywhere, every time. -
-
- -
+

What the LLM actually sees: integer token IDs

+
+ Animated diagram showing the 32 integer token IDs the model receives: e.g. 28133 for 'Does', 17554 for ' Chat', 162016 for 'GPT', 97481 for ' Claude'. Generated tokens are also shown as IDs. Vocab size V ≈ 200,000. +
+ Notice the comma is always ID 11 — the same punctuation mark maps to the same integer, everywhere, every time. +
+