LLMs don’t see letters or words the way you do — they see tokens, chunks of text produced by a tokenizer that sit somewhere between individual characters and whole words. Understanding tokenization explains a surprising amount of otherwise-confusing model behavior.
Seeing Tokenization in Action
import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")
text = "Tokenization isn't always intuitive!"
tokens = enc.encode(text)
print(tokens)
# [11346, 2065, 6602, 4536, 956, 2744, 46940, 0]
for t in tokens:
print(repr(enc.decode([t])))
# 'Token' 'ization' " isn't" ' always' ' intuit' 'ive' '!'
Why Tokens Aren’t Words
Tokenizers use a technique called byte-pair encoding, which learns common subword chunks from training data. Common English words often become a single token; rare words, unusual names, and non-English text frequently split into multiple smaller tokens — which is exactly why prompts in some languages cost more tokens (and money) than equivalent English text.
Why This Explains Weird Model Behavior
- Struggling with letter-counting — a model asked “how many letters are in ‘strawberry’” is reasoning over tokens, not individual characters, which is why this specific class of question trips models up more than you’d expect.
- Pricing is per-token, not per-word — roughly 100 tokens ≈ 75 English words, but this ratio varies significantly by language and content type (code tends to tokenize less efficiently than prose).
- Context window limits are token limits — a “128K context window” means 128,000 tokens, not characters or words, so actual text capacity varies with content.
Estimating Cost Before You Send a Request
import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")
prompt = "Summarize the following document: " + "..." * 500
token_count = len(enc.encode(prompt))
cost_per_1k_input = 0.0025 # example pricing, check current rates
estimated_cost = (token_count / 1000) * cost_per_1k_input
print(f"{token_count} tokens, approx ${estimated_cost:.4f}")