◉ ai
Tokens
The basic units AI uses to process text
What is Tokens?
The fundamental units of text that AI models process, roughly ¾ of a word in English, though the ratio varies by language. Every input and output is measured in tokens, which determines both cost and context window usage. A 1,000-word article is approximately 1,300 tokens. Understanding token counting is essential for managing API costs and fitting content within a model's context window.
How it works
Key takeaways
-
Roughly ¾ of a word in English
-
Determines cost and context capacity
-
Tokenizers differ between models
▸
Real-world example
'Hello world' is roughly 2 tokens. A typical paragraph is ~150 tokens. Every token costs money and fills your context window, so being concise isn't just good writing, it's good economics.
