◉ ai

Tokens

The basic units AI uses to process text

What is Tokens?

The fundamental units of text that AI models process, roughly ¾ of a word in English, though the ratio varies by language. Every input and output is measured in tokens, which determines both cost and context window usage. A 1,000-word article is approximately 1,300 tokens. Understanding token counting is essential for managing API costs and fitting content within a model's context window.

Take the next useful step

LV’s AI hub

Original context on AI applied to creation, research, and systems.

Creator OS

Useful when AI is part of a content workflow that still needs human judgment.

💡

In plain words

"Think of tokens like syllables — a word might be one token or several, and they're the unit AI counts."

How it works

Key takeaways

Roughly ¾ of a word in English
Determines cost and context capacity
Tokenizers differ between models

▸

Real-world example

'Hello world' is roughly 2 tokens. A typical paragraph is ~150 tokens. Every token costs money and fills your context window, so being concise isn't just good writing — it's good economics.

Related terms

Context Window

Maximum text an AI can process at once

LLM (Large Language Model)

AI models that understand and generate text

Inference

Using a trained AI model to generate outputs

Editorial

Services

Shop