Ask ChatGPT or Claude to read a long article and it doesn't see words the way you do. It sees tokens, and almost everything about how these systems behave, how much of a document they can hold in view, how a subscription gets billed, how much a single request costs, is counted in them, not in words or characters. If you've never had reason to ask what a token in AI actually is, here's the plain version, and three ways knowing it changes how you write and brief content.
What is a token in AI?
A token is the smallest unit of text an AI model actually processes, and it isn't the same thing as a word. OpenAI's own help centre article on the subject puts it directly: a token can be a whole word, part of a word, a single character, or a piece of punctuation, and models split text into tokens before doing anything else with it (OpenAI Help Center, "Understanding and counting tokens"). A short, common word like "the" is usually one token. A longer or rarer word gets broken into several: OpenAI's own example splits "tokenization" into "token" and "ization", two tokens for one word.
Both OpenAI and Anthropic publish the same rough conversion for English text: roughly one token per four characters, or about three-quarters of a word. Anthropic's own pricing documentation states it as a plain rule of thumb: "1 token is approximately 4 characters or 0.75 words in English" (Anthropic, Claude API pricing FAQ). Run that in reverse and a rough estimate follows for free: a thousand words of ordinary English prose comes to roughly 1,300 to 1,400 tokens.
Why AI systems measure everything in tokens, not words
Two separate limits inside an AI system are both set in tokens, and confusing them is easy because they show up as different problems. The first is the context window, the maximum amount of text a model can hold in view at once. We've written before about what happens once a document is longer than that window, or long enough that the model starts favouring the start and the end over the middle: that entire mechanism is measured and capped in tokens, never in words.
The second limit is cost, and it's a separate thing from the context window entirely. Every request to a commercial AI API is billed by counting the tokens going in and the tokens coming back out, and the two are priced differently. Anthropic's current published rate for Claude Haiku 4.5, for example, is $1 per million input tokens and $5 per million output tokens, a five-to-one gap that holds roughly across the model range (Anthropic, Claude API pricing). That asymmetry exists because generating new text is more computationally expensive than reading existing text, which is also why a chatty, verbose AI-written reply costs a business more per response than a terse one, not just more of your reader's patience.
What that looks like in practice
Token pricing sounds abstract until it's applied to something concrete. Anthropic's own documentation gives a worked example: processing 10,000 customer-support conversations, at roughly 3,700 tokens per conversation on Claude Haiku 4.5, comes to about $37 in total (Anthropic, Claude API pricing). The table below shows the same per-million-token rates for three Claude models, current as of this pricing page.
| Model | Input, per million tokens | Output, per million tokens |
|---|---|---|
| Claude Haiku 4.5 | $1 | $5 |
| Claude Sonnet 5 | $2 | $10 |
| Claude Opus 5 | $5 | $25 |
Source: Anthropic's own Claude API pricing page, current at the time of writing; per-model rates change over time, so check the live page before quoting one in a contract.
Every row keeps the same shape: output costs five times what input costs, and the gap between models is the same multiple all the way up the range. That structure is worth knowing if any tool your team uses is billed this way behind the scenes, because a workflow that keeps re-sending the same long document as input, rather than caching or trimming it, is paying that rate every single time.
Does a shorter page use fewer tokens?
Yes, directly and proportionally. A page written in 400 words uses roughly 550 tokens; the same idea padded out to 1,200 words uses roughly 1,600. That matters for the context-window limit covered in our piece on how AI chatbots summarise long content: a page that says the same thing in fewer tokens leaves more of the model's available window for everything else it's trying to read and compare, your page included.
Does token count affect whether an AI assistant cites your page?
Nobody has published direct evidence linking a page's token count to whether an assistant cites it, so we won't claim one. What we can say, flagged plainly as our own reasoning rather than a measured finding, is that an assistant working from a limited budget per answer has more room for a fact stated once, plainly, in a self-contained sentence than for the same fact buried in three padded paragraphs. A shorter, specific statement is cheaper for the model to include, in the same literal sense that it's cheaper for a business paying per token to generate.
What this means for how you write and brief AI content
None of this calls for counting tokens by hand. Three habits follow from the mechanics above:
- State the specific fact once, plainly, rather than building up to it. A padded sentence costs more tokens to say the same thing, whether that cost is context-window space or API spend.
- Keep sections self-contained. A section that depends on three paragraphs of build-up above it can't be lifted out and quoted on its own, however few tokens it uses.
- If your team pays for an AI tool that processes your own content repeatedly, a long, cluttered source document costs more every single time it's re-read, not just once.
Checking what gets read
Knowing how tokens work explains the mechanism. Whether your own content is landing with an AI assistant is a separate, checkable question. Below is the overview dashboard from a demo workspace in our own AI Visibility product, tracking an email marketing brand, Brevo, across two AI assistants: according to that dashboard, the brand appears in 100% of buyer journeys, wins the recommendation 25% of the time, and holds 24% of the share of AI voice against its most-recommended rival, Omnisend.
That's the layer underneath why some brands get named by an AI assistant and others don't: the assistant is reading a fixed, token-limited slice of the web and deciding what's worth repeating back. Understanding AI search visibility starts with knowing how an assistant reads at all, and tokens are the unit that mechanism runs on.
The short version
A token is the actual unit an AI system reads in, roughly four characters or three-quarters of a word in English per OpenAI's and Anthropic's own documentation, and it isn't the same as a word or a character. Two separate limits are both measured in tokens: the context window, which caps how much text a model can hold in view at once, and cost, since commercial APIs bill separately for tokens going in and tokens coming out, output priced several times higher than input. A shorter, more specific page uses fewer tokens either way, which is a mechanical reason, not just a stylistic preference, to say the concrete thing once rather than padding it out.
