Models process text as tokens rather than words or characters. Common words are typically single tokens while rare words, names, and numbers split into several. Token counts determine context window usage and, in most pricing models, cost.
A rough conversion is that a page of text runs several hundred tokens.
Alternative Names:
Tokens, Text Token
Why it Matters?
Token counts are the practical measure of what fits and what costs, which matters when evaluating claims about document capacity. A vendor stating a large context window is describing tokens, not pages, and legal documents with citations, tables, and Bates numbers tokenize less efficiently than ordinary prose. Estimating a matter's token volume gives a realistic picture of both feasibility and expense.
Frequently Confused with
Related terms
Frequently asked questions
How many tokens is a page of text?
Why do tokens matter to buyers?
