The atomic units a language model processes, produced by splitting text into words, subwords, or punctuation marks before inference.
↑ ContentsDefinition
Text is split into tokens before inference — a token can be a word, part of a word, or a punctuation mark. The model does not see characters or words directly; it sees a sequence of token IDs. Token count determines cost and context usage: every piece of text in a session (system prompt, rules, tool definitions, user messages, model responses) consumes tokens from the context window budget.
↑ Contents