Skip to content
← Back to Glossary

Tokens

ai-engineering llm

The atomic units a language model processes, produced by splitting text into words, subwords, or punctuation marks before inference.

↑ Contents

Definition

Text is split into tokens before inference — a token can be a word, part of a word, or a punctuation mark. The model does not see characters or words directly; it sees a sequence of token IDs. Token count determines cost and context usage: every piece of text in a session (system prompt, rules, tool definitions, user messages, model responses) consumes tokens from the context window budget.

↑ Contents

Source

↑ Contents