---
published: true
id: tokens
title: Tokens
description: Knowledge captured in João Lopes' learning graph.
tags:
- ai-engineering
- llm
locale: en
resource: https://joaolopes.dev.br/knowledge/concepts/en/tokens/
created: '2026-03-30'
---

# Tokens
The atomic units a language model processes, produced by splitting text into words, subwords, or punctuation marks before inference.

## Definition
Text is split into tokens before inference — a token can be a word, part of a word, or a punctuation mark. The model does not see characters or words directly; it sees a sequence of token IDs. Token count determines cost and context usage: every piece of text in a session (system prompt, rules, tool definitions, user messages, model responses) consumes tokens from the context window budget.

## Source
- [Large language model - Wikipedia](https://en.wikipedia.org/wiki/Large_language_model)
- [Tokenization in NLP - DataCamp](https://www.datacamp.com/blog/what-is-tokenization)

## Related
- [Context Window](/concepts/en/context-window.md)
- [System Prompt](/concepts/en/system-prompt.md)