Vocabulary
ai-engineering llm tokenization
The fixed set of all tokens a language model can read and produce.
↑ ContentsDefinition
A model’s vocabulary is built during training. A tokenizer analyzes large amounts of text and determines which chunks appear often enough to become their own token. Common words like “the” are a single token. Rare or compound words get split into pieces. Code syntax, punctuation, and whitespace all get entries. The vocabulary is fixed after training. The model can only produce tokens that exist in this set. When the model generates a response, it assigns a probability to every token in the vocabulary and picks one.
↑ Contents