A tokenizer is the component that splits text into tokens, the units a language model processes, and maps each token to an integer ID. Modern LLMs typically use subword methods such as byte pair encoding (BPE) or SentencePiece, which break words into smaller pieces so that any text can be represented with a fixed vocabulary.
Vocabulary size and segmentation rules differ between models. The same text can yield a different number of tokens depending on the tokenizer, which affects context window usage and API cost.