You type a message and hit send. A response appears a few seconds later. What happened in between is more interesting than it looks.
The first step is tokenization. The model doesn't read your message the way you wrote it. It breaks the text into tokens, which are roughly syllable-sized chunks rather than whole words. "Unbelievable" might become three or four tokens. A single punctuation mark is often one token on its own. This matters because the model's costs, speed, and limits are all measured in tokens, not words. Understanding the token economy is useful for anyone building with these systems.
Once tokenized, your message gets loaded into the context window. This is the model's working memory: everything it can see at once, including your current message, the conversation history, any documents you've shared, and the system instructions that shape how it behaves. Context windows have grown dramatically in recent years, but they're still finite. When a conversation exceeds the limit, earlier content gets dropped. The model doesn't know what it can't see.
With the input loaded, the model runs inference: the forward pass through all its layers that produces the output. This is computationally expensive, and the AI runtime is the infrastructure that manages it, handling the hardware, the memory, and the execution environment that makes inference fast enough to be useful.
One optimization worth knowing about is the KV cache. During inference, the model computes key-value representations for every token in the context. Without caching, it would recompute these from scratch for every new token it generates. The KV cache stores those representations so they can be reused, which is a significant part of why inference is as fast as it is.
The whole pipeline runs in a few seconds for most requests. Each step has real constraints on length, cost, and what the model can hold in mind. Understanding them is what separates people who use AI tools effectively from people who keep running into invisible walls.
