Short-Term Memory in large language models is the dynamic, temporary workspace where all active reasoning, context processing, and generation occur during a single inference session. Unlike the static knowledge baked into a model's weights during training, this memory is entirely volatile. It resets completely between sessions, holds only the specific tokens loaded into it for the current task, and disappears the moment the computational process ends. In modern transformer architectures, this workspace is physically implemented as the context window and the associated Key-Value (KV) cache.
To understand how artificial intelligence processes information, it is helpful to look at how cognitive science models human working memory. In 1974, psychologists Alan Baddeley and Graham Hitch proposed a multicomponent model of working memory that remains the dominant framework in cognitive science today (Baddeley and Hitch, 1974). They argued that short-term memory is not a single, passive storage bin, but an active system comprising a central executive controller and several specialized storage buffers.
This cognitive model maps surprisingly well onto the architecture of modern large language models (LLMs). The tokens loaded into the context window act as the temporary storage buffer, holding the sequence of information to be processed. The multi-head attention mechanism functions as the central executive, selectively allocating computational focus across those tokens to determine which pieces of information are most relevant at any given step. Just as human working memory is strictly capacity-limited by design, the LLM's active workspace is constrained by the physical limits of GPU memory and the mathematical properties of attention.
This temporary quality is not a defect. It is a fundamental architectural feature. Every session starts with a blank slate, ensuring that the model only reasons over the information deliberately provided to it. When you paste a document into a chat interface, you are not teaching the model new facts; you are temporarily loading external data into its short-term memory so the attention mechanism can process it. The model does not "know" that document in any permanent sense. It can only reason about it while it sits in the active context, and once the session is closed, that knowledge is gone.
The Physical Implementation of the Workspace
The context window is often discussed as if it were a simple software variable (a text box that holds a certain number of words). In reality, it is a hardware-allocated block of GPU memory that stores the mathematical representations of every token the model is currently considering. This physical grounding is crucial for understanding why short-term memory behaves the way it does.
During inference, a transformer model must calculate how every token in the sequence relates to every other token. This process relies on computing key (K) and value (V) projections for each attention head. If the model had to recalculate these projections from scratch for every single new token it generates, the computational cost would scale quadratically, making real-time generation impossible.
To solve this, engineers use the KV cache. The KV cache stores these intermediate projections across generation steps, allowing the model to generate text in linear time (Raschka, 2025). This cache is the physical embodiment of the model's short-term memory. Holding a thought in an LLM's working memory requires maintaining its K and V projections in the GPU's high-bandwidth memory (HBM).
This physical reality imposes severe constraints. A single 128,000-token context window for a 70-billion parameter model requires approximately 40 gigabytes of HBM just for the KV cache, before the model weights are even loaded (Introl, 2025). Because attention scales quadratically, 10,000 tokens create 100 million pairwise relationships, while one million tokens create one trillion. The sheer volume of data required to maintain a large short-term memory is staggering, and it represents one of the primary bottlenecks in modern AI infrastructure.
To manage this immense memory pressure, researchers have developed several architectural optimizations. Grouped-Query Attention reduces the number of KV heads required, significantly lowering the memory footprint without sacrificing too much accuracy. PagedAttention, popularized by the vLLM framework, applies operating system paging concepts to the KV cache, reducing fragmentation and improving throughput.
Some models, like Mistral, employ sliding window attention, which restricts the model's focus to a rolling window of recent tokens. This acts as a form of architectural forgetting, intentionally dropping older short-term memories to maintain processing speed and reduce memory consumption (Jiang et al., 2024). By limiting the attention span, these models can process infinitely long sequences in theory, but their functional short-term memory is strictly bounded by the size of the sliding window.
The RAM Analogy and Architectural Failures
Former Tesla AI director Andrej Karpathy popularized a highly effective analogy for understanding this system: the LLM is the CPU, the context window is the RAM, and the model weights are the ROM. The weights are burned-in during training and cannot be directly addressed or altered during inference. Everything outside the context window, such as vector databases or conversation logs, acts as disk storage. It is vast and passive, requiring an explicit retrieval operation to load it into RAM before the CPU can process it.
This framing clarifies why many AI agents fail in production. Developers often treat the context window as if it were a database, appending every tool result, user preference, and conversation turn into the prompt until it hits the token limit. But the context window is volatile RAM, not persistent storage (Mem0, 2026).
When a model is forced to hold too much information in its short-term memory, its performance degrades predictably. This degradation is not a binary failure where the model simply crashes; it is a gradual loss of attention and reasoning capability. A model might perfectly follow a complex formatting constraint for the first five turns of a conversation, only to completely ignore it on the sixth turn. The model hasn't changed, and the prompt hasn't changed. What has happened is that the attention weight on that specific constraint has dropped below the threshold required to enforce it, diluted by the sheer volume of other information crowding the short-term memory.
Serial Position Effects and Attention Degradation
The most significant limitation of short-term memory in LLMs is how it allocates attention across long sequences. Just as human memory is subject to cognitive biases, transformer models exhibit strong serial position effects. They do not process all information in their context window equally.
In cognitive psychology, the primacy effect describes the tendency to remember information presented at the beginning of a sequence, while the recency effect describes the tendency to remember information at the end. Research has consistently shown that LLMs suffer from identical biases (Vosoughi et al., 2024).
This phenomenon was rigorously quantified in a landmark 2023 study by researchers at Stanford University and UC Berkeley, who identified the "Lost in the Middle" problem (Liu et al., 2023). They found that models exhibit a distinct U-shaped accuracy curve when retrieving information from their short-term memory. When relevant information is placed at the very beginning of the context window, accuracy is high. When it is placed at the very end, accuracy is also high. But when critical information is buried in the middle of a long prompt, retrieval accuracy drops precipitously, often by more than 30 percentage points.
This degradation is not a bug in the training data; it is an artifact of the transformer architecture itself. The causal attention mask used in generative models inherently biases attention toward earlier tokens as the network gets deeper. Simultaneously, relative positional encodings, such as Rotary Position Embedding (RoPE), tend to favor tokens that are closer together, supporting a strong recency bias (Intuition Labs, 2026). The net result is that the model focuses intensely on the edges of its short-term memory while losing track of the center.
Furthermore, this degradation worsens as the context window expands. While model providers now advertise context windows of one million or even ten million tokens, independent testing reveals that performance drops continuously as input length increases. A 2025 study by Chroma evaluated 18 frontier models and found that every single one exhibited "context rot," meaning a steady decline in reasoning capability as the short-term memory load grew, particularly on tasks requiring semantic understanding rather than simple lexical matching (Chroma, 2025).
This means that simply buying a larger context window does not solve the short-term memory problem. It merely delays the inevitable degradation. A model with a one-million-token context window might be able to ingest an entire codebase, but its ability to reason about the intricate connections between files buried in the middle of that codebase will be severely compromised compared to its ability to reason about the files at the very beginning or the very end.
The Illusion of Infinite Context
The marketing around large language models often focuses heavily on the size of the context window, presenting it as a proxy for the model's overall capability. A model with a 128K context window is assumed to be inherently superior to one with a 32K window, simply because it can "remember" more. However, this perspective fundamentally misunderstands the nature of short-term memory in these systems.
The illusion of infinite context suggests that as long as information fits within the token limit, the model will process it perfectly. This is demonstrably false. The reality is that short-term memory in LLMs is a highly lossy environment. Every additional token added to the context window dilutes the attention available for all other tokens.
Consider a scenario where an AI agent is tasked with summarizing a long legal contract. If the entire contract is loaded into the context window, the model must distribute its attention across tens of thousands of tokens. The clauses at the beginning and end will likely be summarized accurately, but crucial stipulations buried in the middle may be entirely ignored or hallucinated. The model's short-term memory is simply overwhelmed by the volume of information competing for its limited attention span.
This limitation is particularly problematic for tasks that require complex reasoning over multiple pieces of information scattered throughout a document. In a simple retrieval task (like finding a specific name in a list), the model might succeed even if the name is in the middle of the context window, because the lexical match provides a strong signal. But in a reasoning task (like determining whether a specific clause contradicts another clause earlier in the document), the model's performance will degrade rapidly as the distance between the two clauses increases.
The failure to recognize this limitation leads to poorly designed AI systems. Developers often try to solve complex problems by simply stuffing more information into the prompt, hoping that the model will somehow figure it out. This approach is akin to trying to solve a complex math problem by holding all the intermediate steps in your head at once; eventually, your working memory will fail, and you will make a mistake.
Engineering the Active Workspace
Understanding the physics and failure modes of short-term memory fundamentally changes how developers must approach prompt engineering and agent design. Because the context window is volatile, expensive, and prone to degradation, it must be managed aggressively.
The most immediate practical application of this knowledge is prompt structuring. Because of the primacy and recency effects, the placement of instructions within the context window is critical. The most important constraints and the final task instructions should always be placed at the very end of the prompt, leveraging the model's recency bias to ensure compliance. Foundational context and system personas should be placed at the very beginning, utilizing the primacy effect. The middle of the prompt should be reserved for the least critical information, such as few-shot examples or supplementary reference data.
More broadly, developers must separate their memory architectures into distinct layers. Working memory should only hold the information strictly necessary to complete the current reasoning step. Everything else (historical conversation logs, user preferences, and broad semantic knowledge) must be offloaded to persistent, non-parametric storage.
When an agent needs historical context, it should not load its entire past into the context window. Instead, it should use a retrieval system to pull only the most relevant facts, inject them into the short-term memory for the duration of the specific task, and then evict them when the task is complete. This approach, treating the context window as a tightly controlled computational workspace rather than a sprawling database, is the only way to build reliable, long-running AI systems that do not collapse under the weight of their own memories.
This requires a shift in mindset from "context stuffing" to "context curation." Instead of asking, "How much information can I fit into the prompt?" developers should ask, "What is the absolute minimum amount of information the model needs to complete this specific step?" By ruthlessly pruning the context window and treating short-term memory as a scarce and valuable resource, developers can build AI agents that are faster, more reliable, and less prone to the catastrophic failures that plague poorly designed systems.
The future of AI memory is not just about building bigger context windows; it is about building smarter systems for managing the information within those windows. As models become more capable and tasks become more complex, the ability to effectively engineer the active workspace will become one of the most critical skills in the field of artificial intelligence.
