Memory Systems (LLMs) refer to the complete architectural stack of storage mechanisms that allow an artificial intelligence to retain, organize, and recall information across time. While a single conversation relies on working memory, a full memory system orchestrates everything from the knowledge baked into the model's weights during training to the external databases it queries during operation. It is the infrastructure that transforms a stateless text predictor into a persistent, context-aware agent.
The engineering of these systems is currently the most active frontier in AI development. To understand how they work, we have to look at memory through two different lenses: how the data is physically stored (the engineering reality), and what cognitive function it serves (the design framework).
The Engineering Reality of Parametric vs. Non-Parametric Storage
In 2020, researchers at Facebook AI (now Meta) published a paper on Retrieval-Augmented Generation (Lewis et al., 2020) that formalized a crucial distinction for the field. They divided AI memory into two physical categories: parametric and non-parametric. This distinction remains the foundation of all modern memory systems.
Parametric memory is the knowledge encoded directly into the model's neural network weights during its initial training. When you ask a model for the capital of France and it answers "Paris" without looking anything up, it is relying on parametric memory. This storage is incredibly fast because it requires no external retrieval step. However, it is also opaque—you cannot easily inspect what the model "knows"—and it is static. To update parametric memory, you have to retrain or fine-tune the model, which is computationally expensive.
Non-parametric memory, by contrast, is knowledge stored outside the model in external databases. This could be a vector database holding semantic embeddings, a graph database mapping relationships, or a simple key-value store. Non-parametric memory is transparent, auditable, and easily updatable. If a fact changes, you simply update the database record. The tradeoff is speed and complexity: the system must actively retrieve this information and inject it into the model's context window before it can be used.
The Four Types of Memory of the Cognitive Framework
While the parametric/non-parametric divide explains how data is physically stored on disk or in silicon, it does not explain how an agent actually uses that data to make decisions. For that, the field relies on the CoALA framework (Sumers et al., 2023), which maps human cognitive memory types onto AI architecture. This framework provides the vocabulary engineers use to design agentic behavior, shifting the focus from database queries to cognitive functions.
Working memory is the active context window. It holds the system prompt, the current conversation history, and any information recently retrieved from external storage. It is the only memory the model can directly reason over. Everything else must be brought here to be useful. Because working memory is strictly limited by the model's maximum token count, it acts as the primary bottleneck for all cognitive operations. In human terms, this is the information you are actively holding in your mind right now to understand this sentence.
Episodic memory is the record of specific past events and interactions. It is the log of what happened, when it happened, and what the outcome was. As researchers have noted, true episodic memory requires binding an event to its temporal and causal context (Pink et al., 2025). It allows an agent to say, "The last time the user asked for this report, they corrected my formatting." Without episodic memory, an agent cannot learn from its own mistakes or maintain continuity across multiple sessions. In production systems, episodic memory is typically stored non-parametrically in a database and retrieved based on semantic similarity to the current task. The challenge with episodic memory is volume: an agent interacting with a user every day will quickly generate millions of episodic logs, requiring aggressive compression and eviction strategies.
Semantic memory is the repository of generalized facts and world knowledge. When episodic memories are abstracted away from their specific time and place, they become semantic knowledge. For example, an episodic memory might be "On Tuesday, the user told me they prefer Python over JavaScript." The consolidated semantic memory would simply be "User prefers Python." Semantic memory can be stored parametrically (baked into the weights during training) or non-parametrically (stored in a vector database for RAG). The transition from episodic to semantic memory—a process called consolidation—is currently one of the most difficult engineering challenges in AI memory design.
Procedural memory contains the rules, skills, and behavioral instructions the agent follows. This includes the system prompt instructions, the definitions of the tools the agent is allowed to use, and the underlying logic of how to execute a multi-step plan. In human cognition, procedural memory is "muscle memory"—knowing how to do something rather than what something is. In AI systems, procedural memory is often split: the core reasoning capabilities are parametric, while specific tool definitions and operational rules are injected into working memory via the system prompt. When an agent learns a new skill, it is updating its procedural memory.
The Context Window Bottleneck
The central engineering challenge of any memory system is the composition problem. You can have petabytes of non-parametric memory stored in external databases, but the model can only process what fits inside its working memory—the context window. This creates a fundamental tension: the agent has access to near-infinite storage, but strictly finite attention.
Every piece of retrieved episodic history, every semantic fact, and every procedural rule must compete for space in this finite bottleneck. This is why memory systems are not just storage systems; they are retrieval and ranking systems. The system must constantly evaluate the current user input against the vast external memory banks to decide what is most relevant right now. If the retrieval system pulls in too much episodic history, it might crowd out the procedural instructions, causing the agent to forget how to use its tools. If it pulls in too much semantic knowledge, it might push out the recent conversation history, causing the agent to lose the thread of the current dialogue.
To manage this competition, modern memory systems employ multi-stage retrieval pipelines. First, a fast, lightweight algorithm (like BM25 or a simple vector similarity search) pulls a broad set of potentially relevant memories from the non-parametric store. Then, a more computationally expensive cross-encoder model re-ranks these memories, scoring them not just on semantic similarity, but on recency and salience. Only the highest-scoring memories are allowed to cross the threshold into the context window.
This is where the KV cache (Key-Value cache) plays a critical role. While not a cognitive memory type, the KV cache is a vital piece of memory infrastructure. It stores the mathematical representations (attention matrices) of tokens the model has already processed. If an agent uses the same massive procedural memory (system prompt) for every interaction, the KV cache prevents the model from having to recompute that prompt every time, saving massive amounts of compute and latency. In modern systems, the KV cache acts as a bridge between static storage and active computation, allowing agents to maintain large working memories without incurring prohibitive costs on every turn of the conversation.
Furthermore, the composition problem requires sophisticated eviction strategies. When the context window fills up, the system must decide what to drop. Does it summarize the oldest episodic memories? Does it truncate the semantic context? Does it compress the procedural rules? These decisions define the architecture of the memory system and ultimately determine the intelligence and coherence of the agent. Some systems use a "sliding window" approach, simply dropping the oldest tokens. More advanced systems use "loss-aware pruning," actively evaluating which memories are least likely to be needed for the current task and evicting them first.
The Security and Privacy Implications of Memory
As memory systems become more sophisticated, they introduce entirely new vectors for security vulnerabilities and privacy breaches. When an agent maintains a persistent, non-parametric memory store across thousands of interactions, that database becomes a high-value target. The attack surface expands from the immediate prompt to the entire historical archive of the agent's existence.
The most prominent threat is indirect prompt injection via memory poisoning. If an attacker can trick an agent into storing a malicious instruction in its episodic or semantic memory, that instruction will lie dormant until the retrieval system surfaces it in a future session. Once injected into the working memory, the model will execute the malicious instruction as if it were a legitimate procedural rule. This means an attack executed on Tuesday could compromise the agent's behavior on Friday, long after the original malicious input has vanished from the active conversation history. This "sleeper agent" vulnerability is particularly dangerous because it bypasses traditional input filtering mechanisms that only scan the immediate user prompt.
Privacy presents an equally daunting challenge. In human cognition, memory naturally degrades over time—a feature that provides a biological "right to be forgotten." AI memory systems, however, are perfectly durable unless explicitly engineered otherwise. If an agent stores sensitive personal data in its non-parametric memory, that data might be retrieved and inadvertently leaked in a completely different context. For example, an agent might retrieve a user's medical history when answering a seemingly innocuous question about scheduling, simply because the vector similarity search found a weak semantic link.
To mitigate these risks, enterprise memory systems are beginning to implement memory compartmentalization. This involves strictly partitioning episodic and semantic stores by user, session, and authorization level. When the retrieval system queries the database, it must pass through an access control layer that verifies the agent is currently authorized to access those specific memories. Additionally, some systems are experimenting with "memory decay" algorithms, where the relevance score of a memory artificially degrades over time unless it is repeatedly reinforced, mimicking the natural forgetting curve of human cognition. Other approaches involve encrypting memories at rest and only decrypting them when a specific cryptographic key is provided by the active session context.
Evaluating Memory System Performance
As memory systems grow more complex, evaluating their performance has become a distinct engineering discipline. You cannot simply measure a memory system by its storage capacity; you must measure its retrieval accuracy, its context utilization, and its impact on the agent's overall coherence. This requires a shift from traditional database metrics to cognitive performance metrics.
The primary metric for non-parametric retrieval is Recall@K, which measures whether the correct piece of information was successfully retrieved in the top K results from the database. However, in agentic systems, Recall@K is insufficient. Even if the correct episodic memory is retrieved, the model might fail to utilize it if it is buried too deep in the context window or overshadowed by conflicting semantic facts. A system with perfect Recall@K can still produce an agent that acts as if it has amnesia.
To address this, engineers use Faithfulness and Answer Relevancy metrics. Faithfulness measures whether the agent's final output is strictly supported by the information in its working memory, ensuring it hasn't hallucinated a response based on outdated parametric weights. Answer Relevancy measures whether the retrieved memory actually helped answer the user's prompt. These metrics are often calculated using a secondary "judge" LLM that evaluates the primary agent's output against the retrieved context.
Furthermore, memory systems must be evaluated on their Consolidation Efficiency. This measures how well the system abstracts raw episodic logs into semantic facts. A highly efficient consolidation process reduces the total token count required to store a user's profile while maintaining the same level of personalization accuracy. As agents operate over longer time horizons, consolidation efficiency becomes the primary driver of cost and performance. If an agent requires 10,000 tokens of retrieved episodic history to remember a user's preferences, the memory system is failing; if it can consolidate that history into a 50-token semantic profile, the system is succeeding.
The Future of Memory Architecture
The field is currently fragmenting as researchers explore new ways to bridge these tiers. A comprehensive survey of memory mechanisms (Wu et al., 2025) notes that the most pressing challenge is consolidation—the automated process of taking raw episodic logs from non-parametric memory and abstracting them into durable semantic knowledge. Without effective consolidation, non-parametric databases simply grow larger and noisier over time, eventually overwhelming the retrieval system with redundant or conflicting information.
We are also seeing the rise of dedicated memory orchestration layers. Instead of application developers manually wiring vector databases to their prompts, specialized infrastructure now handles the encoding, retrieval, and eviction of memories automatically. These systems treat the context window like RAM and the external databases like a hard drive, paging information in and out as the agent needs it. This operating-system-inspired approach allows agents to maintain the illusion of infinite memory while strictly adhering to the hard limits of the context window.
Another active area of research is the transition of knowledge from non-parametric to parametric storage. While fine-tuning is currently too expensive to run continuously, researchers are exploring methods for "implicit memory updates" where an agent can slowly bake its most frequently accessed non-parametric memories into its own weights during idle periods. This would mimic human sleep consolidation, where the day's episodic experiences are integrated into the brain's semantic network.
Ultimately, a robust memory system is what separates a reactive chatbot from a proactive agent. By carefully composing parametric weights, non-parametric databases, and active working memory, developers can build systems that learn from their mistakes, adapt to their users, and maintain coherence over months or years of interaction. The agents of the future will not just be defined by the size of their models, but by the sophistication of the memory systems that support them.
