Short-Term and Long-Term Memory Architecture for Agents

Enterprise agents are moving fast: industry projections point to rapid growth in enterprise applications integrating task-specific agents over the next year or two. But industry analysts widely expect a significant share of agentic projects to get cancelled within the next few years, and memory failures are near the top of the reason list. An agent that forgets everything between sessions is broken because nobody built the architecture to hold what the model needs to remember, not because the model is weak. It's broken because nobody built the architecture to hold what the model needs to remember.

What short-term memory actually is and its responsibilities during inference

Short-term memory is everything the agent holds in its context window while it's actively working. Split it into two buckets.

First, there is configuration, which includes the system prompt, the persona, the tool descriptions, and the rules the agent has to follow. Second, events: the user's messages, tool call results, anything the environment just handed back. Both live in the same space, and both disappear the moment the session ends.

Think of it like RAM on a computer. Fast, right there, gone the second you close the app. An agent can be smart about how it manages that space, trimming what it keeps, summarizing what's getting long, prioritizing the last few turns over the first few. None of that changes the basic fact: short-term memory was never built to survive past the conversation it's in. It's necessary for the agent to reason at all, but by itself it guarantees the agent starts from zero every single time a new session opens.

The three types of long-term memory an agent needs and what each one stores

Long-term memory lives outside the model, in a database somewhere, and it survives session boundaries. But calling it "long-term memory" like it's one thing is misleading. It splits into three distinct types, and each one gets written differently, retrieved differently, and fails differently when the design is wrong.

Episodic memory stores specific past events, tagged with time and context. A 2025 paper, "Episodic Memory is the Missing Piece for Long-Term LLM Agents" (arXiv:2502.06975), lays out five properties this kind of memory needs: long-term storage, explicit reasoning over it, single-shot learning from one example, instance-specific detail, and context tied to the moment it happened.

Most teams get it wrong here. If an agent summarizes an episode at write time, it collapses a specific event into a generalized fact, and the specific detail is gone before anyone ever tries to retrieve it. That's one of the most common mistakes in production systems right now. The fix is straightforward in principle: log structured events with timestamps and tags in something lightweight like SQLite or Redis, then retrieve by vector similarity when a new situation looks like an old one.

Semantic memory stores the distilled, general stuff: facts, user preferences, domain rules, profiles on people or entities. This is the knowledge that doesn't belong to one moment, it accumulates across many. It gets built by summarization pipelines that compress episodic logs over time, and it gets pulled back out by semantic similarity search. This is the layer that lets an agent remember "this user prefers short answers" without remembering the exact conversation where that became clear.

Procedural memory stores skills: how to do something, not what happened or what's true. It comes from experience, success and failure trajectories the agent can learn from and reuse on the next task. Increasingly, teams are pulling procedural memory out of reinforcement learning over agent trajectories, not just handwriting static instructions into a prompt.

This three-way split, episodic, semantic, procedural, traces back to a framework psychologists call the Tulving trichotomy. What's notable is that open-source agent projects have landed on the same three categories independently, each with its own extraction pipeline. It's a sign the split reflects something real, given that open-source agent projects have landed on the same three categories independently, each with its own extraction pipeline. It's a sign the split reflects something real about how memory needs to work.

The mechanics of writing, retrieval, and promotion between the two tiers

The handoff between short-term and long-term memory happens through a write path, and figuring out what to write is harder than it sounds.

Log everything and the system drowns. Noise piles up, retrieval quality drops because the signal gets buried, and storage costs climb every single day the agent runs. The better path is selective: extract what matters, tag it as episodic, semantic, or procedural at write time, and only then commit it to storage. Most production deployments skip this step entirely, and the effect appears later in an agent that either forgets things it should know or drags irrelevant history into every response.

Retrieval has its own split. Vector search finds things that read as similar, but it misses how facts connect to each other, who's related to whom, what happened before what. Graph-based retrieval catches those connections but can miss a fuzzy match that vector search would catch instantly. Neither one alone is enough. Production systems increasingly run both, and reconcile the results before the agent ever sees them.

OS-inspired hierarchical designs that manage the boundary between short-term and long-term memory

The comparison to computer operating systems isn't just a metaphor for the sake of it. Operating systems solved this exact problem decades ago: fast registers, slower RAM, and a hard disk underneath for whatever needs to stick around. Data moves between those layers based on rules, promotion when something gets used a lot, eviction when it doesn't. Agent memory design is now borrowing that logic directly.

MemoryOS, from Kang and colleagues (EMNLP 2025), is one concrete version of this. It splits memory into three tiers: short-term, mid-term, and long-term personal memory, run by four modules called Storage, Updating, Retrieval, and Generation. Moving from short-term to mid-term follows a dialogue-chain FIFO rule, first in, first out, based on conversation structure. Moving from mid-term to long-term uses something called segmented page organization, which is basically grouping related mid-term memories into chunks before they get committed permanently.

The results back up the design. On the LoCoMo benchmark, MemoryOS showed a 48.36% average improvement in F1 score and a 46.18% improvement in BLEU-1, tested against baselines running on GPT-4o-mini. Those aren't just cleaner diagrams, those numbers reflect an agent that stays coherent and personalized over long conversations instead of drifting. The promotion logic between tiers is a critical design decision for anyone building on this pattern, and getting it wrong means memories either never graduate to long-term storage or do so without the right structure.

A second framework, the Multi-Layer Memory Framework (arXiv:2603.29194), splits dialogue history into working, episodic, and semantic layers, with gating logic that decides what gets retrieved and regularization that keeps false memories from building up. That false memory control keeps retrieval systems from quietly accumulating wrong associations the longer they run. Retrieval systems without it tend to accumulate wrong associations the longer they run, quietly getting worse in ways that become visible only when an agent confidently states something untrue.

Across all three of these systems, the pattern repeats: explicit rules for what moves where, when, and in what shape. Skip that step, and memory degrades on its own, slowly, in ways that are hard to catch until an agent is already giving bad answers.

Graph-based memory as the frontier for relational and temporal reasoning across sessions

Standard long-term memory setups use linear structures: token sequences, vector databases, log buffers. These store facts fine, but they don't capture how facts relate to each other or how those relationships shift over time. Graph-based memory is a different approach entirely: instead of a flat log, it builds a structured map of experience, with relationships, hierarchy, and paths an agent can actually traverse.

Zep (arXiv:2501.13956) builds this as a temporal knowledge graph made for agent memory specifically: entities, the relationships between them, and time-stamped records of how those relationships change. That lets an agent pull up not just "what is true" but "what was true when."

MemoriesDB (arXiv:2511.06179, November 2025) pushes further. Every memory in the system is stored as one entity that holds time, meaning, and relationship all at once, so a single record answers when something happened, what it meant, and what it connects to. It's built on top of PostgreSQL with the pgvector extension, combining a time-series store, a vector database, and a graph system into one append-only schema.

The payoff appears clearly in benchmark numbers from Mem0's 2026 algorithm update. The two biggest jumps came on temporal queries, up 29.6 points over the prior version, and multi-hop reasoning, up 23.1 points. Those are exactly the categories where flat vector search tends to fall apart, because answering "what changed after the contract renewal" or "who introduced this person to that one" takes structure, not just similarity. Per mem0.ai, the state of the art is 92.5 on LoCoMo and 94.4 on LongMemEval at around 6,900 tokens per query. Scores like that don't come from semantic search alone. They come from memory that understands relationships and time as first-class information, not an afterthought bolted onto a vector index.

Live web data in agent memory: the staleness problem that no internal store solves

Episodic, semantic, and procedural memory all cover what an agent has learned from its own past. None of them help with something else agents constantly need: the current state of the world outside the conversation. Prices change. Documentation gets rewritten. News happens. An agent's internal memory, no matter how well designed, has no way to know about any of it unless it was told.

Staleness occurs in two ways. Coverage failure: a page that didn't exist the last time anything crawled the web simply isn't in the index, full stop. Drift failure: a page that was indexed has since changed, but the stored embedding still points at the old version. Both failures are invisible to the model itself. It answers confidently without any internal signal that the information is wrong. It just answers confidently, using information that's out of date, and there's no internal signal telling it otherwise.

No amount of better memory consolidation fixes this, because the problem isn't retrieval quality, it's access. The agent was never given the current information to begin with. Live web retrieval solves a different problem than memory management, and it needs to be treated as a separate system, not folded into the memory stack and hoped for.

The practical shape of live-web retrieval looks different from the old crawl-and-index model. Instead of pre-fetching the entire internet and hoping the right page got captured before it changed, sources get discovered and pulled at the moment a query needs them. That flips the cost structure: instead of paying to crawl continuously, an agent pays to fetch the handful of pages a specific question actually requires.

Making that work inside an agent pipeline takes real engineering. Raw HTML has to turn into clean Markdown that doesn't burn through the token budget. Pages that render with client-side scripting need to be handled, not skipped. And the output has to land in a format the model can reason over directly, no extra cleanup step in between. Building that in-house from scratch is weeks of scraper maintenance before an agent ships a single feature. An API that turns any URL into ready-to-use Markdown is the piece that completes a memory architecture built around persistent internal storage. It handles the crawling, rendering, and extraction underneath.

Deciding what each layer of memory should hold in a production agent

Getting agent memory right means figuring out which layer owns which kind of content, not picking one system and calling it done. The real question is which layer owns which kind of content, and answering it wrong means retrieving the wrong thing at the wrong moment, or writing so much down that retrieval slows to a crawl under its own weight.

Short-term memory owns the current task: the live user turn, tool results that just came back, whatever configuration the agent needs to function right now. Nothing here should be expected to survive past the session's close.

The semantic store owns durable, distilled facts, things like user preferences, entity details, stable domain rules. These shouldn't get written directly in real time. They should get generated from episodic memory through a summarization step, after the fact, once there's enough signal to distill something worth keeping.

The procedural store owns reusable strategies and workflows that have actually been validated. This is the hardest layer to fill automatically, and most teams start it by hand, writing down the strategies that work before any system learns to generate them on its own.

Layer the three long-term types under a well-managed short-term window, add explicit rules for what promotes and what gets evicted, and pair the whole thing with live web access for anything that changes outside the conversation. That combination is what keeps an agent coherent past the first conversation and into the hundredth.