AI Agent Memory Design: 4 Patterns for Context That Persists Beyond a Single Conversation

The first real frustration when building agents isn't prompt engineering or model selection — it's realizing that the agent forgets everything the moment a session ends. Every API call is stateless by design, which means building a useful, relationship-aware agent requires deliberately engineering memory.

Here are four patterns I've implemented across different projects. None of them is universally correct; the right choice depends on your use case.

Pattern 1: In-Session Compression (Short-Term Memory)

The simplest approach: when a conversation gets long, compress the older exchanges into a summary block rather than letting the context window overflow.

def compress_history(messages: list[dict], keep_recent: int = 6) -> list[dict]:
    if len(messages) <= keep_recent:
        return messages
    
    old_messages = messages[:-keep_recent]
    recent_messages = messages[-keep_recent:]
    
    summary_prompt = [
        {"role": "user", "content": f"Summarize the following conversation in 3-5 sentences:\n{format_messages(old_messages)}"}
    ]
    summary = call_llm(summary_prompt)
    
    compressed = [
        {"role": "system", "content": f"Summary of earlier conversation: {summary}"}
    ] + recent_messages
    
    return compressed

This pattern doesn't survive session boundaries — there's no "picking up where we left off yesterday." But it requires no external storage, keeps implementation simple, and handles most single-session use cases cleanly. It's the right starting point for prototypes and task-scoped agents.

Pattern 2: Key-Value Store for Persistent Facts

When users share information about themselves — their name, preferences, project details, configuration choices — you want that to stick across sessions. This pattern extracts discrete "facts" and stores them in a persistent key-value store.

The critical design decision here is letting the agent decide what's worth storing, rather than logging everything.

def extract_and_store_facts(user_message: str, agent_response: str, user_id: str):
    extraction_prompt = f"""
From the following conversation, extract any persistent facts worth storing for future reference.
Exclude transient information ("I'm tired today") and capture only stable user information or preferences.
 
User: {user_message}
Assistant: {agent_response}
 
Return a JSON array of facts, or an empty array if none apply.
Format: [{{"key": "fact_type", "value": "content"}}]
"""
    facts = call_llm_json(extraction_prompt)
    
    for fact in facts:
        key = f"user:{user_id}:{fact['key']}"
        redis_client.set(key, fact['value'], ex=86400*365)  # 1-year TTL

One practical issue to plan for: stale facts. A "lives in Tokyo" entry stored six months ago may no longer be accurate. Set TTLs on stored facts, or record a timestamp and build in logic to prompt re-verification for facts older than a threshold. Without this, you'll eventually surface outdated information confidently.

Pattern 3: Vector Search for Semantic Recall

The most commonly discussed pattern: store past exchanges in a vector database, then retrieve semantically similar entries to include in the current context window.

from openai import OpenAI
import chromadb
 
client = OpenAI()
chroma = chromadb.Client()
collection = chroma.get_or_create_collection("agent_memory")
 
def store_exchange(user_msg: str, agent_msg: str, session_id: str):
    text = f"User: {user_msg}\nAssistant: {agent_msg}"
    embedding = client.embeddings.create(
        model="text-embedding-3-small",
        input=text
    ).data[0].embedding
    
    collection.add(
        documents=[text],
        embeddings=[embedding],
        ids=[f"{session_id}_{int(time.time())}"],
        metadatas=[{"session_id": session_id, "timestamp": time.time()}]
    )
 
def recall_relevant_memory(query: str, n_results: int = 3) -> str:
    query_embedding = client.embeddings.create(
        model="text-embedding-3-small",
        input=query
    ).data[0].embedding
    
    results = collection.query(
        query_embeddings=[query_embedding],
        n_results=n_results
    )
    
    if not results["documents"][0]:
        return ""
    
    return "Relevant past conversations:\n" + "\n---\n".join(results["documents"][0])

Two production gotchas worth knowing before you deploy this:

Gotcha 1: Low-relevance results injecting noise

Without a similarity threshold, "somewhat close" past exchanges get injected into every response — often hurting quality rather than helping it. The query result includes a distances field; filter out anything below a minimum cosine similarity (e.g., 0.75) before inserting into context.

Gotcha 2: Conflicting information from different points in time

If a user's position on something changed over time, you may retrieve both the old and new version in the same search. Sorting by timestamp descending and prioritizing recent entries — or explicitly surfacing the conflict to the LLM — is necessary for time-sensitive facts.

Pattern 4: Episodic Memory with Structured Logs

The highest-cost pattern, but the most powerful for long-running agents: storing conversations as structured episodes with topic, decisions, and follow-ups — enabling queries like "summarize last week's discussion about Project X."

from dataclasses import dataclass
from datetime import datetime
 
@dataclass
class Episode:
    session_id: str
    timestamp: datetime
    topic: str
    participants: list
    summary: str
    key_decisions: list
    follow_ups: list
 
def create_episode(messages: list[dict], session_id: str) -> Episode:
    analysis_prompt = f"""
Analyze the following conversation and return structured JSON:
- topic: main topic (one sentence)
- summary: overall summary (3-5 sentences)
- key_decisions: list of decisions made
- follow_ups: items to revisit next session
 
Conversation:
{format_messages(messages)}
"""
    structured = call_llm_json(analysis_prompt)
    
    return Episode(
        session_id=session_id,
        timestamp=datetime.now(),
        topic=structured["topic"],
        participants=["user"],
        summary=structured["summary"],
        key_decisions=structured.get("key_decisions", []),
        follow_ups=structured.get("follow_ups", [])
    )

This pattern earns its complexity cost when you have agents that maintain ongoing relationships with users over months, or agents managing project work where "what did we decide last month" needs a reliable, structured answer. Vector search alone often can't surface that kind of temporally-structured information accurately.

Choosing the Right Pattern

In practice, these patterns combine. My current setup for a personal agent uses Pattern 2 (fact storage) plus Pattern 3 (vector search): explicit user information goes to Redis, and conversation context goes to the vector database.

Starting guidance: if you want the agent to remember things users explicitly tell it, start with Pattern 2. If you want it to recall the flow of past conversations, add Pattern 3. Only reach for Pattern 4 when you're building something intended to run continuously for a year or more.

The Design Decision That Comes First

Regardless of which pattern you choose, the most important question to answer before implementing is: what should this agent be allowed to forget?

Trying to remember everything produces noise. Defining upfront what's worth retaining — and actively discarding the rest — is what keeps memory useful rather than cluttered. The boundary between "memorable" and "ephemeral" is a product decision, not a technical one, and it's worth getting explicit about before you write a line of storage code.

Start with one piece of information your users would want the agent to remember. Implement Pattern 2 for it. Once that works, you'll have a much clearer intuition for whether you need Pattern 3 on top of it.