$ cat ai-engineering/memory-evaluation.md

Memory Evaluation: Benchmarks and Custom Evals

How public benchmarks and custom evals measure memory recall, updates, temporal reasoning, and cost.

Jatin Bansal@blog:~/ai-engineering$ open memory-evaluation

Memory dashboards can report writes, reads, latency, and store size without measuring whether the system recalls the right fact. Evaluation needs multi-session traces, later questions with known answers, relevant episode IDs, and separate scores for temporal updates and abstention.

The memory-eval contract

A memory eval ingests a fixed corpus of multi-session interactions before presenting its questions. Questions are tagged by ability, including single-hop recall, multi-hop reasoning, temporal reasoning, knowledge updates, and abstention. Reports should include retrieval and answer quality alongside tokens, latency, and cost.

RAG evaluation scores retrieval against a mostly static corpus. Memory evaluation scores a corpus the system created during prior interactions. A needle-in-a-haystack test asks whether a model can find a literal fact already present in its prompt; a memory test asks whether the full system can recover a paraphrased fact across sessions.

Public benchmarks

Five benchmarks define the field in 2026. Each probes a different facet. A defensible memory system has numbers on at least the first two; serious teams run all five.

LoCoMo (Maharana et al., 2024). Fifty multi-session conversations between two fictional speakers, each spanning up to 32 sessions and averaging ~600 turns (~16K tokens), with timestamps and multimodal context. The 1,540-question QA set covers five categories: single-hop (intra-session recall), multi-hop (cross-session synthesis), temporal reasoning, open-domain, and adversarial (questions that should be refused). The paper reports GPT-4 at ~32.1 F1 against a human ceiling of ~87.9; the field’s headline gap. Recent framework numbers: Mem0’s 2026 token-efficient algorithm reports 92.5% on LoCoMo; the 2025 Mem0 paper reports 66.9% vector-only and 68.4% with the Mem0g graph extension, both versus 52.9% for OpenAI’s built-in memory feature. The wide range reflects different LLM judges, prompts, and ingest pipelines; read the protocol before comparing.

LongMemEval (Wu et al., ICLR 2025). Five hundred manually-curated questions embedded in freely-scalable chat histories, probing five memory abilities: information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention. The construction is more rigorous than LoCoMo’s; each question is human-authored, with histories scalable to test ingest-time behavior at multiple sizes. The paper documents commercial chat assistants dropping ~30% on sustained interactions versus single-session baselines. Zep’s January 2025 paper reports 71.2% overall on LongMemEval with GPT-4o (versus 60.2% for vanilla full-context at 29s latency), with the largest gains in temporal-reasoning (+17.3pp) and multi-session (+13.6pp). Mem0’s 2026 algorithm reports 94.4% on LongMemEval.

MemoryAgentBench (Hu et al., 2025). Probes four cognitive competencies; accurate retrieval, test-time learning, long-range understanding, and selective forgetting. The selective-forgetting category finally exposed the contradiction-resolution failure mode every framework had been papering over: all paradigms fail dramatically on multi-hop conflict resolution, with best accuracy at or below 6% on CR-MH (Contradiction Resolution, Multi-Hop) even in the most advanced models. Run it if your product depends on the agent updating beliefs as facts change.

BEAM (Mem0’s State of AI Agent Memory 2026). Scales to 1M and 10M token regimes, with ten categories including preference following, knowledge update, event ordering, abstention, and contradiction resolution. Mem0’s 2026 numbers: 64.1 and 48.6 on BEAM 1M/10M. The contradiction-resolution and event-ordering categories are where production systems silently regress as corpora grow.

∞Bench (Zhang et al., 2024), NoLiMa (Modarressi et al., ICML 2025), and RULER (Hsieh et al., 2024) are long-context evals, not memory evals; they measure whether a model can find a fact buried in a 100K+ or 32K+ token prompt, not whether a system can recall across sessions. They calibrate the long-context fallback: NoLiMa’s finding that 11 of 13 tested 128K-claiming models drop below 50% of their short-context baseline at 32K is the empirical case for memory-as-a-system over context stuffing.

A defensible benchmark story: numbers on LoCoMo and LongMemEval for the headline; MemoryAgentBench or BEAM if contradiction and selective forgetting matter; ∞Bench/NoLiMa/RULER numbers to justify why the system uses retrieval rather than bigger context.

Retrieval and answer metrics

Two metric families do the work. Precision/recall/F1 over retrieved memories measures the retrieval substrate; given a query, how many of the top-K retrieved episodes are actually relevant, and how many of the relevant episodes in the store made it into the top-K? End-to-end answer correctness (LLM-as-judge, sometimes string-match against a normalized gold) measures whether the whole system; retrieval plus generation; answers right.

Both matter and fail differently. A retrieval substrate at 90% precision/recall with a generator that fumbles synthesis scores low end-to-end. A substrate that retrieves garbage paired with a generator whose parametric memory carries the answer scores high. The Memory for Autonomous LLM Agents survey (2025) lays out the taxonomy; the production rule is measure both, separately, and pay attention when they diverge.

The precision/recall mechanics, for episodic recall against a known gold:

Top-K retrieval: the system returns K episodes for a query.
Gold-relevant set: the eval pre-computes which episodes are relevant. For LoCoMo this is human-annotated; for a custom eval, you build it.
Precision@K: (relevant retrieved) / K. Recall@K: (relevant retrieved) / (total relevant in store). F1@K: harmonic mean.

Pair these with temporal-correctness for the temporal category; “what did the agent believe last Tuesday?” should return the belief that held as of last Tuesday, not the current one. Bi-temporal stores like Graphiti score this category much higher than vector-only stores.

For contradiction resolution, the metric is supersession behavior: when two episodes claim contradictory facts, does the answer reflect the most recent (or most-trusted) claim, or average / pick at random? MemoryAgentBench CR-MH scores this; for a custom eval, the simplest version is hand-built “fact A at session N; ¬A at N+5; question at N+10” tuples.

Code: a precision/recall harness (Python)

Phase-separated ingest-then-query, category-tagged questions, per-category metrics with token-cost alongside. Store-interchangeable across Mem0, LangGraph store, Letta, or a homegrown vector store.

python

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
# pip install mem0ai openai
from dataclasses import dataclass, field
from typing import Any, Callable, Literal
import json, time
from collections import defaultdict

Category = Literal["single_hop", "multi_hop", "temporal", "knowledge_update", "abstention"]

@dataclass
class EvalQuestion:
    qid: str
    user_id: str
    question: str
    gold_answer: str | None  # None for abstention category
    relevant_episode_ids: set[str]  # ground truth for retrieval metrics
    category: Category

@dataclass
class IngestEvent:
    user_id: str
    text: str
    episode_id: str
    timestamp: float  # seconds since epoch; preserves multi-session structure

@dataclass
class EvalRun:
    by_category: dict[Category, list[dict]] = field(default_factory=lambda: defaultdict(list))

    def add(self, cat: Category, **fields):
        self.by_category[cat].append(fields)

    def summary(self) -> dict:
        out = {}
        for cat, rows in self.by_category.items():
            if not rows: continue
            n = len(rows)
            out[cat] = {
                "n": n,
                "precision_at_k": sum(r["precision"] for r in rows) / n,
                "recall_at_k": sum(r["recall"] for r in rows) / n,
                "f1_at_k": sum(r["f1"] for r in rows) / n,
                "answer_correct": sum(r["answer_correct"] for r in rows) / n,
                "avg_tokens": sum(r["tokens"] for r in rows) / n,
                "avg_latency_ms": sum(r["latency_ms"] for r in rows) / n,
            }
        return out


def f1(p: float, r: float) -> float:
    return 0.0 if (p + r) == 0 else 2 * p * r / (p + r)


def run_eval(
    ingest_events: list[IngestEvent],
    questions: list[EvalQuestion],
    memory_write: Callable[[IngestEvent], None],
    memory_search: Callable[[str, str, int], list[dict]],     # (user_id, query, k) -> [{episode_id, text, score}]
    answer_with_context: Callable[[str, list[str]], tuple[str, int]],  # (q, retrieved_texts) -> (answer, tokens_used)
    judge_correct: Callable[[str, str | None, Category], bool],  # (pred, gold, category) -> bool
    k: int = 5,
) -> EvalRun:
    # Phase 1: ingest. The memory system has no access to the questions here.
    for ev in sorted(ingest_events, key=lambda e: e.timestamp):
        memory_write(ev)
    # Phase 2: query. Per-question retrieval, scoring, and category attribution.
    run = EvalRun()
    for q in questions:
        t0 = time.perf_counter()
        retrieved = memory_search(q.user_id, q.question, k)
        retrieved_ids = {r["episode_id"] for r in retrieved}
        relevant_retrieved = retrieved_ids & q.relevant_episode_ids
        # Edge: abstention questions have no relevant episodes; recall is 1 by convention,
        # precision is 0 if any episodes were retrieved (system should have abstained on empty).
        if q.category == "abstention":
            precision = 0.0 if retrieved else 1.0
            recall = 1.0
        else:
            precision = (len(relevant_retrieved) / len(retrieved_ids)) if retrieved_ids else 0.0
            recall = (len(relevant_retrieved) / len(q.relevant_episode_ids)) if q.relevant_episode_ids else 0.0
        answer, tokens = answer_with_context(q.question, [r["text"] for r in retrieved])
        correct = judge_correct(answer, q.gold_answer, q.category)
        run.add(
            q.category,
            qid=q.qid,
            precision=precision,
            recall=recall,
            f1=f1(precision, recall),
            answer_correct=1.0 if correct else 0.0,
            tokens=tokens,
            latency_ms=(time.perf_counter() - t0) * 1000.0,
        )
    return run

The memory_write and memory_search callables are the only contract; the same harness scores Mem0, LangGraph, and a homegrown vector store identically. For abstention questions, the judge should accept any refusal phrasing as correct when the gold is None. EvalRun.summary() is the publishable artifact: per-category precision, recall, F1, end-to-end correctness, and token/latency budget in one dict.

Designing a custom eval

Public benchmarks are necessary, not sufficient. They probe generic conversational memory; they don’t probe your product’s recall patterns. A customer-support agent has return-policy preferences and account-tier context that LoCoMo doesn’t touch; a medical-history agent has drug interactions and prior diagnoses that LongMemEval doesn’t touch. The custom eval is where workload-specific failure modes show up.

The minimum shape:

20-50 real or near-real multi-session traces: Real if privacy permits; otherwise synthetic in the product’s actual conversational shape. Curated 20-50 beats lazy 500.
Per-trace question set, 10-20 questions: Tagged by the LongMemEval five-ability frame. Each has a gold answer (or None for abstention) and relevant-episode IDs for retrieval metrics.
An ingest-then-query harness: The Python/TS code above is the template.
A weekly cron run: Every memory-policy change passes the eval before shipping. A per-category regression is blocking even if the aggregate holds.
A growing adversarial subset: Every production incident caused by a memory bug becomes a regression question. Over time this set becomes more useful than the seed corpus.

The trap to avoid: scoring only end-to-end answer correctness and ignoring retrieval precision/recall. If retrieval is at 30% recall but the generator’s parametric knowledge masks it for 80% of questions, the system will fail catastrophically the moment the question distribution shifts. Retrieval metrics are the leading indicator; end-to-end is the lagging one.

For Ragas-style metrics applied to memory: context precision and context recall translate directly, and faithfulness catches “generator confabulates a memory.” The multi-session-specific categories (temporal correctness, contradiction resolution) diverge; Ragas doesn’t have those out of the box.