AgentOS on LongMemEval: 85.6% on S, 70.2% on M

"Not everything that can be counted counts, and not everything that counts can be counted."

— William Bruce Cameron, Informal Sociology, 1963

Memory benchmarks for AI agents reward retrieval over inference. The score goes up when the system dumps more context into the reader's window and lets the LLM sort the result out. That's not what most people mean by "memory" when they ask for it. It's a search engine on top of a smart enough reader to compensate for the noise.

The numbers people quote (LongMemEval, LOCOMO) inherit this. So when Dhravya publishes 99% and Mastra publishes 95%, the right reaction is not "huh, our 85.6% looks bad" but "what reader, what retrieval, what judge, and can I rerun it." Most of the time, at least one of those answers is missing. (MemPalace also publishes 100% on every memory bench, but that's a broken evaluator that returns 100% no matter what you feed it, so it's not a competitor result, it's a methodology bug.)

AgentOS is the open-source TypeScript runtime I work on. It implements eight cognitive mechanisms from published memory research (retrieval-induced forgetting, reconsolidation, source-confidence decay, more) on top of Ebbinghaus decay, so the agent forgets on purpose. The bench numbers below are how that design holds up against everyone else's, each with the answer model that produced it, which is the part the headline percentages alone don't give you.

Two results, both at full N=500, both from the same reader setup: a gpt-5-mini classifier guesses each question's category, gpt-4o answers the ones it labels temporal-reasoning or single-session-user, and gpt-5-mini answers the other four categories.

LongMemEval-S: 85.6% at $0.0090 per correct answer, 3.6-second median latency. Mastra Observational Memory publishes 84.23% with gpt-4o as the reader, the strongest published memory-library number at that reader. With gpt-4o answering every question, the same AgentOS retrieval scored 84.2%, level with Mastra; routing four categories to gpt-5-mini adds the other 1.4 points. EmergenceMem Internal publishes 86.0% (0.4 points above us), but Internal is their closed-source SaaS (emergence.ai/web-automation-api). Not a library you can pull into your own project. Their public reference repo emergence_simple_fast scores 79% on the same benchmark and ships with no license, meaning the code is publicly readable but not legally redistributable. Where AgentOS differs is the rest of the record: both headlines ship with a 95% range, cost per correct, latency, and the command that produced them.

LongMemEval-M: 70.2% at $0.0078 per correct answer. M is the harder variant: 1.5M tokens of conversation per question, 500 sessions per haystack, bigger than any production LLM context window I've seen. Of the 14 memory-library vendors I audited, none of the rest publish an M number at all. AgentOS at 70.2% is competitive with the strongest published M results in the original LongMemEval paper (Wu et al., ICLR 2025, Table 3) ¹. At reader-Top-5, that's +4.5 points above the round-level configuration (65.7%) and 1.2 points below the session-level configuration (71.4%); the paper's strongest GPT-4o result is 72.0% at round-level Top-10. Those paper configurations have GPT-4o answer every question, while the 70.2% uses the same reader router as S. The closest published external M number is AgentBrain's 71.7% from their closed-source SaaS.

(Aside: M's $0.0078 per correct is lower than S's $0.0090, which sounds backwards because M's haystacks are 13× larger. The reader never sees the haystack. Retrieval narrows it. M's headline config passes --reader-top-k 5, so the reader receives 5 chunks per question regardless of corpus size. S's headline leaves the flag at the adapter default of 10. Both headlines run the same gpt-5-mini classifier and reader router, so on the reader's side the difference is 5 chunks against 10.)

Both numbers are recorded in the bench's leaderboard with their per-category breakdowns, confidence ranges, cost and latency. The per-case run JSONs sit in the repository only as Git LFS pointers, and GitHub holds none of their objects, so a per-question comparison means rerunning the configuration yourself. The runtime is Apache-2.0 at github.com/framerslab/agentos; the bench harness is Apache-2.0 at github.com/framerslab/agentos-bench. One CLI command at the bottom of this post reproduces each headline.

The rest of the post covers: the architecture changes that produced each number, the audit of the vendor landscape (including the 99% / 95% claims that fall apart once you check what answer LLM the competitor used), the methodology checks behind every number above, and the reproduction commands. There's a lot of skepticism baked in. I built the bench harness because I wasn't going to trust my own numbers without a way to make them break.

TL;DR for the busy reader

Full benchmarks reference · Transparency audit framework

Part 1: LongMemEval-S, with and without the reader router

LongMemEval-S has 115K tokens of conversation per question and roughly 50 sessions per haystack. It fits in a single gpt-4o call. Every memory-library vendor with a public LongMemEval claim publishes on S.

The table below lists the published results with gpt-4o as the reader, plus the AgentOS headline that hands four of the six categories to gpt-5-mini. The AgentOS row with gpt-4o answering every question is the one that isolates memory architecture from base-LLM capability. AgentOS rows are full runs at N=500 questions, gpt-4o-2024-08-06 as judge, rubric 2026-04-18.1 (judge false-positive rate 1%).

With the router, AgentOS is 1.4 points above the Mastra OM gpt-4o number and 0.4 points below EmergenceMem Internal. With gpt-4o answering every question, AgentOS and Mastra are level (84.2% and 84.23%). Internal is closed-source SaaS behind emergence.ai/web-automation-api, not a library you can install. Their public reference, EmergenceAI/emergence_simple_fast, publishes 79% (we reproduced it at 80.6%), but the repo has no license: the code is publicly visible but not legally usable in derivative work. The 86% Internal number cannot be reproduced from public code at all.

So the practical comparison: at the gpt-4o reader, the two best "you can install this and use it in your product under a permissive license" memory-library numbers on LongMemEval-S are level, AgentOS at 84.2% and Mastra Observational Memory (Apache-2.0) at 84.23%. The reader router takes AgentOS to 85.6% by handing four categories to gpt-5-mini. EmergenceMem's 86% Internal is a SaaS endpoint; their 79% public reference is a license-less repo.

Median latency: AgentOS p50 is 3,558 ms; EmergenceMem's published median is 5,650 ms. The remaining vendors do not publish per-case latency.

Architecture

Every question flows through a single retrieval path: BM25 + dense + cross-encoder rerank (canonical-hybrid). With text-embedding-3-small as the dense embedder, recall@10 sits at 0.981 across the full N=500 set, so the reader sees the relevant chunks on essentially every query. Verbatim temporal detail and preference statements survive the pipeline intact, which is what the multi-session and single-session-preference categories require.

A lightweight classifier (gpt-5-mini, one extra LLM call per case at ~$0.000138) picks the reader model per category. Temporal-reasoning and single-session-user run through gpt-4o; the other four categories run through gpt-5-mini. Reader-model selection is bounded by the classifier and explicit per-category measurements, not by guesswork. The calibration table is below. Without the router, gpt-4o answers every question and the same retrieval scores 84.2%. The leaderboard puts the dominant lift from routing in single-session-preference, where gpt-5-mini scores 86.7% against gpt-4o's 76.7% on this retrieval.

Cost at scale: at $0.0090 per memory-grounded answer, 1,000 RAG calls cost $9. A chatbot averaging 5 RAG calls per conversation across 1,000 conversations costs ~$45.

Reader-router calibration

1export const MIN_COST_BEST_CAT_2026_04_28_TABLE = {
2  preset: 'min-cost-best-cat-2026-04-28',
3  mapping: {
4    'temporal-reasoning': 'gpt-4o',         // +11.8 points on TR vs gpt-5-mini
5    'single-session-user': 'gpt-4o',        // +4.3 points on SSU
6    'single-session-preference': 'gpt-5-mini', // +23.4 points on SSP
7    'single-session-assistant': 'gpt-5-mini',  // +1.8 points + cheaper
8    'knowledge-update': 'gpt-5-mini',          // +1.5 points + cheaper
9    'multi-session': 'gpt-5-mini',             // +3.5 points + cheaper
10  },
11};

Per-category at the 85.6% headline:

Twelve adjacent configurations, all regressed

Each of the following single-variable variants was tested as a change to the 85.6% configuration. Phase A rows ran on a 54-question stratified sample, so their deltas compare against a Phase A baseline of 85.2%; Phase B rows ran all 500 questions against 85.6%. None lifted the aggregate score.

The leaderboard records twelve single-variable variants against this configuration: the ten above, plus two S-tuned retrieval routers that each scored 77.8% at Phase A. All twelve regressed. The 85.6% configuration is a local optimum in the tested parameter space.

A note on HyDE at S scale

The Gao et al. (2022) HyDE paper introduced hypothetical document embedding as a recall-improver: ask the LLM to draft a plausible answer, embed that instead of the question, and let the answer-form embedding land closer to the answer-form documents in the index. It's a real effect at the right scale. The Lei et al. (2025) Adaptive HyDE paper shows the same on a 3M-post Stack Overflow corpus where the haystack is large and the question/answer registers are far apart.

LongMemEval-S is not that setting. The haystack is 50 sessions, the canonical pipeline already saturates retrieval, and BM25 + dense + Cohere rerank-v3.5 is precise enough that the right chunk is in the top-K most of the time. HyDE adds an extra LLM call to produce a hallucinated answer-shaped embedding, and at this scale that embedding is often less precise than the original question embedding for the categories that matter (single-session-user, knowledge-update, single-session-preference). The hypothetical chunks compete with the real ones in the rerank pool and displace correct hits. Net: -1.9 points and added latency. We turn it off.

The lesson isn't "HyDE is bad" — it's "HyDE earns its keep when the haystack outgrows the precision of the surface retrieval, and pays cost the rest of the time." Save it for M.

Part 2: LongMemEval-M, with the same reader router

LongMemEval-M has 1.5M tokens of conversation per question and roughly 500 sessions per haystack. It exceeds every production LLM context window, which forces evaluation through retrieval rather than prompt-stuffing.

What LongMemEval is, and what M means

LongMemEval ² is an academic memory benchmark introduced in "LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory" (Wu et al., ICLR 2025) ¹. The dataset, evaluation harness, and rubric are open source at github.com/xiaowu0162/LongMemEval ². The 12 paper authors are academic researchers, none affiliated with a memory-library vendor.

The benchmark ships two variants by haystack scale:

The S-to-M jump is a category change rather than a 13× scaling exercise. At S scale a memory architecture competes against the option of dumping the full conversation into the context window. Mastra's full-context baseline ³ at gpt-4o is 60.20%, and their Observational Memory configuration at the same model is 84.23%; the 24-point lift partly reflects token compression rather than memory architecture, because the OM config fits in fewer tokens and the reader has less to process. Penfield Labs makes the same point in their April 2026 LOCOMO audit ⁴: when the corpus fits in the context window, the benchmark is partly measuring context-window management.

At M scale the corpus does not fit in context. Retrieval is the only path, and the benchmark measures whether retrieval can find the relevant chunks in roughly 25,000 candidates across 500 sessions.

The vendor landscape on LongMemEval, audited 2026-04-29

The table below covers every memory library or platform with a public LongMemEval claim found in research pages, blog posts, GitHub repos, and peer-reviewed papers as of 2026-04-29.

Each row links the vendor's own published claim, and the transparency audit scores what six of them disclose on 12 axes.

Three operational barriers to publishing on M

1. Context window. S (115K tokens) fits in every modern long-context LLM. M (1.5M tokens) exceeds every production context window. Architectures that rely on prompt-stuffing or compression-then-stuffing score lower on M than on S.

2. Dataset loading. longmemeval_m.json is 2.7 GB. Node's V8 engine has a max-string-length cap that rejects fs.readFile on a file of that size. The streaming fix is chain([createReadStream, parser(), streamArray()]) from stream-json + stream-chain, routed by a file-size probe at >1 GB. The loader is src/benchmarks/longmemeval/dataset.ts.

3. Run cost. A memory-augmented full-context M run consumes ~750M input tokens at GPT-4o-128K pricing, roughly $1,250 per run. Retrieval-augmented M runs are $5 to $15.

What the LongMemEval paper reports on M

Wu et al., Table 3 of arXiv:2410.10813, reports academic-baseline configurations on LongMemEval-M. The strongest configuration in the paper:

72.0% on LongMemEval-M with GPT-4o + Stella V5 retriever + Value=Round + K=V+fact + Top-10.

The same Table 3 reports several other GPT-4o configurations: Round Top-5 K=V+fact at 65.7%, Session Top-5 K=V+fact at 71.4%, Session Top-10 K=V+fact at 70.0%. The 72.0% number is the strongest result in Table 3 across all GPT-4o configurations.

The dataset, evaluation harness, and rubric are open source at xiaowu0162/LongMemEval. The paper's GPT-4o results at Top-5 retrieval (the configuration AgentOS uses) span 65.7% (round-level) to 71.4% (session-level). At Top-10 the strongest is 72.0% (round-level).

Where we land

AgentOS at 70.2% is competitive with the strongest published M results in the LongMemEval paper. The paper's strongest GPT-4o result is 72.0% at round-level Top-10; at matched Top-5 retrieval the paper's results span 65.7% (round) to 71.4% (session). The paper has GPT-4o answer every question and the 70.2% routes four categories to gpt-5-mini, so the two match on retrieval budget and differ on reader. The closest published external number is AgentBrain's 71.7% from their closed-source SaaS, which requires access to a hosted endpoint to reproduce. agentos-bench publishes the configuration and a CLI reproduction at github.com/framerslab/agentos-bench.

Architecture

The 70.2% configuration uses HyDE-augmented BM25 + dense retrieval over text-embedding-3-small, Cohere rerank-v3.5 cross-encoder rerank with a candidate-pool multiplier of 5, reader-top-K=5, and the same per-category reader router used at S scale. A LongMemEval-M haystack contains ~1.5M tokens spread across 500 sessions, producing ~25,000 candidate chunks. The reader sees only the 5 highest-scoring chunks the cross-encoder returns.

Top-K=5 matches the LongMemEval paper's strongest M configuration (Wu et al., ICLR 2025, Table 3). At higher K, chunks ranked 6 and below frequently share lexical surface with the query but do not contain the answer; including them lowers the reader's signal-to-noise ratio. Liu et al. (2024), "Lost in the Middle" reports the same shape of failure at the long-context-LLM level.

The takeaway: a memory system that retrieves more is not always one that remembers better. At M scale, a retriever that hands the reader fewer (and better-ranked) chunks scores higher than one that hands it more. Recall is necessary; what the reader does with what it got is the rest of the work.

M cost at scale: at $0.0078 per correct over a 1.5M-token haystack, 1,000 RAG calls cost $7.80.

Per-category at the 70.2% M headline

Four one-knob probes all regressed on M

Each variant was tested as a single-variable change on top of the 70.2% configuration.

Top-K=5 with HyDE on and rerank-multiplier 5 is the local optimum in the tested parameter space.

The HyDE-off ablation sits within statistical noise of the headline (69.2% vs 70.2%, -1.0 within bootstrap range), so I'll call this one marginal rather than decisive. But the directionality matches the Lei et al. (2025) result for Stack Overflow: at 500-session, 1.5M-token haystack scale, BM25 + dense recall starts missing the right bridge sessions on multi-hop synthesis, and the hypothetical-answer embedding lands in the right neighborhood often enough to recover them. This is the regime where the HyDE paper's claim is load-bearing — recall-bound retrieval with a wide question-to-answer register gap. S didn't have that gap; M does.

So the policy across both variants ends up clean: HyDE off on LongMemEval-S, HyDE on for LongMemEval-M. The retriever doesn't decide for you — you do, based on whether your haystack outruns surface-form retrieval. AgentOS ships HyDE as opt-in (enabled: false by default) for exactly this reason.

Part 3: Why M is harder than S

Multi-session and temporal-reasoning together account for 53% of all M cases and post the lowest per-category scores at M scale (48.9% and 66.2%). Across 500 candidate sessions per haystack instead of 50, the relevant session is harder for the cross-encoder to surface. Multi-session bridge queries require the model to combine evidence from two distinct sessions; at S scale this means picking 2 of 50, at M scale 2 of 500. The remaining headroom on M is concentrated in those two categories. Improvements there will move the aggregate.

The single-session categories (assistant, user, preference) translate cleanly between scales because the relevant evidence sits in one session and Top-5 retrieval reaches it. Knowledge-update and temporal-reasoning lose more between S and M because both involve cross-session synthesis where Top-5 sometimes drops the second relevant chunk.

Mastra's gpt-5-mini configuration scores above their gpt-4o configuration

Mastra publishes 84.23% on LongMemEval-S with gpt-4o as the reader, and 94.87% with gpt-5-mini as the reader. The architectural reason for this ordering is that the gpt-5-mini configuration moves reasoning upstream of the reader, into ingest-time observers and reflectors.

In Mastra's gpt-4o-only configuration, the reader handles retrieval-context parsing, cross-session reasoning, and answer generation in a single query-time LLM call.

In Mastra's gpt-5-mini + Observational Memory configuration, the pipeline does additional ingest-time work. A gemini-2.5-flash "observer" runs over each session at ingest and extracts structured observation logs. A second gemini-2.5-flash "reflector" synthesizes the observations into long-term cross-session insights. At query time, the gpt-5-mini reader answers from the pre-distilled observation log plus reflections rather than raw chunks. The architecture is documented on Mastra's Observational Memory research page and described in VentureBeat's coverage.

The 94.87% is excluded from the comparison tables in this post for two reasons:

1. AgentOS at the same stack (gpt-5-mini reader + gemini-2.5-flash observer) on LongMemEval-S Phase A produced 70.4%, a 24-point gap from the published headline. The methodology disclosed on Mastra's research page does not contain enough detail for direct reproduction.

2. No confidence range is published on the 94.87%. Mastra's 84.23% gpt-4o headline falls inside the 95% range of AgentOS's 85.6% run [82.4%, 88.6%], and AgentOS with gpt-4o answering every question scored 84.2%; at this resolution the two are tied.

The matched cross-vendor comparison in Part 1 is AgentOS's gpt-4o-only run against Mastra's gpt-4o configuration.

Part 4: Reproducibility issues in the published vendor record

The patterns documented below are the methodology checks the agentos-bench harness applies before publishing any number, and the gaps between agentos-bench numbers and other vendors' published numbers.

LOCOMO answer-key and judge error rates

Penfield Labs (April 2026) audited LOCOMO and reported 99 errors in 1,540 answer-key entries (6.4% ground-truth error rate), and a 62.81% false-positive rate on LOCOMO's default LLM judge against an intentionally-wrong topically-adjacent answer set.

Implications:

• LOCOMO scores above 93.6% include benefit from answer-key errors.
• LOCOMO score differences below ~6 points are within the judge false-positive band.

For context, Northcutt et al. (NeurIPS 2021) found a 3.3% label-error rate sufficient to destabilize benchmark rankings.

LongMemEval-S overlap with current context windows

LongMemEval-S uses 115K tokens of conversation per question. GPT-4o (128K), Claude 3.5 (200K), Gemini 1.5 Pro (1M), and GPT-5 (400K) all hold the full S corpus in a single prompt.

Mastra's full-context baseline at gpt-4o is 60.20%; their Observational Memory configuration at the same model is 84.23%. The 24-point delta partly reflects token-level compression rather than retrieval-architecture quality.

The M variant exceeds every production context window, removing this confound.

Cross-vendor reimplementation discrepancies

In May 2025, Mem0 published a research paper positioning their product as state-of-the-art on LOCOMO. Their comparison table scored Zep at 65.99%. Zep responded, reran the evaluation with their own configuration, and reported 75.14% ±0.17 for Zep. Zep attributed the gap to Mem0 running Zep with sequential search instead of concurrent search.

Zep's self-reported LongMemEval-S number is 71.2% at gpt-4o, from their SOTA blog post. An independent reproduction at arXiv:2512.13564 measured Zep at 63.8%, a 7.4 points gap.

Notable methodology-disclosure findings

• EmergenceMem "Simple Fast" hardcodes top_k=42 in retrieval.
• Mastra's research page publishes 84.23% at gpt-4o; the observer and reflector models in the same configuration are gemini-2.5-flash (cross-provider).
• Mem0's research page reports 92.0% on LongMemEval; their research-2 page reports 93.4% on the same benchmark.
• MemPalace publishes 100% on every memory bench they touch because their evaluator is broken and returns 100% no matter the input. Not a competitor result. Public post-mortem: HackerNoon.

What competitors actually publish on 12 transparency axes

Judge FPR comparison

LOCOMO's default gpt-4o-mini judge measures at 62.81% FPR on the Penfield adversarial set. agentos-bench LOCOMO runs use gpt-4o-2024-08-06 with rubric 2026-04-18.1, which measures at 0% FPR on the same adversarial set.

Part 5: Reproducing both headlines

OpenAI and Cohere API keys are required. The setup command downloads the LongMemEval dataset from its upstream on Hugging Face.

LongMemEval-S 85.6% headline

1git clone https://github.com/framerslab/agentos-bench
2cd agentos-bench
3pnpm install && pnpm build
4node ./dist/cli.js setup --benchmark longmemeval-s
5
6# Set OPENAI_API_KEY and COHERE_API_KEY in your environment
7NODE_OPTIONS="--max-old-space-size=8192" node ./dist/cli.js run longmemeval-s \
8  --reader gpt-4o \
9  --memory full-cognitive --replay ingest \
10  --hybrid-retrieval --rerank cohere \
11  --embedder-model text-embedding-3-small \
12  --reader-router min-cost-best-cat-2026-04-28 \
13  --concurrency 5 \
14  --bootstrap-resamples 10000 \
15  --seed 42

Leave out --reader-router and the bench uses --reader for every case: that is the configuration behind the leaderboard's 84.2% gpt-4o-only run.

LongMemEval-M 70.2% headline

1node ./dist/cli.js setup --benchmark longmemeval-m
2
3NODE_OPTIONS="--max-old-space-size=8192" node ./dist/cli.js run longmemeval-m \
4  --reader gpt-4o \
5  --memory full-cognitive --replay ingest \
6  --hybrid-retrieval --rerank cohere --rerank-candidate-multiplier 5 \
7  --reader-top-k 5 \
8  --hyde \
9  --embedder-model text-embedding-3-small \
10  --reader-router min-cost-best-cat-2026-04-28 \
11  --concurrency 5 \
12  --bootstrap-resamples 10000 \
13  --seed 42

Both runs write a per-case run JSON under results/runs/. --seed 42 seeds the bootstrap resampling; without it the confidence ranges draw from Math.random. Cross-run comparison is possible against the leaderboard at results/LEADERBOARD.md.

Architecture

The AgentOS memory decomposition follows the CoALA framework (Sumers et al., 2023): explicit memory partitions and a decision-making module selecting a strategy per query. The MemoryRouter corresponds to the CoALA memory module; the reader router (selectReader) corresponds to the decision module.

The closest comparable architecture in the published record is Letta (formerly MemGPT, Packer et al., 2023), which models the LLM as a virtual operating system with paged memory. Letta has not published a LongMemEval number under the post-MemGPT branding.

Eight cognitive-memory mechanisms underlie the architecture (reconsolidation, retrieval-induced forgetting, involuntary recall, feeling-of-knowing, gist extraction, schema encoding, source confidence decay, emotion regulation), running on top of Ebbinghaus decay, and are documented with primary-source citations in the AgentOS Cognitive Memory docs.

Remaining headroom

Multi-session is the lowest per-category score on both variants. On M it measures at 48.9% (up from 29.3% under top-K=50), against 96.4% on single-session-assistant and 91.4% on single-session-user. On S it measures at 74.4% against the 85.6% aggregate. The MS category requires bridge queries across distinct sessions; pure retrieval-broadening does not close the gap.

Two mechanisms that target that gap are untested here:

1. Stage E: Hindsight 4-network typed-observer, adding a typed-graph signal orthogonal to BM25 + dense + Cohere rerank. Architecture follows Hindsight (vectorize.io, 2025).
2. K=V+fact key augmentation (Wu et al., Table 3 configuration): index sessions by raw content and extracted facts, with dual-key vector lookup. The Phase B --rerank-candidate-multiplier 10 ablation regressed -10.2 points on the same retrieval-heavy categories K=V+fact would affect, suggesting bounded expected lift.

Methodology disclosures

What's apples-to-apples in the comparisons above:

• The AgentOS run with gpt-4o answering every question (84.2%) uses the same reader as Mastra OM gpt-4o, Supermemory gpt-4o and EmergenceMem. The 85.6% headline routes four categories to gpt-5-mini.
• Same benchmark dataset (LongMemEval-S, 500 cases; LongMemEval-M, 500 cases).
• Same judge harness on every AgentOS run (gpt-4o-2024-08-06 with rubric 2026-04-18.1); judge false-positive rate 1% on S, 2% on M, 0% on LOCOMO.
• 95% confidence ranges at 10,000 resamples; most vendors don't publish ranges at all.

What isn't, with caveats:

• Judge methodology differs across vendors: each vendor's published number comes from its own evaluation.
• Cost and latency comparisons against Mastra, Supermemory, and EmergenceMem aren't directly measurable, because those vendors don't publish $/correct or per-case latency. The cost and latency numbers above are absolute: $0.0090 per correct on S, $0.0078 on M, p50 latency 3,558 ms on S.
• Mastra's 94.87% headline uses gpt-5-mini + gemini-2.5-flash observer. We can't reproduce it from their public methodology page, so it sits outside the Part 1 table.
• Mem0 v3's 93.4% is a managed-platform number with no published confidence range, no judge disclosure, and no reader disclosure. Their own State of AI Agent Memory 2026 post reports 66.9% on LOCOMO for the production stack.
• Hindsight's 91.4% uses gemini-3-pro as reader. Supermemory's 85.2% uses gemini-3-pro as reader. Both are cross-provider, so they sit outside the Part 1 table.
• Managed-platform numbers (Mastra, Mem0 v3, agentmemory) run on infrastructure with platform-specific optimizations that aren't necessarily portable.

Evaluating memory libraries

Three open-source benchmark harnesses cover the LongMemEval / LOCOMO space:

• agentos-bench ⁵: LongMemEval-S/M, LOCOMO, BEAM, and seven cognitive-mechanism micro-benchmarks. 95% confidence ranges, judge-adversarial probes, per-stage retention metric, and a --seed flag that makes the bootstrap ranges repeatable.
• Supermemory memorybench ⁶: LoCoMo, LongMemEval, ConvoMem against Supermemory, Mem0, and Zep with multi-judge support.
• Mem0 memory-benchmarks ⁷: LOCOMO and LongMemEval against Mem0 Cloud and OSS.

Reproducible memory benchmarks require a published seed, configuration, and per-case run JSONs alongside the headline number.

Closing

Two numbers end up here. 85.6% on LongMemEval-S at $0.0090 per correct, with gpt-5-mini answering four of the six categories; with gpt-4o answering everything, the same retrieval scores 84.2%, level with Mastra. 70.2% on LongMemEval-M at $0.0078 per correct, the only open-source library on the public record above 65% on the variant whose haystacks no production context window can absorb.

The intent of the design behind both numbers is not perfect recall. AgentOS implements Ebbinghaus decay and eight cognitive-science mechanisms on top of it, retrieval-induced forgetting and reconsolidation among them, so the agent generalizes from what it has seen rather than drowns in it. The benchmark numbers are the measurable part of that argument.

The runtime is Apache-2.0 at github.com/framerslab/agentos. The bench is at github.com/framerslab/agentos-bench. Reproducing the headlines is the setup and run commands above, on the dataset the LongMemEval upstream publishes.

FAQ

What's the difference between LongMemEval-S and LongMemEval-M?

S has 115K tokens of conversation per question and ~50 sessions per haystack: it fits in one gpt-4o call. M has 1.5M tokens per question and 500 sessions, exceeding every production LLM context window. S measures retrieval over a single-session-shaped corpus; M measures retrieval at scale where the reader can never see the whole haystack. AgentOS scores 85.6% on S and 70.2% on M.

What's the highest LongMemEval-S score anyone has claimed?

99% (Dhravya, as a gaming demonstration against the published bench rather than a reproducible architecture claim). Mastra publicly claims 95% but at a different (cheaper) answer LLM and with retrieval config that doesn't match the original paper's evaluation protocol. At the gpt-4o answer LLM, Mastra Observational Memory posts 84.23%, AgentOS posts 84.2%, and EmergenceMem Internal (closed-source SaaS) posts 86.0%. AgentOS's 85.6% headline adds a reader router that hands four of the six question categories to gpt-5-mini. Headline percentages that don't say which answer LLM produced them are pricing observations, not architecture claims. (MemPalace also publishes 100%, but their evaluator is broken in a way that returns 100% on any input, so it's not really in the conversation.)

Why publish 85.6% when others claim higher numbers?

Because the argument I care about is reproducibility, not headline percentage. Every number above comes with stated answer LLM, stated retrieval config, stated judge, a seed flag for the confidence ranges, a CLI to reproduce, and Apache-2.0 code. The 99% / 95% claims that beat AgentOS at face value miss at least one of those. The honest cost rule says I can't compare scores until those gaps close. If a competitor publishes a gpt-4o number tomorrow that beats our 84.2%, or a routed one that beats 85.6%, with the same disclosures, I'll cite them. That's the deal.

Is the AgentOS bench code public?

Yes. github.com/framerslab/agentos-bench, Apache-2.0. Includes the harness, the EmergenceMem Simple Fast adapter, the judge config, and the leaderboard with every reported headline. The per-case run JSONs are committed only as Git LFS pointers, without their objects.

What reader model does AgentOS use?

Both headlines route by question category. A gpt-5-mini classifier labels each question; gpt-4o answers the ones labeled temporal-reasoning or single-session-user, and gpt-5-mini answers the other four categories. In the S headline that sent 235 questions to gpt-4o and 265 to gpt-5-mini. With gpt-4o answering every S question, the same retrieval scored 84.2%. AgentOS exports the routing table and selectReader from @framers/agentos/memory-router, agentos-bench carries its own copy in src/core/readerRouter.ts, and Part 1 above has the calibration table.

What about Mem0's claimed numbers?

Mem0's published numbers sit in the vendor table and the methodology disclosures above, as their own pages report them; the reader model and config aren't always matched to the LongMemEval paper. agentos-bench has no Mem0 adapter, so none of those numbers is a rerun. The one vendor rerun in the bench is EmergenceMem Simple Fast.

How often are these numbers refreshed?

The numbers in this post are the leaderboard's as of the post's date, 2026-04-29. The bench is open; if you ship a memory library and want it in the comparison, open a PR with your adapter.

Further reading

• Full benchmarks reference: canonical comparison tables, methodology disclosure matrix, LOCOMO judge-FPR data.
• Memory Benchmark Transparency Audit: the broader transparency framework behind every number above.
• Cognitive Memory architecture (docs): the eight cognitive-memory mechanisms and the decay model behind AgentOS, with primary-source citations.
• agentos-bench v1 evaluation matrix: the per-cell comparison table.
• agentos-bench docs: the bootstrap and judge-adversarial methodology, and the April 2026 LongMemEval rollout writeup.

Built by Frame. AgentOS and agentos-bench are open source under Apache-2.0. GitHub · npm · Discord

References

Vendor research pages cited in the comparison table

The vendor table inline-links each vendor's own published research. Those source links remain inline (per-row attribution rather than prose claims). Canonical entries:

• Mem0 v3 / Mem0 OS research
• Mastra Observational Memory
• Hindsight (vectorize.io)
• Zep / Graphiti · Independent repro arXiv:2512.13564
• EmergenceMem
• Supermemory
• MemMachine · Memoria · agentmemory · Backboard · ByteRover · Letta · Cognee · AgentBrain