0.488 → 0.603 in One Day: Five Retrieval Lessons from Benchmarking Our Memory Engine
We ran MindTwin's memory engine on LoCoMo — the benchmark every AI memory vendor quotes — and scored below the published leaders. We published the number anyway. Then the diagnosis taught us more in one day than the previous month of building: about fact extraction, retrieval channels, negation, and a model that quietly invented dates.
Why we put a number on ourselves
"We have memory" is an unfalsifiable claim, and the AI memory space is full of it. So we ran the full LoCoMo suite — 10 very long two-person conversations (5,882 turns), 1,540 answerable questions — through our real product path: every turn ingested via the same pipeline customers use (safety gate included), answers and grading by gpt-4o, mirroring Mem0's public harness so the numbers would be as comparable as cross-harness numbers get.
Baseline result: 0.488 LLM-judge accuracy overall. Mem0 publishes ~0.64–0.67 on its own harness. We were meaningfully below the leaders, and the honest options were: don't mention it, or diagnose it in public. This post is the second option.
Lesson 1 — Split your misses before you fix anything
One diagnostic changed everything. We conditioned the judge score on whether the gold evidence actually made it into the retrieved context:
| Judge accuracy | |
|---|---|
| When the evidence was retrieved (69.5% of questions) | 0.647 |
| When it wasn't (30.5%) | 0.126 |
Read that top row again: 0.647 — at Mem0's published overall — whenever retrieval succeeded. Our answering stack was already at parity. The entire gap was retrieval coverage. Without this split we might have spent weeks tuning prompts on the wrong side of the pipeline.
Lesson 2 — Raw turns lose to facts, structurally
Why did retrieval miss 30% of the time? Because we stored raw conversation turns. A question like "Where did Caroline move from?" has to find one specific turn among ~600. Systems that score higher on LoCoMo extract facts at ingest: "Caroline moved from Sweden in 2019" exists as a first-class, dated, standalone memory — nearly trivial to retrieve.
So we built ingest-time fact extraction in a day: a batched LLM pass that distills new turns into standalone dated facts, stored alongside the raw turns as their own memory type — semantic-rate decay, never filed cold, fail-safe (no LLM key means you just get raw-turn behavior). Re-run:
| Category | Baseline | With facts |
|---|---|---|
| single-hop (n=841) | 0.530 | 0.666 |
| temporal (n=321) | 0.601 | 0.688 |
| multi-hop (n=282) | 0.262 | 0.376 |
| open-domain (n=96) | 0.396 | 0.427 |
| Overall (n=1,540) | 0.488 | 0.603 |
The mechanism worked exactly where predicted: on questions whose evidence retrieval used to miss entirely, accuracy jumped from 0.126 to 0.458 — the facts channel carries answers raw-turn search can't find. (And the safety gate stayed free: zero crisis false-positives across all 5,882 emotionally varied turns, both runs.)
Lesson 3 — New memory types need their own retrieval channel
Our first attempt at this hurt one category badly. Blended into a single similarity ranking, the distilled facts — clean, dense, on-topic — crowded the raw turns out of the top-k entirely. Multi-hop recall collapsed to zero: the model got beautiful summaries and none of the scattered supporting turns it needed to chain across.
The fix is architectural, not a tuning knob: query facts and raw turns as separate channels and give the answerer both — facts for what's true, receipts for the details. Memory types aren't a taxonomy; they're retrieval architecture.
Lesson 4 — Models will invent dates unless you make it impossible
Days later, while capturing real product output, our extractor confidently produced a fact dated "October 3, 2023" — a date that appears nowhere in the source. The cause: chat messages (Slack, and LoCoMo turns without our benchmark preprocessing) don't carry dates in their text, and an extractor asked for "dated facts" will helpfully fabricate one.
The fix has two halves, and you need both: stamp every turn with its true timestamp from metadata before extraction, and instruct the model to include a date only when it's stated or computable — never guessed. After the fix, facts came out like "On 16 February 2026, Dave called the renewal 'a no-brainer'" — real dates, load-bearing for every temporal question that followed.
Lesson 5 — Negation breaks similarity retrieval
Building our own benchmark (TWIST) surfaced the subtlest one. To check whether a draft message like "heard you're not doing pottery anymore" contradicts the record, you must retrieve the positive statements it contradicts ("pottery class was amazing!"). But the negated sentence embeds far from the positive one — cosine similarity actively works against you precisely when contradiction-checking matters most.
Our mitigation: extract the topics a draft makes claims about ("pottery hobby") and retrieve per topic rather than by whole-draft similarity. It helps; it doesn't fully solve it. We think negation-aware retrieval is an open problem for every memory system doing consistency checking, and we said so in TWIST's public results — along with a mechanism we shipped that didn't work.
The honest scoreboard
The remaining gaps have names: multi-hop needs cross-session fact linking, not just extraction; and our full methodology, per-question outputs, and the reproduce-it-yourself harness ship with the product. If a number can't be re-run, it's marketing.