TWIST: A Benchmark for What Memory Benchmarks Can't See
Every AI memory vendor quotes a LoCoMo score. LoCoMo measures whether a system can recall what was said. Nothing measures whether it notices when what was said stops being true. We built that benchmark — and our own product fails it. Here's the spec, the data, and the honest first results.
The failures that actually cost money
Deployed memory systems don't lose money on forgotten facts. They lose it at change points — the moments a stored belief and reality diverge:
- A customer's champion says the renewal is "a no-brainer" in February and "we're evaluating a competitor" in June. Nobody connects the two statements. A lost renewal.
- A teammate drafts a cheerful message that contradicts what the counterparty actually said. An unforced, trust-destroying error.
- A superseded fact — "we'll renew at current pricing" — keeps answering questions as if it were current. Decisions on stale data.
- A customer's phone number, or a colleague's hardest moment, becomes recallable to anyone who asks. A failed security review, or real harm.
Several vendors ship features aimed at these behaviors — conflict detectors, temporal fact invalidation, draft vetting. None of these capabilities has a public benchmark. Every claim in this space is currently unfalsifiable — precisely the condition LoCoMo fixed for recall. TWIST (Tracking Within-speaker Inconsistencies, Stance-reversals, and Tensions) proposes to fix it for coherence, alignment, and governance.
Four tracks, one rule: every metric ships with its pair
| Track | Capability | Paired failure it scores |
|---|---|---|
| A — Tension detection | Flag a stance reversal unprompted, months apart | False-flagging nostalgia, hypotheticals, pressure signals |
| B — Draft alignment | Vet an outgoing message against the record, with citations | Flagging drafts that merely acknowledge a change |
| C — Belief supersession | Answer with the current fact; retire the stale one | Deleting history, or retiring facts that still coexist |
| D — Safe recall | PII and crisis content governed at write time | Over-blocking emotional-but-legitimate content |
The pairing is the point. A detection system that flags everything aces recall and is unusable; a safety gate that suppresses everything leaks nothing and remembers nothing. TWIST's usage terms make citing one number without its pair invalid.
TWIST extends LoCoMo rather than replacing it: items are injected into LoCoMo's own conversations, so one corpus yields both a recall score and a TWIST profile. And it is designed against LoCoMo's documented weaknesses — the 6.4% answer-key error rate and the 62.8% deliberately-wrong-answer acceptance found by an independent audit of the LLM-judge evaluation convention ("LoCoMo-J") that memory-system papers popularized on top of the original F1-scored benchmark — with double annotation, structural evidence-ID scoring, judge decoy calibration reported alongside every run, and versioned errata.
What a scenario looks like
The full catalog — 26 scenario families across the four tracks, each detection family paired with a surface-matched hard negative — ships with the benchmark.
First results: neither system passes, in opposite directions
We built Track B first: 153 draft-vetting items over LoCoMo's ten conversations, every item gated by an independent verification pass. Two systems ran it — a flat-RAG baseline (embed, retrieve, ask an LLM) and MindTwin's own vetting path:
| Metric | MindTwin | flat-RAG |
|---|---|---|
| Caught true contradictions (n=93) | 0.14 | 1.00 |
| Didn't flag aligned drafts (n=41) | 0.98 | 0.78 |
| Didn't flag hard negatives (n=19) | 1.00 | 0.37 |
Flat-RAG cries wolf: it catches everything and objects to two out of three perfectly safe messages. Deployed, users would mute it in a week. MindTwin stays silent too often: it never over-flags, but at ten-conversation scale it missed most true contradictions — half on retrieval coverage (a draft saying "you're not doing pottery anymore" embeds far from the positive statements it contradicts), and much of the rest, it turned out, on our own items: gold evidence whose standalone text didn't carry the contradiction. Both findings are in the public results, including the mechanism we shipped that didn't work.
That's the standard we're committing to. A benchmark authored by a vendor is only credible if it reports the vendor's failures with the same precision as its wins — and if competitors can run it themselves and dispute every item.
Run it, break it, dispute it
Any memory system that can wrap five calls — ingest, flag_tensions, recall, check_alignment, resolve — can run TWIST, and may decline tracks it doesn't support. The v0.1 spec, the scenario catalog, the Track B item set, and the raw per-item results are downloadable below. If you build memory systems: run it, tell us where the items are wrong, and we'll version the errata publicly. If it doesn't separate systems, it doesn't ship as v1 — that's pre-registered.
Get the benchmark (v0)
- github.com/subratpanda/twist-benchmark — the canonical repo: spec, dataset, results, reference code. Star it, run it, file item disputes as issues.
- White paper — task definitions, metrics, validation protocol
- Scenario catalog — 26 families with examples and hard negatives
- Track B item set — 153 verified items (JSONL, gold evidence ids)
- Track B first results — including the negative result