TWIST: A Benchmark for What Memory Benchmarks Can't See

Every AI memory vendor quotes a LoCoMo score. LoCoMo measures whether a system can recall what was said. Nothing measures whether it notices when what was said stops being true. We built that benchmark — and our own product fails it. Here's the spec, the data, and the honest first results.

The failures that actually cost money

Deployed memory systems don't lose money on forgotten facts. They lose it at change points — the moments a stored belief and reality diverge:

• A customer's champion says the renewal is "a no-brainer" in February and "we're evaluating a competitor" in June. Nobody connects the two statements. A lost renewal.
• A teammate drafts a cheerful message that contradicts what the counterparty actually said. An unforced, trust-destroying error.
• A superseded fact — "we'll renew at current pricing" — keeps answering questions as if it were current. Decisions on stale data.
• A customer's phone number, or a colleague's hardest moment, becomes recallable to anyone who asks. A failed security review, or real harm.

Several vendors ship features aimed at these behaviors — conflict detectors, temporal fact invalidation, draft vetting. None of these capabilities has a public benchmark. Every claim in this space is currently unfalsifiable — precisely the condition LoCoMo fixed for recall. TWIST (Tracking Within-speaker Inconsistencies, Stance-reversals, and Tensions) proposes to fix it for coherence, alignment, and governance.

Four tracks, one rule: every metric ships with its pair

The pairing is the point. A detection system that flags everything aces recall and is unusable; a safety gate that suppresses everything leaks nothing and remembers nothing. TWIST's usage terms make citing one number without its pair invalid.

TWIST extends LoCoMo rather than replacing it: items are injected into LoCoMo's own conversations, so one corpus yields both a recall score and a TWIST profile. And it is designed against LoCoMo's documented weaknesses — the ~6.4% answer-key error rate and the LLM judge that accepted up to 63% of deliberately wrong answers in an independent audit — with double annotation, structural evidence-ID scoring, judge decoy calibration reported alongside every run, and versioned errata.

What a scenario looks like

The full catalog — 26 scenario families across the four tracks, each detection family paired with a surface-matched hard negative — ships with the benchmark.

First results: neither system passes, in opposite directions

We built Track B first: 153 draft-vetting items over LoCoMo's ten conversations, every item gated by an independent verification pass. Two systems ran it — a flat-RAG baseline (embed, retrieve, ask an LLM) and MindTwin's own vetting path:

Flat-RAG cries wolf: it catches everything and objects to two out of three perfectly safe messages. Deployed, users would mute it in a week. MindTwin stays silent too often: it never over-flags, but at ten-conversation scale it missed most true contradictions — half on retrieval coverage (a draft saying "you're not doing pottery anymore" embeds far from the positive statements it contradicts), and much of the rest, it turned out, on our own items: gold evidence whose standalone text didn't carry the contradiction. Both findings are in the public results, including the mechanism we shipped that didn't work.

That's the standard we're committing to. A benchmark authored by a vendor is only credible if it reports the vendor's failures with the same precision as its wins — and if competitors can run it themselves and dispute every item.

Run it, break it, dispute it

Any memory system that can wrap five calls — ingest, flag_tensions, recall, check_alignment, resolve — can run TWIST, and may decline tracks it doesn't support. The v0.1 spec, the scenario catalog, the Track B item set, and the raw per-item results are downloadable below. If you build memory systems: run it, tell us where the items are wrong, and we'll version the errata publicly. If it doesn't separate systems, it doesn't ship as v1 — that's pre-registered.