An argument about agent memory with receipts
An agent’s memory should be a folder you own.
Swivel Labs is a small lab that has run a fleet of AI agents every day since 22 February 2026. Keeping them coherent turned out to be mostly a memory problem, so we built Anamnesis, the memory engine they run on.
Anamnesis top-ten recall
82.5%
of forty verified questions found the right memory in the top ten, up from 27.5%, with the same embedding model.
Caveat. The forty are private, drawn from our own fleet’s memory, and the same forty chose the settings, so read the after-figure as optimistic.
Measured 11 October 2026.
Our answer is plain. The memories are markdown files we own and can read, and every search index is built from them, so any index can be deleted and rebuilt without putting a single memory at risk. Below is the part with numbers: what we measured, the four things we fixed, and the fix that made things worse.
---
id: 1385439425adc63f
kind: thread
salience: 1.0
---
# anamnesis upgrade: measured first, then fixed
**The answer to "should we swap the embedder?"** is no, and now there's a number behind it.The measurement 11 October 2026Measured first, then fixed
On 11 October the question on the table was whether to switch to a newer embedding model. Before changing anything, we built a test: forty questions, each with one known answer, a specific line we had checked at its source. The questions cover every agent and every month of the record, and most of them are paraphrases, which is how you actually ask for something you half remember. A result only counted if the memory that came back contained that exact line. Nobody judged anything by hand.
of the forty found the right memory in the top ten, up from 27.5%. The right memory came first 55% of the time, up from 10%. About two seconds per recall once the models are loaded. Same embedding model in both runs.
| Measure | Before | After |
|---|---|---|
| Right memory ranked first | 10% | 55% |
| Right memory in the top ten | 27.5% | 82.5% |
| Mean reciprocal rank | 0.143 | 0.651 |
| Answers buried mid-document, in the top ten | 23.5% | 82.4% |
| Seconds per recall, models loaded | 3.7 | ≈ 2 |
So the answer to the embedding question was no. The model was never the bottleneck. Four other things were.
The reranker had never worked. The quantized model we had installed shipped with every quantized weight set to zero, so it gave every candidate the same score. Its checksum matched the publisher’s. This was the file as published. We replaced it with one that passed a check for dead weights.
Keyword search needed every word. It only returned documents containing all the words of a question, and real questions rarely match like that. Matching any term, across every kind of record, took that search’s top-ten rate from 0.20 to 0.60 on its own.
Meaning search saw each file through a keyhole. The embedder was shown the first 1,500 and the last 1,000 characters of a file. A line in the middle of a long document did not exist for it.
Part of the record was on a machine nobody mirrored. Since 11 October it is copied over read-only, and the search covers it.
One fix made things worse. Splitting documents into passages should cure the keyhole. But with the passage index 15.8% built, turning it on made the first result worse: right-first fell from 55% to 45%, because a partial index tilts which memories get considered at all. So passage vectors shipped behind a gate: they serve only once the index is at least 95% built and that night’s run of the forty questions scores no worse with them on.