GoodMem reproduces RAW-RAG’s strong results
GoodMem reproduced RAW-RAG’s LoCoMo results and scored higher on MemoryAgentBench using the paper’s embedding and reranking models.
In a LoCoMo conversation dated May 8, 2023, Melanie shows Caroline a painting of a lake at sunrise. She says she painted it “last year,” and Caroline compliments the colors. Asked later when she made it, a model has to connect those two words to the date of the conversation. A memory that says only that Melanie paints would be accurate, but would have discarded the answer.
Zhengye Bao and colleagues examine this problem in Is Complex Agentic Memory Always Needed? Naive RAG Is Stronger Than We Thought. Their baseline, RAW-RAG, retrieves passages from the original history and uses a reranker to choose which ones the answering model sees. Across the paper’s evaluations, it ranked first among the compared memory systems in six of seven combinations of benchmark and reader model.
In a MemoryAgentBench experiment, the authors replaced raw history with extracted facts. Overall accuracy fell from 48.32 to 28.70. Extracting facts can preserve individual statements while losing the context that connects them.
We repeated the LoCoMo experiment with GoodMem supplying the context. Each
conversation went into its own space, with the original turns and dates
preserved. We used OpenAI’s text-embedding-3-small and Cohere Rerank-v3.5,
matching the models in the paper’s table 10, and kept the same reader and
grading prompts. GoodMem retrieved 128 candidate chunks and reranked them to
15; we included neighboring turns before passing the passages to the reader.
GoodMem’s chunking and index differ from the paper’s implementation, as
documented in the
methodology.
GoodMem reproduced RAW-RAG’s strong results on the 1,540 non-adversarial questions, scored by an LLM judge:
Results across rerankers
Accuracy (%)
Select a row to inspect a run. Intervals use GoodMem’s question-level bootstrap; the paper’s scores are shown as published.
Both published scores fall within our 95% bootstrap intervals. We also tried other rerankers: the four configurations sharing the same embedding model and retrieval depth spanned 0.21 points with GPT-4o-mini and 0.25 with GPT-4.1-mini.
On MemoryAgentBench, the same OpenAI and Cohere models gave GoodMem a score of 52.07, compared with RAW-RAG’s 48.32. GPT-4o-mini answered all 2,071 questions, again using 128 candidates reranked to 15. GoodMem scored higher in each of the four capability groups.
Higher scores in all four groups
Difference from RAW-RAG · points
GPT-4o-mini · one complete run.
Overall averages the four capability scores; source weighting follows the paper’s appendix D.5.
Here the choice of retrieval models made a larger difference. Compared with Voyage-4-large and Rerank-2.5, OpenAI and Cohere scored 7.80 points lower on EventQA but 17.00 higher on fine-grained TREC classification. Removing reranking cost RAW-RAG 3.92 points in the paper’s ablation.
The repository contains the prompts and per-question answers. For the lake sunrise, GPT-4.1-mini answered 2022, citing Melanie’s May 8 remark that she had painted it the year before.