GoodMem · Benchmark
Memory That Pays for Itself
Cutting LLM token burn on a production enterprise BI agent — without trading away answer quality
PAIR Systems engineering ·
- Aggregate token burn
- −28%
- 95% CI 2–47% · significant
- Agent reasoning steps
- −23%
- 95% CI 5–37% · significant
- Change in answer quality
- ≈ 0
- paired Δ, interval spans 0
Enterprise AI agents are expensive because they re-derive context on every call — searching, inspecting schemas, previewing data, and reasoning from scratch on questions the system has effectively solved before. In a controlled A/B benchmark on a production enterprise BI agent, GoodMem's memory and retrieval layer cut aggregate token consumption 28% (95% CI 2–47%) and agent reasoning iterations 23% (95% CI 5–37%), with no measurable change in answer quality (paired Δ ≈ 0; 95% CI −23.5 to +23.5). On the heaviest, retrieval-intensive queries — the ones that dominate the bill — token reductions reached 30–66%.
Headline results
| Metric | Memory OFF | Memory ON | Change | 95% CI |
|---|---|---|---|---|
| Aggregate token burn (17 scenarios) | 6.92M | 4.99M | −28% | 2–47% |
| Median tokens per query | 318K | 150K | −53% | — |
| Agent iterations (aggregate) | 216 | 167 | −23% | 5–37% |
| Agent iterations (median / query) | 11 | 7 | −36% | — |
| Answer quality (judge pass rate) | 46% | 46% | ≈ 0 pts | −23.5, +23.5 |
17 matched scenarios · 3 replicates / condition (57 scored runs per arm) · identical frontier agent model and independent LLM judge in both arms · the only variable changed is GoodMem memory · intervals are nonparametric bootstrap estimates (10,000 resamples, clustered by scenario).
The problem: agents pay full price for every question
Agents that answer analytical questions over enterprise data are token-hungry by construction: every query reruns semantic search, schema inspection, data preview, SQL generation, and self-correction, filling the context window to rediscover the same tables, joins, and pitfalls it navigated yesterday. At scale, production BI agents now generate six-figure monthly token bills that grow with adoption rather than amortizing across it.
The approach: serve memory, don't re-derive it
GoodMem is a self-hostable memory and retrieval control plane for enterprise AI. It captures the durable lessons of prior conversations — which business views to use, how a metric is derived, which joins and filters are correct, what pitfalls to avoid — and prefetches only the relevant guidance into each turn. The agent starts informed instead of from zero and converges in fewer steps, with no change to its model, prompts, or logic.
Where the savings live
Savings are not uniform — and that is the point. They concentrate on the expensive, retrieval-heavy queries that drive the bill: multi-table aging, cross-module joins, and year-over-year roll-ups saw 30–66% fewer tokens, while trivial lookups changed little. Memory savings track spend — 10 of 17 scenarios consumed fewer tokens, with the largest gains exactly where enterprise agent cost is largest.
What it means for cost
Illustratively, a deployment running a $100K / month token bill — concentrated, as bills are, on heavy queries — would recover roughly $28K / month, about $340K / year, from a 28% aggregate reduction with no change to the agent. Because savings scale with the retrieval-heavy traffic that grows fastest, the benefit compounds as adoption rises.
Quality, held constant
Cost reduction only matters if quality holds. It did: the paired difference in judge pass rate was indistinguishable from zero, with a confidence interval centered on it. Memory changed how efficiently the agent worked, not whether it was right — fewer steps, fewer tokens, the same answers.
Why PAIR Systems
GoodMem is built by PAIR Systems, which develops the information-retrieval layer for enterprise AI — bringing techniques once limited to specialized search teams into any product as self-hostable software. Its founder helped pioneer practical zero-shot neural retrieval at Google and productized enterprise retrieval at Vectara. For the heaviest token spenders, GoodMem extends into model distillation and self-tuning that compound the gains shown here.
Methodology. Controlled, paired A/B on a production enterprise BI agent. 17 matched scenarios across accounts-payable, sales, inventory and multi-step drill-down workloads; 3 scored replicates per scenario per condition. Identical frontier agent model and an independent LLM judge across both arms; the sole intervention is GoodMem's memory layer (prefetch of distilled prior-conversation guidance). Confidence intervals are nonparametric bootstrap estimates (10,000 resamples, clustered by scenario). Infrastructure failures (provider rate-limits, transport faults) are excluded as non-quality events. Pilot-scale benchmark; effect sizes are large and directional within the stated intervals; broader multi-tenant validation in progress. © 2026 PAIR Systems, Inc.
Want us to follow up?
This report is free to read and share — no sign-up, no wall. If you’d like to discuss a benchmark on your workload, deployment, pricing, or a security review, leave a few details and our team will contact you.