Benchmarks

Measured results, with the method shown.

How fast and how accurate M.N.E.M.O. retrieval is, and how that holds up as memory grows. For why it works this way, read the architecture.

The data structure on its own

OAT lookup time as memory grows

Measured separately from the product benchmark further down this page: the OAT library itself in Java 21, three axes (Topic × TimeRange × Speaker), one memory per unique combination, looked up by its full intersection key. Each figure is the median of six runs, and each run takes the median of 500 batches of 1,000 random lookups. The linear scan searches the same memories one by one for a randomly chosen target.

Memories	OAT lookup	Linear scan	OAT advantage
1,000	60 ns	15 µs	~260×
5,000	190 ns	130 µs	~670×
20,000	270 ns	0.79 ms	~3,000×
50,000	470 ns	2.7 ms	~5,600×

The lookup stays under half a microsecond at 50,000 memories. Its small rise with size comes from CPU cache misses as the index outgrows the processor cache, not from extra work: with the same keys held in cache, a lookup takes 30–60 ns at every size. The scan's cost grows with the data.

Benchmarked, not just argued

OAT vs. vector search vs. GraphRAG, measured

These are real numbers from mnemo-benchmark, measured on a synthetic benchmark dataset — 25 hand-authored queries across five complexity categories over 101 records, a controlled comparison rather than a large-scale production sample. At this size, differences that only appear as memory grows can't show up here; the data-structure measurement above shows how OAT behaves as memory grows. All three retrieval approaches run against the same dataset: OAT through a real, unmodified MNEMO agent (not a reference implementation), vector search through real OpenAI embeddings and brute-force cosine similarity, GraphRAG through real LLM entity/relationship extraction and graph traversal. Every synthesized answer uses the same LLM for all three systems, so the numbers below isolate retrieval quality, not model quality.

Pure retrieval speed is measured separately, with no LLM in the loop, in the data-structure comparison further down (warmed up, averaged over 50 runs, in nanoseconds). The numbers here are full turns. Inside a turn, MnemoRAG's retrieval step also parses the chosen path and renders the results as text, so it takes longer than the bare OAT lookup. The LLM step that decideswhere to look is real, necessary work the other two systems' simpler indexes skip, and it's counted in the full turn, not hidden.

Across these 25 queries, MnemoRAG's average full turn of 1.7s breaks down into 43ms of path selection (the LLM reading the Memory Path index), 5ms of retrieval, and 1.6s of answer writing. Most of MnemoRAG's time goes to path selection, not to OAT.

Methodology: what each metric measures

Accuracy grades the full pipeline's final synthesized answer against ground truth — not retrieval in isolation. A wrong or incomplete retrieval fails accuracy too, since the LLM can't correctly answer from data it never got.

Groundedness is graded independently of correctness: does the synthesized answer's claims trace to what was actually retrieved? A confidently wrong answer built from mis-retrieved data can still score as grounded by this definition — it's checking traceability, not correctness. The two numbers are meant to be read together, not collapsed into one.

Hallucination rate is not simply 100% minus the average groundedness score above, and the two aren't expected to sum to 100% — they're computed differently. Groundedness is a continuous score (0–1) averaged across every query. Hallucination is a per-query flag, true whenever that query's individual groundedness score falls below 0.8 — so a batch of queries clustered just above that threshold can pull the average groundedness score down further than the hallucination rate alone would suggest.

The correctness judge sees the question, the reference answer, and the candidate answer — not the retrieved content. The groundedness judge sees the candidate answer and the retrieved content — not the reference answer. Both use the same fixed judge model and prompt applied uniformly across all three systems, so no system is graded more leniently than another.

On Multi Axis Pivot and Aggregation queries, the number itself comes from OAT's accumulate, which computes the exact total or average directly. The LLM only phrases the final sentence around that value; it never adds up the figures itself.

Timing resolution. Full-turn times are recorded in whole milliseconds. Pure retrieval is timed separately in nanoseconds, warmed up and averaged, in the data-structure comparison.

Primary comparison

End-to-end time by query type

Retrieval + reasoning + answer synthesis combined — the number a user actually waits for.

MnemoRAGVector RAGGraphRAG

Content shape and confidence

Accuracy and refusal rate by document format

Structured records, unstructured prose, multi-axis pivot tables, and flat lookup tables — the same three systems, four content shapes. Every LLM call in this benchmark, judge included, uses the same model: Claude Haiku 5.5. Each format category is a small, single-digit set of queries drawn from the 25-query dataset above — read the percentages below as directional signal, not statistically robust point estimates.

A wrong answer can mean two very different things: the system invented a fact, or it retrieved nothing and honestly said so. Refusal rate below is the second case — a per-query judge flag for an honest "I don't know" or "not found," independent of whether the question even had an answer available. It is not the same chart as hallucination rate (fabrication), which stays near-zero across every format shown here — see "Reading these numbers" further down for that number. A low accuracy bar paired with a high refusal bar means the system is failing honestly, by retrieving badly, not by making things up.

The Vector arm is a deliberately plain baseline: the top 5 chunks of 800 characters by cosine similarity, no re-ranking, no hybrid search. That is why it scores 0% on pivot tables: matching a transaction ID such as TXN-00151 is not something embeddings do well, and a total or average over hundreds of rows cannot fit in five chunks. A tuned retrieval stack would do better; this chart shows what the plain approach does.

Accuracy

MnemoRAGVector RAGGraphRAG

Refusal rate

MnemoRAGVector RAGGraphRAG

Gloves off — no LLM, no agent layer

Pure data-structure performance: OAT vs. Vector vs. GraphRAG

Every chart above still includes MnemoRAG's LLM path-selection cost — real, necessary work for a product, but it obscures what OAT the data structure actually costs to query. Below, every system's one-time LLM/ embedding call (deciding where to look) happens once and is excluded from the timed loop; the native data-fetch itself then repeats 60 times per query with zero further API calls, and both speed and accuracy are measured directly against the same ground truth used everywhere else on this page.

This is deliberately not an apples-to-apples operation, and no system's JSON or text rendering is timed (JSON exists only because an LLM reads and writes text; it is not part of any data structure). OAT's request is resolved to tree objects beforehand, then the timed work is the native lookup: an intersection-index lookup when the path names several axes, a walk of the path's subtree when it names one, or a scan and aggregate over the matching leaves for sums and averages. Vector's native operation is a full brute-force cosine-similarity scan over its entire indexed corpus, because that scan is what vector search is. GraphRAG's is a linear substring scan over every known entity name plus a capped tabular-row scan. Comparing these numbers compares what each data structure actually costs to query on its own terms. Aggregation has to read every matching value, so its cost follows the number of matching records; OAT reaches those records through an inverted index rather than scanning the tree.

The first 10 of the 60 runs per query are discarded as warm-up. Times are shown in nanoseconds so the sub-millisecond differences between the three are visible.

Speed (no LLM)

OATVector RAGGraphRAG

Accuracy (no LLM)

OATVector RAGGraphRAG

Overall: OAT 15,634 ns / 100% accurate · Vector 1.82 ms / 40% accurate · GraphRAG 163,124 ns / 75% accurate. Semantic Fuzzy queries are excluded — no deterministic ground truth to grade without an LLM judge.

View full numbers — every query type, every system

Query type	System	Full turn	Accuracy	Groundedness
Exact Lookup	MnemoRAG	1.4s	100%	100%
	Vector RAG	1.6s	0%	100%
	GraphRAG	1.6s	100%	95%
Multi Axis Pivot	MnemoRAG	1.4s	100%	100%
	Vector RAG	2.5s	0%	100%
	GraphRAG	1.6s	0%	99%
Aggregation	MnemoRAG	2.6s	100%	91%
	Vector RAG	3.8s	40%	100%
	GraphRAG	13.6s	100%	90%
Entity Relationship	MnemoRAG	1.3s	100%	100%
	Vector RAG	2.0s	100%	100%
	GraphRAG	1.5s	100%	100%
Semantic Fuzzy	MnemoRAG	1.7s	100%	98%
	Vector RAG	1.9s	100%	97%
	GraphRAG	2.0s	80%	100%
Overall	MnemoRAG	1.7s	100%	98%
	Vector RAG	2.4s	48%	99%
	GraphRAG	4.0s	76%	97%

Reading these numbers

Faster and more accurate.

There are two different clocks here. With no LLM in the loop, OAT is the faster data structure in every category measured: an exact lookup takes 3,204 ns against GraphRAG's 157,793 ns (49× faster), a multi-axis pivot 9× and an aggregation 4×. In a full turn, MnemoRAG skips its LLM path-selection call when the question names its data (an ID, a customer and a product, or the values that fix an aggregate), so exact lookups (1.4s against GraphRAG's 1.6s), multi-axis questions (1.4s against 1.6s), aggregations (2.6s against 13.6s), entity-relationship questions (1.3s against 1.5s) and semantic questions (1.7s against 2.0s) are all at or ahead of GraphRAG. On aggregation the lead is largest because OAT computes the total itself while GraphRAG's LLM adds up the rows. Every question in this benchmark names the records it is about, so this is the fast case: a question that describes a problem without naming the customer or the product still needs the LLM to decide where to look, and takes a few seconds longer.

The speed does not cost any accuracy. On accuracy, MnemoRAG leads (100% vs. GraphRAG's 76%). On hallucination, MnemoRAG flagged 4% of its answers and GraphRAG 0% in this run — a per-query flag from a judge model (see Methodology), so treat small differences as noise. Over the whole benchmark a full MnemoRAG turn averages 1.7s against GraphRAG's 4.0s, with the higher accuracy.

Next step

See why retrieval works this way, or talk with Panaxon AI.

Get API access Read the architecture