​

Search Mode Evaluation Methodology

​

1. What this measures and what it doesn’t

• Your specific brain content (this is a benchmark, not your bill).
• Your specific query distribution.
• End-user satisfaction or downstream task success.
• Latency under concurrent load.
• Production cost (the cost numbers are model-pricing estimates × dataset size, not your actual API spend).

​

2. Datasets and sizes

• LongMemEval — public split, n=500 questions. Downloaded from Hugging Face. The corpus + answer keys are pinned to a specific commit; recorded in every per-run record.
• Replay captures — NDJSON from the sibling modusbrain-evals repo, n=200 queries. Each query carries a retrieved_slugs baseline + a latency_ms measurement from the original production run.
• BrainBench v1 — n=1240 documents / n=350 qrels (binary relevance judgments). Lives in the sibling modusbrain-evals repo, SHA-pinned at every run.

​

3. Sample selection

• Random seed: 42 throughout. Set via --seed N on modusbrain eval run-all; recorded in every per-run record.
• No per-question curation. Splits are taken whole; no question is filtered for reporting.
• No mode-specific tuning. The same dataset + same seed feeds every mode. The mode is the only independent variable.
• Stability across re-runs: with --seed 42 and the same dataset SHA, two runs of the same (mode, suite) produce identical retrieval orderings (modulo the optional Haiku expansion call, which is non-deterministic). Persisted in eval_results so anyone can re-score from the committed dumps.

​

4. Run procedure

# Setup: in your modusbrain working tree, with OPENAI_API_KEY + ANTHROPIC_API_KEY exported.
git rev-parse HEAD  # record the commit for the methodology footer

# Sweep all 3 modes × 2 retrieval-focused suites with seed 42.
modusbrain eval run-all \
  --modes conservative,balanced,tokenmax \
  --suites longmemeval,replay \
  --seed 42 \
  --limit 500 \
  --budget-usd-retrieval 5 \
  --budget-usd-answer 20 \
  --output docs/eval/results/v0.32.3/

# Render the comparison.
modusbrain eval compare --md > docs/eval/results/v0.32.3/README.md
modusbrain eval compare --json > docs/eval/results/v0.32.3/comparison.json

​

5. Threats to validity

• LongMemEval skews English + technical. The questions are software-engineering and consumer-product flavored. Performance on a brain rich in non-English / non-technical content (writing, art history, etc.) may differ.
• BrainBench is small (1240 docs) relative to a production brain (10K-100K pages). Absolute scores aren’t predictive of your hit rate; the delta between modes is.
• char/4 token heuristic. Token-budget enforcement and cost estimates use a character-count / 4 heuristic. Accurate within ~5-10% for English with the OpenAI tiktoken family; off worse for Voyage (we don’t use Voyage in chat retrieval, so it doesn’t bias the reported numbers, but if you do, your budget caps will be approximate).
• Expansion’s quality lift varies by query distribution. The eval data shows ~97.6% relative quality with LLM expansion vs without (i.e., barely measurable lift) on the LongMemEval corpus. On rarer-entity / longer-tail queries, the lift can be larger. We report the corpus we measured; YMMV.
• Paired bootstrap assumes question-level independence. Multi-hop questions within the same conversation thread aren’t independent; the bootstrap CI is slightly tighter than reality.
• Single brain instance per benchmark. The benchmark spins up an in-memory PGLite per question. Cache hit rate measured here doesn’t reflect a long-running production brain’s cache state.

​

6. Per-question raw outputs

• Wins: the 3 questions where this mode’s score exceeded the next-best mode by the largest margin.
• Losses: the 3 questions where this mode’s score fell short of the next-best mode by the largest margin.

​

7. Pre-registered expectations

1. tokenmax wins Recall@10 by 5-15 percentage points over conservative. LLM expansion + 50-result ceiling helps rare-entity surface forms.
2. conservative wins cost-per-query by 5-15× over tokenmax. No Haiku expansion + tight 4K budget cap = single-digit-cent queries.
3. balanced lands within 3pp of tokenmax on Recall@10. Intent weighting (zero-LLM cost) closes most of the expansion gap on common queries.
4. No mode breaks nDCG@10 ≥ 0.65 — the published “ship it” threshold for hybrid retrieval on technical corpora.

​

8. Re-run cadence

• src/core/search/**
• src/core/embedding.ts
• src/core/chunkers/**
• src/core/ai/recipes/anthropic.ts
• src/core/ai/recipes/openai.ts
• src/core/operations.ts

​

Statistical-significance discipline

• Paired bootstrap with 10,000 resamples per metric. Each resample draws question-level pairs (same question, mode A vs mode B), so question-level variance is differenced out.
• Bonferroni correction across the 12 comparisons (3 modes × 4 metrics). The reported p-value is the comparison’s raw p-value × 12 (clamped at 1.0).
• 95% confidence intervals computed from the bootstrap distribution.

​

Glossary

​

Cost anchors

• T = avg tokens per search-result chunk. The recursive chunker targets 300 words / chunk → ~400 tokens (English, OpenAI tiktoken approx).
• N = chunks delivered per query (capped by the mode’s searchLimit).
• R = downstream model input rate. Sonnet 4.6 = $3/M. Opus 4.7 = $5/M. Haiku 4.5 = $1/M.
• Q = queries per month.

• Query embedding (text-embedding-3-large @ $0.13/M tokens): ~$0.00001 per query. Negligible at every scale.
• Tokenmax Haiku expansion call ($1/M input, $5/M output, ~500 input + 200 output per call): ~$0.0015 per query, or $150/mo at 100K queries. Cache hits cut this in half.
• Per-page indexing (one-time): bounded by your import volume, not query volume. Not modeled here.

• Your agent’s system prompt + reasoning tokens add input that modusbrain doesn’t see.
• Compaction reduces input over a long session.
• Most agents make 1-5 searches per turn; cost-per-turn is what bills you, not cost-per-query.
• The model price column drifts as providers reprice; pin the rate via src/core/model-pricing.ts (the canonical chat-pricing table) for a current snapshot.

​

Mode × Model matrix (the 25x spread)

• tokenmax + Haiku: Haiku gets 20K of search results stuffed into its context per query. Haiku’s reasoning is weaker; more chunks = more noise, not more signal. You pay Haiku rates but get sub-Haiku quality. Wrong direction.
• conservative + Opus: Opus has 200K context window and can synthesize across many chunks. Capping at 10 chunks / 4K tokens leaves Opus reasoning underfed. You pay Opus rates but get conservative-shape retrieval. Wasted spend.

• Haiku struggles past ~5-10 chunks of cross-referenced content → conservative
• Sonnet handles ~25-40 chunks well → balanced
• Opus benefits from 50+ chunks for multi-hop reasoning → tokenmax

​

Realistic-scale anchor (single power-user agent loop)

• history + reasoning + search), not just modusbrain’s chunks. Mode choice contributes ~10-20% on top of that base.

1. At realistic agent-loop scale with disciplined prompt caching, mode choice saves 10-20% of total agent spend — meaningful, but smaller than the per-query 5x ratio implies. Disciplined prompt-cache layouts blunt the mode delta because most of the per-turn cost is the cached prefix, not the search payload.
2. Without that prompt-cache discipline, the per-query framing reasserts itself. Setups that churn the prompt prefix on every turn (frequent system-prompt edits, untemplated tool defs, no prompt-cache structuring) see search payload contribute a much larger fraction of total cost. Those setups should care about mode choice more, not less.
3. The cache hit rate quoted here (~88%) is achievable but not automatic. It requires structuring the prompt so the cached prefix stays stable across turns: system prompt + tool defs first, history compacted but cache-aware, retrieved chunks appended LAST (where their volatility doesn’t invalidate the prefix). Agents that interleave search results inside the cached region pay the prefix-rebuild tax on every turn.

• The anchor represents ONE power-user loop. Multi-user fleets aggregate proportionally; the per-user shape doesn’t change.
• The “3 searches per turn” assumption varies wildly. A code-review agent might issue 10+ searches per turn; a chat-only loop might do 0.
• The 88% cache hit rate is the high end of what’s achievable. Half that is closer to a default agent without cache-aware prompt layout.
• The “Δ vs tokenmax” math assumes the OTHER cost components (system, tools, history, reasoning) stay constant. In practice, conservative’s smaller per-turn payload also leaves more room in the context window for history → which can change agent behavior in either direction.

​

Reproducibility footer

• Code commit SHA
• Dataset SHA (LongMemEval, BrainBench, Replay)
• --seed N
• Run commands verbatim
• API model identifiers used (Anthropic + OpenAI + judge model)