Standalone benchmark suite for retrieval quality evaluation

[bm-clawd]

Summary

Create a standalone benchmark suite (basic-memory-bench) for evaluating retrieval quality across BM deployments using academic datasets. Designed to be publicly shareable, runnable by anyone, and integrated into CI.

Why

1. Internal quality tracking — benchmark before/after every release
2. Cloud vs Local comparison — validate that Cloud's better embeddings produce better retrieval
3. Public credibility — reproducible numbers on academic benchmarks
4. Marketing — "we benchmark in the open" content
5. Competitive positioning — compare against Mem0/Supermemory on the same datasets

Current Results (from prototype in openclaw-basic-memory plugin)

Full LoCoMo benchmark — 1,982 queries across 10 conversations:

Metric	BM Local (v0.18.5)
Recall@5	76.4%
Recall@10	85.5%
MRR	0.658
Content Hit Rate	25.4%
Mean Latency	1,063ms

By category:

Category	N	R@5
open_domain	841	86.6%
multi_hop	321	84.1%
adversarial	446	67.0%
temporal	92	59.1%
single_hop	282	57.7%

Architecture

• Python (not TS) — same ecosystem as BM, uses BM's importer framework
• Provider abstraction — BM Local (MCP stdio), BM Cloud (API), Mem0 (optional)
• Two eval modes — retrieval metrics (R@K, MRR) + LLM-as-Judge (for Mem0 comparison)
• Deterministic conversion — LoCoMo/LongMemEval → BM markdown via EntityMarkdown
• CI-ready — just full-locomo runs everything, fail if recall drops >2%

Datasets

• LoCoMo (ACL 2024, Snap Research) — 10 conversations, 1,986 QA pairs. Mem0 publishes numbers on this.
• LongMemEval (ICLR 2025) — Supermemory uses this. More challenging.
• Synthetic (our hand-crafted 38-query suite for CI smoke tests)

Known Improvement Opportunities

From analysis of the 375 failures:

1. RRF scoring flattens results (#577) — hybrid search scores all ~0.016, ranking destroyed. FTS alone finds observations that hybrid misses.
2. Single-hop recall at 57.7% — specific fact lookups need better chunk matching
3. Temporal at 59.1% — date-aware scoring needed

Implementation Phases

1. Repo setup + LoCoMo (Python port from current TS prototype)
2. BM Cloud provider + LongMemEval dataset
3. LLM-as-Judge + competitor comparison
4. CI integration + public results dashboard

Reference

• Full spec: drafts/spec-benchmark-suite.md in openclaw workspace
• Current TS prototype: benchmark/ in openclaw-basic-memory repo
• LoCoMo dataset: github.com/snap-research/locomo
• LongMemEval: github.com/xiaowu0162/LongMemEval
• Supermemory's harness: github.com/supermemoryai/memorybench

[claude]

Claude finished @bm-clawd's task in 1m 47s —— View job

---

Issue Triage

• Gather context and analyze issue
• Check for duplicate issues
• Apply labels
• Provide triage summary

---

Analysis

Type: Feature Request — New standalone benchmark suite package (basic-memory-bench)

No duplicates found. The repo has existing internal benchmarks (test-int/test_sync_performance_benchmark.py, test-int/semantic/) but nothing targeting academic retrieval quality datasets like LoCoMo or LongMemEval.

Labels applied: enhancement (already present) + cloud (cloud vs local comparison is core to the proposal)

---

Triage Summary

Dimension	Assessment
Type	Feature request — new tooling/package
Priority	Medium — valuable for quality tracking & marketing, not blocking users
Component	Testing (benchmark suite) + Cloud (provider comparison)
Complexity	Complex — multi-phase, new package structure, dataset integrations, optional provider abstraction

Strengths of this proposal:

• Clear business motivation (internal QA, cloud validation, public credibility, competitive positioning)
• Working TypeScript prototype already exists with real numbers (76.4% Recall@5 on LoCoMo)
• Well-structured 4-phase implementation plan
• Directly surfaces known issues: BUG: Hybrid RRF fusion dilutes vector scores with weak FTS scores basic-memory#577 (RRF score flattening), single-hop recall, temporal queries

Implementation considerations:

1. Standalone repo vs. test-int/ addition — The issue proposes a separate basic-memory-bench package. Given the existing test-int/semantic/ infrastructure, it's worth considering whether Phase 1 (LoCoMo) could live in test-int/ first before splitting to a standalone repo. This would reuse existing metrics.py, report.py, and conftest patterns.

2. Dataset licensing — LoCoMo (Snap Research, ACL 2024) and LongMemEval (ICLR 2025) need license review before bundling or auto-downloading in CI.

3. CI gate — "fail if recall drops >2%" requires storing a baseline. Recommend storing baselines as JSON fixtures committed to the repo rather than computed dynamically.

4. Phase ordering — The known issues (#577 RRF scoring, single-hop recall) should ideally be fixed before locking in baseline numbers, otherwise the baseline captures known-broken behavior.

Suggested next steps:

1. Fix BUG: Hybrid RRF fusion dilutes vector scores with weak FTS scores basic-memory#577 (RRF scoring) first — it directly impacts the benchmark numbers
2. Port LoCoMo evaluation to test-int/ using existing metrics.py infrastructure (validates dataset integration before creating a new package)
3. Then extract to standalone basic-memory-bench once the eval loop is validated

[bm-clawd]

Reference Implementation

The prototype benchmark harness, LoCoMo converter, and competitive analysis are in the openclaw-basic-memory plugin repo:

• Harness: benchmark/run.ts — MCP stdio client, scoring, per-category breakdown
• LoCoMo converter: benchmark/convert-locomo.ts — JSON → BM markdown notes
• Competitive comparison: benchmark/COMPETITIVE.md — MemMachine, Mem0, Zep, etc.
• Standalone suite spec: benchmark/SPEC.md — Full architecture for Python port
• Synthetic corpus: benchmark/corpus/ + benchmark/queries.json — 11 files, 38 queries

Quick start (to reproduce LoCoMo results)

cd openclaw-basic-memory
# Download LoCoMo dataset
mkdir -p benchmark/datasets
curl -sL https://raw.githubusercontent.com/snap-research/locomo/main/data/locomo10.json -o benchmark/datasets/locomo10.json

# Convert to BM markdown
bun benchmark/convert-locomo.ts

# Index a conversation
bm project add locomo-0 $(pwd)/benchmark/corpus-locomo/conv-0
bm reindex --project locomo-0

# Run benchmark
bun benchmark/run.ts --project=locomo-0 --queries=benchmark/corpus-locomo/conv-0/queries.json --corpus=locomo-0

The Python port should use BM's importer framework (src/basic_memory/importers/) for the converter, particularly the EntityMarkdown + MarkdownProcessor pattern from the ChatGPT importer.

[bm-clawd]

LLM-as-Judge Evaluation via Pydantic Evals

For direct comparison with Mem0/MemMachine published numbers, we need an LLM-as-Judge step. Pydantic Evals is the right tool — same ecosystem as BM (PydanticAI), case-specific rubrics, Logfire observability for free.

Pipeline

Retrieve (existing harness) → Generate Answer (LLM) → Judge (Pydantic Evals) → Aggregate

1. Retrieve — search BM for relevant context given a query (what we have now)
2. Generate — feed retrieved context + question to LLM, produce an answer (GPT-4o-mini at temp 0, matching Mem0's config)
3. Judge — Pydantic Evals LLMJudge compares generated answer to ground truth
4. Aggregate — weighted average across categories (matching LoCoMo scoring protocol)

Implementation sketch

from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import LLMJudge

# Per-query case with ground truth
cases = []
for q in locomo_queries:
    cases.append(Case(
        name=q["id"],
        inputs=q["query"],
        metadata={"category": q["category"], "ground_truth": q["ground_truth"]},
        evaluators=[
            LLMJudge(
                rubric=f"The answer must be semantically equivalent to: {q['expected_answer']}\n"
                       f"PASS if the core facts match. FAIL if the answer is wrong, "
                       f"missing key information, or hallucinates facts not in the ground truth.",
                include_input=True,
            ),
        ],
    ))

# Universal evaluator for all cases
dataset = Dataset(
    cases=cases,
    evaluators=[
        LLMJudge(
            rubric="The response directly answers the question using information "
                   "from the provided context. FAIL if the response says it cannot "
                   "answer when the information was available.",
            include_input=True,
        ),
    ],
)

async def bm_answer(query: str) -> str:
    """Retrieve from BM, then generate answer."""
    context = await bm_search(query, limit=5)
    return await llm_generate(query, context, model="gpt-4o-mini", temperature=0)

report = await dataset.evaluate(bm_answer)
report.print()

Why Pydantic Evals

• Case-specific rubrics — each LoCoMo query gets a rubric tailored to its ground truth answer, not a generic "is this good?" check
• Same ecosystem — BM already uses PydanticAI. No new dependencies.
• Logfire tracing — free observability on every judge call
• Reasoning included — every judge verdict comes with explanation, crucial for debugging failures
• Deterministic + LLM hybrid — run format checks first, save LLM evaluation for semantic quality

Matching competitor methodology

Mem0 uses GPT-4o-mini as judge with binary 0/1 scoring (same/not same). We should match this exactly for fair comparison, then optionally add richer rubrics for our own analysis.

Reference: https://pydantic.dev/articles/llm-as-a-judge