Benchmark: Verify LoCoMo category assignments match source code, not paper

[bm-clawd]

Context

MemMachine's benchmark blog discovered that LoCoMo category assignments in the paper differ from the source code:

'This finding suggests that some public LoCoMo results might be presenting misclassified data, making a direct and fair comparison challenging.'

They use the source code assignments as ground truth, not the paper's descriptions.

Action

1. Compare our category assignments against the LoCoMo source code (github.com/snap-research/LoCoMo)
2. Document any discrepancies with the paper
3. Ensure our per-category results use the correct assignments
4. If our categories were wrong, re-run and report corrected numbers

This is important for credibility — if we publish numbers with wrong categories, competitors will call it out.

Related

• Benchmark: Add LLM-as-Judge evaluation (GPT-4.1) for LoCoMo #9, Benchmark: Adopt Backboard's LoCoMo methodology for reproducible comparison #8
• MemMachine blog: memmachine.ai/blog/2025/12/memmachine-v0.2-delivers-top-scores-and-efficiency-on-locomo-benchmark/

Milestone

v0.19.0

[claude]

Claude encountered an error —— View job

---

I'll analyze this and get back to you.