Skip to content

Add agent memory benchmark - #362

Open
ichko wants to merge 23 commits into
redis:mainfrom
ichko:add-agent-memory-benchmark
Open

Add agent memory benchmark#362
ichko wants to merge 23 commits into
redis:mainfrom
ichko:add-agent-memory-benchmark

Conversation

@ichko

@ichko ichko commented Aug 31, 2026

Copy link
Copy Markdown

Note

Low Risk
Adds an isolated benchmark package and documentation/CI; it does not change V0 server code. Operational risk is limited to optional external API credentials when running provider adapters locally or in CI beyond unit tests.

Overview
Introduces agent-memory-benchmark/, a new Python package with the memory-bench CLI for reproducible LongMemEval v1 runs: load public splits, ingest sessions per example with isolated user_ids, wait for provider extraction where needed, answer with a shared OpenAI step, and write resumable artifacts (answers.jsonl, metadata.json, errors.jsonl). A separate judge command applies task-specific LLM rubrics and writes metrics.json (including task-averaged accuracy).

The harness ships nine MemoryStore adapters (Redis Agent Memory REST, Mem0, LangMem, Zep, Graphiti, Supermemory, Vertex Memory Bank, Bedrock AgentCore, Oracle Agent Memory), optional uv extras, provider recipes under docs/providers/, and pytest coverage for CLI, datasets, judge, and runner resume/redaction behavior. GitHub Actions runs ruff, pytest, and uv build in that directory on push/PR.

Root landing narrative is updated so the repo is explicitly two projects: V0/ as the preserved research server and the benchmark as an open audit surface (not a leaderboard). README.md, AGENTS.md, CLAUDE.md, and V0/README.md emphasize Redis Agent Memory in Iris as the supported product path with clearer quickstart links.

Reviewed by Cursor Bugbot for commit 92b7be7. Bugbot is set up for automated code reviews on this repo. Configure here.

ichko and others added 4 commits August 31, 2026 14:03
Keep Agent Memory Server history and files at the repository root. The public benchmark package lives in a subdirectory so the two trees can ship together.

Co-authored-by: Cursor <cursoragent@cursor.com>
Landing copy now leads with the managed Iris product and points at the
open-source foundation and benchmark without implying they replace it.

Co-authored-by: Cursor <cursoragent@cursor.com>
The vendor classes lived in one providers.py file behind re-export stubs.
Moving them into the named modules, with shared answering helpers in common.py,
makes it easier to find and change a single provider.

Co-authored-by: Cursor <cursoragent@cursor.com>
Neither exposes a Python API this harness can call, so the classes only ever
raised. Carrying unrunnable providers, docs, and a test costs more than it
explains.

Co-authored-by: Cursor <cursoragent@cursor.com>
Comment thread agent-memory-benchmark/src/agent_memory_benchmark/memory/mem0_store.py Outdated
Comment thread agent-memory-benchmark/src/agent_memory_benchmark/memory/graphiti_store.py Outdated
Comment thread agent-memory-benchmark/src/agent_memory_benchmark/benchmark/runner.py Outdated
Comment thread README.md Outdated
Comment thread agent-memory-benchmark/README.md Outdated
Comment thread agent-memory-benchmark/docs/providers/README.md

@tylerhutcherson tylerhutcherson left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks promising. A few of the cursor bug notes might be worth addressing. I also commented on some small readme improvements. Seems like we are still missing the redis implementation afaik

The public Iris REST API is now a first-class provider so runs can target
Redis Agent Memory with an endpoint, store id, and API key.

Co-authored-by: Cursor <cursoragent@cursor.com>
Comment thread agent-memory-benchmark/src/agent_memory_benchmark/prompts/judge.py Outdated
ichko and others added 2 commits September 2, 2026 02:47
Vendor wrappers now match the locked Mem0, Vertex, Graphiti, Zep, and Oracle
APIs. Ingest retries reset first, and per-example user ids cannot be overridden.

Co-authored-by: Cursor <cursoragent@cursor.com>
Stop the judge from crashing on braces, avoid hanging LTM deletes, and clear AgentCore events between examples. Also put Redis first in the harness provider lists.

Co-authored-by: Cursor <cursoragent@cursor.com>
Comment thread agent-memory-benchmark/src/agent_memory_benchmark/memory/base.py Outdated
Comment thread agent-memory-benchmark/src/agent_memory_benchmark/datasets/longmemeval.py Outdated
Comment thread agent-memory-benchmark/LICENSE Outdated
ichko and others added 2 commits September 2, 2026 11:15
Do not query until memory counts stop growing, write answers only after reset succeeds, re-download a truncated LongMemEval cache, and put Redis on the Apache appendix.

Co-authored-by: Cursor <cursoragent@cursor.com>
Match the requested root preamble and make the Iris adapter the first registered provider.

Co-authored-by: Cursor <cursoragent@cursor.com>
Comment thread agent-memory-benchmark/src/agent_memory_benchmark/memory/zep_store.py Outdated
Stop treating a capped search page as extraction complete, walk Supermemory document pages, and stamp haystack dates on Zep, LangMem, and Vertex ingest.

Co-authored-by: Cursor <cursoragent@cursor.com>
ichko and others added 2 commits September 2, 2026 13:26
Wait on real record lists for Bedrock, Graphiti, and Oracle, and stop coercing store_id or project numbers from CLI params.

Co-authored-by: Cursor <cursoragent@cursor.com>
Comment thread agent-memory-benchmark/src/agent_memory_benchmark/datasets/longmemeval.py Outdated
Comment thread agent-memory-benchmark/src/agent_memory_benchmark/memory/base.py Outdated
ichko and others added 2 commits September 4, 2026 11:45
Avoid holding the medium JSON in an HTTP buffer, delete Redis sessions by owner with pagination, and treat an unstable extraction wait as a timeout instead of a scored answer.

Co-authored-by: Cursor <cursoragent@cursor.com>
A failed bulk delete now raises so leftover owner memories cannot mix into the next ingest.

Co-authored-by: Cursor <cursoragent@cursor.com>
Comment thread agent-memory-benchmark/src/agent_memory_benchmark/memory/base.py
Zep and Bedrock inherit a 30-minute wait instead of 120 seconds, and Mem0 ingest includes conversation dates so temporal questions can be answered.

Co-authored-by: Cursor <cursoragent@cursor.com>
Bedrock, LangMem, and Supermemory were calling APIs with the wrong kwargs or a 100-item cap; resume also replaced wall time instead of adding it.

Co-authored-by: Cursor <cursoragent@cursor.com>
Sync adapters were polling as if extraction were async, and Bedrock/Oracle dropped the conversation-date line other providers send.

Co-authored-by: Cursor <cursoragent@cursor.com>
Those adapters extract asynchronously, so the runner must wait; other answering stores still return list_memories immediately.

Co-authored-by: Cursor <cursoragent@cursor.com>
ichko and others added 2 commits September 4, 2026 16:34
Co-authored-by: Cursor <cursoragent@cursor.com>
Ruff wanted a shorter Redis delete path check and formatted three files; typos rejected the unparseable metrics key.

Co-authored-by: Cursor <cursoragent@cursor.com>
Comment thread agent-memory-benchmark/src/agent_memory_benchmark/memory/vertex_memory_bank.py Outdated
Unscoped list plus dict equality missed Memory Bank rows, so later examples could see leftover memories.

Co-authored-by: Cursor <cursoragent@cursor.com>

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

Reviewed by Cursor Bugbot for commit af09623. Configure here.

The SDK cleanup path is delete_thread, so reset was a no-op when delete_user was missing and leftover memories leaked across examples.

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants