An Archive-Grounded Benchmark for Agentic Workplace Document Reasoning
AGORA is a benchmark for archive-grounded agentic workplace document reasoning, with 362 questions across eight domains, 9,664 documents and 372M tokens. Agents search a fixed document collection, combine evidence across files, and return a numerical answer.
Requirements: Python 3.11+, uv, Docker and an E2B account.
git clone https://github.com/KYLN24/Agora.git
cd Agora
uv sync --project evaluation --lockedThe dataset is hosted on Hugging Face. Download and extract it with:
uv run scripts/download_data.pyThe script writes questions and gold answers to data/tasks/ and documents to data/documents/. Use --revision to select a dataset version or --no-extract to download only the archive.
Each task also includes reference_reasoning_path and reference_files. Reference file paths are relative to data/, with optional PDF page ranges.
All models use the same evaluation protocol:
| Setting | Value |
|---|---|
| Evidence scope | Documents from the task's paired domain |
| Agent harness | mini-swe-agent with a bash tool |
| Internet access | Disabled in the sandbox |
| Interaction limit | 200 turns per task |
| Sandbox timeout | 3,600 seconds per task |
| Temperature | 1.0 |
| Reasoning effort | Highest level supported by the model endpoint |
| Scoring | Normalized exact answer match |
Build the baseline sandbox and E2B template after downloading the data. Log in to your container registry and set E2B_API_KEY, then run:
bash evaluation/e2b_template/build.sh --image YOUR_REGISTRY/agora-eval:latestThe script builds and pushes the image, then creates the agora-eval template. See build options for local image builds and existing images. Run the evaluation with:
cd evaluation
uv run --locked src/run_eval.py \
--run-id baseline \
--model PROVIDER/MODEL \
--template agora-eval \
--judge exact \
--step-limit 200 \
--sandbox-timeout 3600 \
--model-kwarg temperature=1.0Pass the endpoint's highest supported reasoning setting with --model-kwarg; the parameter name and value depend on the endpoint. Use --limit 1 for a smoke run. Results are saved to evaluation/runs/<run-id>/. See evaluation options for reasoning parameters, cost and loop limits, and the OpenHands and OpenClaw adapters.
The dataset download already contains the prepared documents. To process source documents yourself, use the parsing script. It converts PDFs with dots.ocr at 200 DPI into five-page Markdown windows, copies Markdown, text, Excel and CSV files unchanged, and records failed pages in parse_report.jsonl.
For PDF input, first install dots.ocr and start its model server as described in the parsing setup. From the repository root, run:
pip install -r scripts/document_parsing/requirements.txt
python scripts/document_parsing/parse_documents.py /path/to/source_documents /path/to/parsed_documentsThe output must be a new directory outside the input. Use --ocr-host and --ocr-port for a server other than localhost:8000, or --dots-ocr-source /path/to/dots.ocr to load the parser from a source checkout.
The code is licensed under the MIT License. The dataset and underlying source documents retain their own usage terms.
@article{guo2026agora,
author = {Honglin Guo and Qi Zhang and Yu Zhang and Weijie Li and Rui Zheng and Zhikai Lei and Qiyuan Peng and Zhiheng Xi and Tao Gui and Qi Zhang},
title = {{AGORA:} An Archive-Grounded Benchmark for Agentic Workplace Document
Reasoning},
journal = {CoRR},
volume = {abs/2606.24526},
year = {2026},
url = {https://doi.org/10.48550/arXiv.2606.24526},
doi = {10.48550/ARXIV.2606.24526},
eprinttype = {arXiv},
eprint = {2606.24526},
timestamp = {Thu, 09 Jul 2026 07:25:37 +0200},
}