Skip to content
KYLN24Public

About

AGORA is a benchmark for archive-grounded agentic workplace document reasoning, with 362 questions across eight domains, 9,664 documents and 372M tokens. Agents search a fixed document collection, combine evidence across files, and return a numerical answer.

Resources

Stars

2 stars

Watchers

0 watching

Forks

Latest commit

 

History

1 Commit

Folders and files

Repository files navigation

AGORA

An Archive-Grounded Benchmark for Agentic Workplace Document Reasoning

arXiv Dataset Code Code License

AGORA is a benchmark for archive-grounded agentic workplace document reasoning, with 362 questions across eight domains, 9,664 documents and 372M tokens. Agents search a fixed document collection, combine evidence across files, and return a numerical answer.

Setup

Requirements: Python 3.11+, uv, Docker and an E2B account.

git clone https://github.com/KYLN24/Agora.git
cd Agora
uv sync --project evaluation --locked

Download data

The dataset is hosted on Hugging Face. Download and extract it with:

uv run scripts/download_data.py

The script writes questions and gold answers to data/tasks/ and documents to data/documents/. Use --revision to select a dataset version or --no-extract to download only the archive.

Each task also includes reference_reasoning_path and reference_files. Reference file paths are relative to data/, with optional PDF page ranges.

Evaluation

All models use the same evaluation protocol:

Setting Value
Evidence scope Documents from the task's paired domain
Agent harness mini-swe-agent with a bash tool
Internet access Disabled in the sandbox
Interaction limit 200 turns per task
Sandbox timeout 3,600 seconds per task
Temperature 1.0
Reasoning effort Highest level supported by the model endpoint
Scoring Normalized exact answer match

Build the baseline sandbox and E2B template after downloading the data. Log in to your container registry and set E2B_API_KEY, then run:

bash evaluation/e2b_template/build.sh --image YOUR_REGISTRY/agora-eval:latest

The script builds and pushes the image, then creates the agora-eval template. See build options for local image builds and existing images. Run the evaluation with:

cd evaluation
uv run --locked src/run_eval.py \
  --run-id baseline \
  --model PROVIDER/MODEL \
  --template agora-eval \
  --judge exact \
  --step-limit 200 \
  --sandbox-timeout 3600 \
  --model-kwarg temperature=1.0

Pass the endpoint's highest supported reasoning setting with --model-kwarg; the parameter name and value depend on the endpoint. Use --limit 1 for a smoke run. Results are saved to evaluation/runs/<run-id>/. See evaluation options for reasoning parameters, cost and loop limits, and the OpenHands and OpenClaw adapters.

Document parsing

The dataset download already contains the prepared documents. To process source documents yourself, use the parsing script. It converts PDFs with dots.ocr at 200 DPI into five-page Markdown windows, copies Markdown, text, Excel and CSV files unchanged, and records failed pages in parse_report.jsonl.

For PDF input, first install dots.ocr and start its model server as described in the parsing setup. From the repository root, run:

pip install -r scripts/document_parsing/requirements.txt
python scripts/document_parsing/parse_documents.py /path/to/source_documents /path/to/parsed_documents

The output must be a new directory outside the input. Use --ocr-host and --ocr-port for a server other than localhost:8000, or --dots-ocr-source /path/to/dots.ocr to load the parser from a source checkout.

License

The code is licensed under the MIT License. The dataset and underlying source documents retain their own usage terms.

Citation

@article{guo2026agora,
  author       = {Honglin Guo and Qi Zhang and Yu Zhang and Weijie Li and Rui Zheng and Zhikai Lei and Qiyuan Peng and Zhiheng Xi and Tao Gui and Qi Zhang},
  title        = {{AGORA:} An Archive-Grounded Benchmark for Agentic Workplace Document
                  Reasoning},
  journal      = {CoRR},
  volume       = {abs/2606.24526},
  year         = {2026},
  url          = {https://doi.org/10.48550/arXiv.2606.24526},
  doi          = {10.48550/ARXIV.2606.24526},
  eprinttype   = {arXiv},
  eprint       = {2606.24526},
  timestamp    = {Thu, 09 Jul 2026 07:25:37 +0200},
}

About

AGORA is a benchmark for archive-grounded agentic workplace document reasoning, with 362 questions across eight domains, 9,664 documents and 372M tokens. Agents search a fixed document collection, combine evidence across files, and return a numerical answer.

Resources

Stars

2 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages