🏆 BracketRank

Large Language Model Document Ranking via Reasoning-based Competitive Elimination

Abdelrahman Abdallah · Mohammed Ali · Bhawna Piryani · Adam Jatowt

BracketRank is a zero-shot document reranker that organizes candidates into a reasoning-driven tournament. It adaptively forms groups that fit an LLM context window, ranks each group with explicit relevance reasoning, splits documents into winner and loser tracks, and repeatedly reranks paired groups until a global order is produced.

✨ Highlights

• 🧠 Reasoning-enhanced ranking: the LLM explains relevance judgments before producing each permutation.
• ⚖️ Fair competition: winner and loser tracks reduce sensitivity to a document's initial position.
• 📐 Adaptive grouping: the number of groups is derived automatically from the candidate count and context limit.
• ⚡ Parallel stages: independent groups and bracket matches can be processed concurrently.
• 🔌 Provider-friendly: works with OpenAI and OpenAI-compatible chat-completions endpoints.
• 📊 Reproducible evaluation: includes the TREC DL 19/20 and eight BEIR configurations used for the main results.
• 🧩 Three research strategies: adaptive BracketRank, fixed-group tournament, and hierarchical ranking.
• 🔀 Two execution modes: deterministic normal scheduling or concurrent independent LLM calls.

🏗️ How it works

For the paper setting of 100 candidates and a maximum group size of 20:

100 BM25 candidates
        │
        ▼
5 balanced groups × 20 documents
        │  reasoning-based listwise ranking
        ▼
top halves ──► winner track ──► paired elimination ──┐
bottom halves ► loser track  ──► paired elimination ──┤
                                                    ▼
                                      final ranking (1–100)

Each bracket round concatenates two competing groups, reranks their documents with the same reasoning prompt, and advances the resulting group. An unpaired group receives a bye. The final winner-track order is followed by the final loser-track order.

🚀 Installation

git clone https://github.com/DataScienceUIBK/BracketRank.git
cd BracketRank
python -m venv .venv
source .venv/bin/activate
pip install -e .

Set credentials using environment variables—never place keys in source files:

export OPENAI_API_KEY="your-api-key"

For an OpenAI-compatible server, also set its base URL:

export OPENAI_BASE_URL="http://localhost:8000/v1"

⚡ Quick start

The input is JSON Lines: one query per line, with a hits list containing docid and content. See examples/sample.jsonl.

bracketrank \
  --input examples/sample.jsonl \
  --output outputs/sample.reranked.jsonl \
  --model gpt-4 \
  --method adaptive \
  --execution parallel \
  --top-k 100 \
  --max-group-size 20 \
  --max-workers 10

The output preserves the input schema and replaces hits with the BracketRank order. Documents beyond --top-k remain in their original order.

🧩 Ranking strategies

1. Adaptive BracketRank — main paper method

Automatically computes the initial group count as ceil(top_k / max_group_size):

bracketrank --input input.jsonl --output output.jsonl \
  --method adaptive --max-group-size 20 --execution parallel --max-workers 10

2. Fixed tournament

Uses an explicit number of balanced initial groups, followed by the same winner/loser competition:

bracketrank --input input.jsonl --output output.jsonl \
  --method tournament --initial-groups 4 --execution normal

3. Hierarchical ranking

Repeatedly partitions and reranks the full candidate list using progressively smaller group sizes. The original research defaults are 25,12,6:

bracketrank --input input.jsonl --output output.jsonl \
  --method hierarchical --levels 25,12,6 --execution parallel --max-workers 10

Normal versus parallel execution

Option	Behaviour
--execution normal	Runs independent group and match calls sequentially (max_workers=1).
--execution parallel	Runs independent calls in the same stage concurrently, up to --max-workers.

Both modes use the same prompts and ranking algorithms. Parallel execution preserves group order, but it can reach API rate limits more quickly.

Python API

from bracketrank import BracketRanker
from bracketrank.clients import OpenAIChatClient

client = OpenAIChatClient(model="gpt-4")
ranker = BracketRanker(
    client,
    top_k=100,
    max_group_size=20,
    max_passage_words=300,
    max_workers=10,
)

reranked_documents = ranker.rerank(query, documents)

# Other research strategies:
fixed = ranker.rerank(query, documents, method="tournament", initial_groups=4)
hierarchical = ranker.rerank(query, documents, method="hierarchical", levels=(25, 12, 6))

Each document must be a dictionary with at least these fields:

{"docid": "document-id", "content": "document text", "score": 12.34}

Additional fields are preserved.

📊 Reproducing the main results

Install the evaluation dependencies:

pip install -e '.[evaluation]'

Run the published BracketRank-20 setting (top_k=100, max_group_size=20, 300 words per passage, temperature 0):

bracketrank-evaluate \
  --dataset dl19 \
  --model gpt-4 \
  --method adaptive \
  --execution parallel \
  --top-k 100 \
  --max-group-size 20 \
  --max-passage-words 300 \
  --max-workers 10

Replace dl19 with any supported benchmark:

Family	--dataset values
TREC Deep Learning	dl19, dl20
BEIR	covid, nfc, touche, dbpedia, scifact, signal, news, robust04

The evaluator downloads the corresponding Pyserini prebuilt BM25 index, retrieves 100 candidates, runs BracketRank, and writes:

• outputs/<dataset>.trec — standard TREC run file
• outputs/<dataset>.metrics.json — NDCG, MAP, and recall at 1, 5, and 10

Before paying for a full run, verify your environment and endpoint on one query:

bracketrank-evaluate --dataset dl19 --model gpt-4 --max-queries 1 --max-workers 2

Important

Exact numerical reproduction requires access to the same GPT-4 model snapshot/deployment used in the paper. Provider-side model updates can change rankings even with temperature 0. API rate limits may require lowering --max-workers.

Paper configuration

Parameter	Main setting
First-stage retrieval	BM25
Candidates per query	100
Maximum group size	20
Passage truncation	300 words
LLM temperature	0
Ranking mode	Reasoning-enhanced listwise
Brackets	Independent winner and loser tracks

The paper reports 77.90 NDCG@5 on TREC DL 19, 75.85 NDCG@5 on TREC DL 20, and 54.66 average NDCG@10 across the eight BEIR datasets.

🧪 Tests

pip install -e '.[test]'
pytest -q

The tests use a deterministic mock ranker and therefore make no API calls.

📁 Repository layout

bracketrank/
├── ranker.py       # adaptive grouping and winner/loser elimination
├── prompts.py      # reasoning-enhanced listwise prompt
├── clients.py      # OpenAI-compatible LLM client
├── cli.py          # JSONL reranking command
└── evaluate.py     # TREC DL and BEIR main-result runner
examples/
└── sample.jsonl
tests/
└── test_ranker.py

This release contains the main adaptive method plus the fixed tournament and hierarchical strategies from the original implementation. Experimental bracket ablations, unrelated baselines, and intermediate research outputs are not included.

📝 Citation

If you use BracketRank, please cite:

@inproceedings{abdallah-etal-2026-bracketrank,
    title = "{B}racket{R}ank: Large Language Model Document Ranking via Reasoning-based Competitive Elimination",
    author = "Abdallah, Abdelrahman  and
      Ali, Mohammed  and
      Piryani, Bhawna  and
      Jatowt, Adam",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.acl-long.153/",
    doi = "10.18653/v1/2026.acl-long.153",
    pages = "3381--3397",
    ISBN = "979-8-89176-390-6"
}

📄 License

Released under the Apache License 2.0.

🔗 Links

• 📄 ACL Anthology paper
• 🔖 DOI: 10.18653/v1/2026.acl-long.153
• 💻 Project repository