Add Real-World Project: Build a Re-Ranking Pipeline

docs/projects/index.mdx

@@ -17,6 +17,12 @@ They're optional and ungraded. Browse them any time — each project's intro say
 
 <ProjectChooser
   projects={mergeProjectMeta([
+    {
+      id: 'reranking-pipeline',
+      title: 'Build a Re-Ranking Pipeline',
+      summary:
+        'Retrieve fast with a cheap keyword stage, re-rank the top-K hits with a cross-encoder model, and benchmark whether the extra compute actually buys better precision — no API key, no LLM.',
+    },
     {
       id: '2027-dependency-freshness-checker',
       title: 'Build a Dependency-Freshness Checker',

docs/projects/reranking-pipeline/_category_.json

@@ -0,0 +1,4 @@
+{
+  "label": "RerankingPipeline",
+  "position": 24
+}

examples/reranking-pipeline/.gitignore

@@ -0,0 +1,5 @@
+.venv
+__pycache__
+*.pyc
+index.npy
+chunks.json

examples/reranking-pipeline/.python-version

@@ -0,0 +1 @@
+3.12

examples/reranking-pipeline/README.md

@@ -0,0 +1,48 @@
+# Re-Ranking Pipeline Example
+
+The local companion to the course's [Build a Re-Ranking Pipeline](https://github.com/abderrahim-lectures/python-data-analysis-course/tree/main/docs/projects/reranking-pipeline) project — a real, runnable two-stage retrieval pipeline that makes the compute/quality tradeoff of re-ranking visible.
+
+## What's here
+
+- `main.py` — the whole pipeline in one CLI:
+  - **Stage 1 (fast):** a pure-Python keyword-overlap scorer. Tokenizes the query, drops stopwords, and counts how much each document's words overlap with it. Milliseconds per query, no model, no download.
+  - **Stage 2 (accurate):** a sentence-transformers `CrossEncoder` (`cross-encoder/ms-marco-MiniLM-L-6-v2`) that reads each (query, document) *pair* together and re-scores just the top-K shortlist.
+  - A benchmark that runs the same test queries through both pipelines and prints `precision@1`, `precision@3`, and average time per query side by side.
+- `data/corpus/*.txt` — 25 short passages on varied topics, with deliberate "same word, different meaning" traps (Python the language vs. the snake, Mercury the planet vs. the god vs. the metal, a river bank vs. a money bank, ...) so the fast stage makes plausible-but-wrong top picks that the cross-encoder corrects.
+- `data/test_queries.json` — 12 test queries, each labeled with the relevant passage id, used by the benchmark.
+- `notebook.ipynb` — the same walkthrough as a self-contained notebook for Colab, Kaggle, or Binder (the corpus is embedded inline).
+
+**No API key, no `.env`, no LLM.** The only download is the cross-encoder model (~80MB) on its first run — it's a local model that scores query–document pairs on your own CPU.
+
+## How to run this
+
+```bash
+uv run python main.py            # benchmark: fast-only vs fast + re-rank
+uv run python main.py --query "what metal is liquid at room temperature"
+uv run python main.py --top-k 8  # hand the re-ranker a bigger shortlist
+```
+
+The first run downloads and caches the cross-encoder (~80MB) and then prints the benchmark. Later runs skip the download.
+
+`uv run` reads `pyproject.toml`/`uv.lock` and creates an isolated environment for this project automatically on first run — no manual virtual environment setup needed.
+
+## Project layout
+
+```
+reranking-pipeline/
+├── main.py                  # the pipeline + CLI + benchmark
+├── data/
+│   ├── corpus/*.txt         # 25 short passages to search over
+│   └── test_queries.json    # benchmark queries with ground-truth relevance
+├── notebook.ipynb           # hosted-notebook version (corpus embedded inline)
+├── pyproject.toml           # project metadata + deps
+└── uv.lock                  # pinned dependency lockfile
+```
+
+## A note on what re-ranking can and can't do
+
+Re-ranking only re-scores the top-K the fast stage already retrieved. If the right document never makes it into that shortlist, no re-ranker can save it — which is exactly why production systems tune their first stage too. What re-ranking *does* fix is the ordering: a correct-but-under-ranked hit (say, Jupiter ranked 2nd by "biggest planet" because a naive stage can't tell "biggest" and "smallest" apart) gets promoted to the top. The benchmark is honest about this — run it and compare.
+
+## Built your own version?
+
+See [`examples/student-projects/`](../student-projects/) for how to share it with the class via a pull request — no git experience required, it walks through every step.

examples/reranking-pipeline/data/corpus/amazon_rainforest.txt

@@ -0,0 +1 @@
+The Amazon rainforest produces about a fifth of the oxygen in Earth's atmosphere.

examples/reranking-pipeline/data/corpus/apple_iphone.txt

@@ -0,0 +1 @@
+Apple designs the iPhone, an iconic smartphone released every September since 2007.

examples/reranking-pipeline/data/corpus/apple_orchard.txt

@@ -0,0 +1 @@
+Apple trees grow in orchards, and their fruit ripens in late summer before it is picked.

examples/reranking-pipeline/data/corpus/baseball_bat.txt

@@ -0,0 +1 @@
+A baseball bat is a wooden club a hitter swings to send the ball into the outfield.

examples/reranking-pipeline/data/corpus/bat_animal.txt

@@ -0,0 +1 @@
+Bats are the only mammals that can truly fly, using echolocation to hunt insects at night.

examples/reranking-pipeline/data/corpus/beethoven.txt

@@ -0,0 +1 @@
+Beethoven, the German composer, wrote his famous Ninth Symphony while completely deaf.

examples/reranking-pipeline/data/corpus/black_holes.txt

@@ -0,0 +1 @@
+Black holes are regions of spacetime where gravity is so strong that nothing, not even light, can escape.

examples/reranking-pipeline/data/corpus/dna.txt

@@ -0,0 +1 @@
+DNA is the molecule that carries the genetic instructions for all living things.

examples/reranking-pipeline/data/corpus/earth_year.txt

@@ -0,0 +1 @@
+The Earth orbits the Sun once every 365 days, which is what defines a year.

examples/reranking-pipeline/data/corpus/great_barrier_reef.txt

@@ -0,0 +1 @@
+The Great Barrier Reef is the world's largest coral reef system, off the coast of Australia.

examples/reranking-pipeline/data/corpus/great_wall_china.txt

@@ -0,0 +1 @@
+The Great Wall of China was built over centuries to protect against invasions from the north.

examples/reranking-pipeline/data/corpus/honeybee.txt

@@ -0,0 +1 @@
+A honeybee colony has a single queen, thousands of workers, and a few hundred drones.

examples/reranking-pipeline/data/corpus/jupiter.txt

@@ -0,0 +1 @@
+Jupiter is the largest planet in our solar system, a gas giant with a storm called the Great Red Spot.

examples/reranking-pipeline/data/corpus/mercury_element.txt

@@ -0,0 +1 @@
+Mercury is the only metal that is liquid at room temperature, once used in thermometers.

examples/reranking-pipeline/data/corpus/mercury_god.txt

@@ -0,0 +1 @@
+In Roman mythology, Mercury was the messenger god, famous for his winged sandals and helmet.

examples/reranking-pipeline/data/corpus/mercury_planet.txt

@@ -0,0 +1 @@
+Mercury is the smallest planet in the solar system and the closest one to the Sun.

examples/reranking-pipeline/data/corpus/microchip.txt

@@ -0,0 +1 @@
+A microchip is a tiny integrated circuit that powers modern computers and smartphones.

examples/reranking-pipeline/data/corpus/money_bank.txt

@@ -0,0 +1 @@
+A bank is a business that keeps your money safe, pays interest on savings, and issues loans.

examples/reranking-pipeline/data/corpus/mount_everest.txt

@@ -0,0 +1 @@
+Mount Everest is the highest mountain on Earth, at about 8,849 meters above sea level.

examples/reranking-pipeline/data/corpus/nile_river.txt

@@ -0,0 +1 @@
+The Nile is traditionally considered the longest river in the world.

examples/reranking-pipeline/data/corpus/photosynthesis.txt

@@ -0,0 +1 @@
+Photosynthesis lets plants use sunlight to make food and release oxygen from their leaves.

examples/reranking-pipeline/data/corpus/potato_chips.txt

@@ -0,0 +1 @@
+Potato chips are thin, crispy slices of fried potato, usually salted for a snack.

examples/reranking-pipeline/data/corpus/python_programming.txt

@@ -0,0 +1 @@
+Python is a high-level programming language known for its readable syntax and a huge ecosystem of libraries.

examples/reranking-pipeline/data/corpus/python_snake.txt

@@ -0,0 +1 @@
+The python is a large non-venomous snake that kills its prey by constriction, squeezing it until it can no longer breathe.

examples/reranking-pipeline/data/corpus/river_bank.txt

@@ -0,0 +1 @@
+A river bank is the strip of land along the water, where plants and trees often take root.

examples/reranking-pipeline/data/test_queries.json

@@ -0,0 +1,50 @@
+[
+  {
+    "query": "What is the biggest planet in the solar system?",
+    "relevant": ["jupiter.txt"]
+  },
+  {
+    "query": "Which planet in our solar system is closest to the Sun?",
+    "relevant": ["mercury_planet.txt"]
+  },
+  {
+    "query": "How does a computer run Python code?",
+    "relevant": ["python_programming.txt"]
+  },
+  {
+    "query": "Which snake squeezes its prey to death?",
+    "relevant": ["python_snake.txt"]
+  },
+  {
+    "query": "What animal flies at night and hunts insects?",
+    "relevant": ["bat_animal.txt"]
+  },
+  {
+    "query": "Which company makes the iPhone smartphone?",
+    "relevant": ["apple_iphone.txt"]
+  },
+  {
+    "query": "What metal is liquid at room temperature?",
+    "relevant": ["mercury_element.txt"]
+  },
+  {
+    "query": "Where can you keep your money and earn interest?",
+    "relevant": ["money_bank.txt"]
+  },
+  {
+    "query": "What is the tallest mountain in the world?",
+    "relevant": ["mount_everest.txt"]
+  },
+  {
+    "query": "Which river is the longest in the world?",
+    "relevant": ["nile_river.txt"]
+  },
+  {
+    "query": "What molecule carries genetic information?",
+    "relevant": ["dna.txt"]
+  },
+  {
+    "query": "What do plants use sunlight to make?",
+    "relevant": ["photosynthesis.txt"]
+  }
+]