feat(data): working end-to-end data pipeline starter

.github/pull_request_template.md

@@ -0,0 +1,32 @@
+## What I built
+
+<!-- One or two sentences. What does this change do? -->
+
+## Why this approach
+
+<!-- The diff already shows what changed. Explain why you did it this way, and
+     what you considered instead. "An AI wrote it" is not an answer here, and it
+     will not be an answer in review either. -->
+
+## Contract impact
+
+<!-- Does this change a mart the backend reads, or the shape of an endpoint?
+     Say "None" if not. If yes, name the table or column and tag the backend
+     reviewer: a rename or a type change breaks them quietly. -->
+
+None
+
+## How to run
+
+<!-- The exact commands a reviewer runs to see this working. -->
+
+```bash
+
+```
+
+## Self-check
+
+- [ ] I ran this and it works
+- [ ] Tests pass locally
+- [ ] No secrets, tokens, or connection strings in the diff
+- [ ] This pull request does one thing

.github/workflows/pr-checks.yml

@@ -0,0 +1,114 @@
+name: PR checks
+
+# Two gates on every pull request:
+#
+# 1. The description uses the template. GitHub only auto-fills the template in
+#    the web "compose" form and in `gh pr create` with no --body. A pull request
+#    opened through the REST API or `gh pr create --body "..."`, which is the
+#    path most AI tools take, silently skips it. This check is the only thing
+#    that actually enforces it.
+#
+# 2. The diff stays reviewable. A two thousand line pull request does not get
+#    reviewed, it gets approved, which is not the same thing. Size is the real
+#    problem behind unreviewable AI-generated changes, so the limit is a number
+#    rather than an awkward conversation between teammates.
+#
+# Recovery is automatic for both: editing the description fires the `edited`
+# event and re-runs these checks. No new commit needed.
+
+on:
+  pull_request:
+    types: [opened, edited, reopened, synchronize]
+
+permissions:
+  contents: read
+
+env:
+  MAX_CHANGED_LINES: 400
+
+jobs:
+  body:
+    name: Description uses the template
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check required sections are present
+        env:
+          PR_BODY: ${{ github.event.pull_request.body }}
+        run: |
+          set -euo pipefail
+          required=(
+            "## What I built"
+            "## Why this approach"
+            "## Contract impact"
+            "## How to run"
+            "## Self-check"
+          )
+          missing=()
+          for section in "${required[@]}"; do
+            if ! printf '%s' "$PR_BODY" | grep -qiF "$section"; then
+              missing+=("$section")
+            fi
+          done
+          if [ ${#missing[@]} -ne 0 ]; then
+            echo "::error::Your pull request description is missing required sections. Start from .github/pull_request_template.md and keep these headings:"
+            for m in "${missing[@]}"; do echo "  - $m"; done
+            echo ""
+            echo "Click 'Edit' on the description, paste the template, and fill it in."
+            echo "Editing the description re-runs this check automatically."
+            exit 1
+          fi
+          echo "All required sections present."
+
+  size:
+    name: Diff stays reviewable
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Count changed lines, excluding generated files
+        id: count
+        env:
+          BASE_SHA: ${{ github.event.pull_request.base.sha }}
+          HEAD_SHA: ${{ github.event.pull_request.head.sha }}
+        run: |
+          set -euo pipefail
+          git fetch --no-tags --depth=1 origin "$BASE_SHA" 2>/dev/null || true
+          changed=$(git diff --numstat "$BASE_SHA" "$HEAD_SHA" -- \
+            . \
+            ':(exclude)**/uv.lock' \
+            ':(exclude)**/package-lock.json' \
+            ':(exclude)**/poetry.lock' \
+            ':(exclude)**/*.lock' \
+            ':(exclude)**/target/**' \
+            ':(exclude)**/dbt_packages/**' \
+            ':(exclude)**/node_modules/**' \
+            | awk '{ add += ($1 == "-" ? 0 : $1); del += ($2 == "-" ? 0 : $2) } END { print add + del + 0 }')
+          echo "changed=$changed" >> "$GITHUB_OUTPUT"
+          echo "Changed lines, excluding generated files: $changed"
+
+      - name: Enforce the limit, unless an override is documented
+        env:
+          PR_BODY: ${{ github.event.pull_request.body }}
+          CHANGED: ${{ steps.count.outputs.changed }}
+        run: |
+          set -euo pipefail
+          if [ "$CHANGED" -le "$MAX_CHANGED_LINES" ]; then
+            echo "$CHANGED changed lines is within the limit of $MAX_CHANGED_LINES."
+            exit 0
+          fi
+          if printf '%s' "$PR_BODY" | grep -qiE '^[[:space:]]*Oversized:[[:space:]]*\S'; then
+            echo "::warning::$CHANGED changed lines exceeds $MAX_CHANGED_LINES, but an 'Oversized:' reason is documented in the description."
+            exit 0
+          fi
+          echo "::error::$CHANGED changed lines exceeds the limit of $MAX_CHANGED_LINES."
+          echo ""
+          echo "Split this into smaller pull requests, one purpose each. A reviewer cannot"
+          echo "meaningfully check a change this size, and approving it without reading is"
+          echo "worse than not reviewing at all."
+          echo ""
+          echo "If it genuinely cannot be split, add a line to the description:"
+          echo "  Oversized: <one sentence saying why>"
+          echo "Editing the description re-runs this check."
+          exit 1

.gitignore

@@ -202,3 +202,31 @@ build/
 # Docker
 .docker/
 .buildx-cache/
+
+# --- Python (data track) ---
+__pycache__/
+*.py[cod]
+.venv/
+venv/
+.pytest_cache/
+.ruff_cache/
+.mypy_cache/
+*.egg-info/
+
+# --- dbt ---
+target/
+dbt_packages/
+logs/
+
+# --- Airflow (Astro) ---
+# .astro/config.yaml must be committed: without it the folder is not an Astro
+# project and `astro dev start` refuses to run.
+.astro/config.yaml.lock
+airflow_settings.yaml
+
+# --- Java / Spring Boot (backend track) ---
+*.class
+build/
+.gradle/
+.mvn/
+.user.yml

data/.dockerignore

@@ -0,0 +1,9 @@
+.env
+.venv
+__pycache__/
+*.pyc
+dbt/target/
+dbt/dbt_packages/
+dbt/logs/
+airflow/
+docs/

data/.env.example

@@ -0,0 +1,25 @@
+# Copy to .env and fill in. Never commit .env.
+#
+# Local development uses the Postgres started by docker compose, so the
+# defaults below work without an Azure account. Swap them for your team's
+# Azure values when you deploy.
+
+# --- Source API -------------------------------------------------------------
+# Arbeitnow needs no key. Replace with your team's source.
+SOURCE_API_URL=https://www.arbeitnow.com/api/job-board-api
+
+# --- Postgres ---------------------------------------------------------------
+POSTGRES_HOST=localhost
+POSTGRES_PORT=5432
+POSTGRES_DB=finalproject
+POSTGRES_USER=postgres
+POSTGRES_PASSWORD=postgres
+POSTGRES_SCHEMA_RAW=raw
+
+# --- dbt --------------------------------------------------------------------
+# The schema dbt builds into. Use your own name so teammates do not collide.
+DBT_SCHEMA=analytics
+
+# --- Azure Blob Storage (optional until you deploy) -------------------------
+# AZURE_STORAGE_CONNECTION_STRING=
+# AZURE_STORAGE_CONTAINER=raw

data/Dockerfile

@@ -0,0 +1,23 @@
+# Container image for the ingestion pipeline.
+#
+# Build:  docker build -t final-project-data .
+# Run:    docker run --rm --env-file .env final-project-data
+#
+# This is the image you push to Azure Container Registry and run as a
+# Container Apps Job, exactly as in Week 6.
+FROM python:3.11-slim
+
+WORKDIR /app
+
+# Dependencies are copied first so Docker can cache the install layer. Change
+# your source code and the rebuild stays fast; change dependencies and it does not.
+COPY pyproject.toml ./
+RUN pip install --no-cache-dir \
+        "requests>=2.32.0" \
+        "pydantic>=2.9.0" \
+        "psycopg[binary]>=3.2.0" \
+        "python-dotenv>=1.0.0"
+
+COPY src/ ./src/
+
+CMD ["python", "-m", "src.pipeline"]

data/README.md

@@ -1 +1,88 @@
-# Final Project Data Pipeline
+# Final Project Data Pipeline
+
+Starter code for the data half of the final project: fetch data from a source,
+validate it, store it, shape it with dbt, and publish a mart the backend team
+reads. It runs end to end the moment you clone it, against a local Postgres and
+a public API that needs no key, so your first hour goes into your product
+rather than into setup.
+
+## Run it in five minutes
+
+```bash
+cd data
+cp .env.example .env
+docker compose up -d db   # local Postgres on :5432
+
+uv venv && uv pip install -e ".[dbt]"
+uv run python -m src.pipeline   # fetch, validate, store
+
+cd dbt && uv run dbt build --profiles-dir .
+```
+
+To run the pipeline the way Azure will run it, in the container:
+
+```bash
+docker compose run --rm pipeline
+```
+
+> Inside a container, `localhost` is the container itself, not your machine.
+> That is why the `pipeline` service overrides `POSTGRES_HOST` to `postgres`,
+> the service name on the compose network. Plain
+> `docker run --env-file .env` cannot reach your local database.
+
+You should see around 175 rows land in `raw.postings`, then `stg_postings` and
+`fct_postings` build with all tests passing. Run the pipeline twice: the row
+count stays the same, because writes are upserts.
+
+## What is here
+
+| Path | What it does |
+|---|---|
+| `src/config.py` | Reads every setting from environment variables and fails loudly when one is missing |
+| `src/models.py` | Pydantic validation for incoming records |
+| `src/ingest.py` | Calls the source API, validates, counts rejects |
+| `src/storage.py` | Creates the raw schema and upserts rows |
+| `src/pipeline.py` | Entry point, wires the three steps together |
+| `dbt/models/staging/` | Cleans and renames. No business logic |
+| `dbt/models/marts/fct_postings.sql` | **The contract with the backend team** |
+| `dbt/tests/` | Two custom tests, including a zero-row check |
+| `airflow/dags/pipeline_dag.py` | Daily schedule: ingest, then dbt build |
+| `Dockerfile` | The image you push to Azure Container Registry |
+| `optional/` | Bicep, Databricks, and Streamlit modules. None required |
+
+## Making it yours
+
+The template ships a job-postings example so it runs immediately. Swapping in
+your team's data source is four edits:
+
+1. `.env`: point `SOURCE_API_URL` at your source.
+2. `src/models.py`: change the Pydantic model to match your records.
+3. `src/storage.py`: change the table definition and upsert to match.
+4. `dbt/models/`: rename the models and columns to your domain.
+
+Do this in your first two days. Everything after that builds on the shape you
+choose here.
+
+> Verify your source before you commit to it: call it once, print a record, and
+> confirm you can parse it. An idea you love with a source you cannot reach is
+> worth less than a plain idea that works.
+
+## The mart is a contract
+
+`fct_postings` is what the backend reads to build endpoints. Adding a column is
+safe. Renaming or removing one breaks the backend, so agree it with them first
+and change it in both places at once.
+
+Every column is documented in `dbt/models/marts/_fct_postings.yml`. Hand that
+file to the backend trainees on day one and they can write endpoints before
+your pipeline is finished. See `docs/mart_contract.md` for how to work on it
+together.
+
+## Secrets
+
+No credentials live in this folder. `dbt/profiles.yml` is committed on purpose:
+every value in it comes from `env_var(...)`, so it holds nothing secret. Real
+values live in `.env`, which is git-ignored, and in your deployment environment.
+
+Never commit `.env`, and never paste a connection string into a chat message or
+an LLM prompt.

data/airflow/.astro/config.yaml

@@ -0,0 +1,2 @@
+project:
+  name: final-project-data

data/airflow/.env.example

@@ -0,0 +1,19 @@
+# Copy to .env before running `astro dev start`. Never commit .env.
+#
+# POSTGRES_HOST is the compose service name "db", not localhost and not
+# "postgres": Airflow's own stack has a service called postgres, and that name
+# would resolve to Airflow's metadata database instead of yours.
+# the Airflow
+# containers join the "finalproject" network created by ../docker-compose.yml.
+# Start the database first with:  (cd .. && docker compose up -d postgres)
+
+SOURCE_API_URL=https://www.arbeitnow.com/api/job-board-api
+
+POSTGRES_HOST=db
+POSTGRES_PORT=5432
+POSTGRES_DB=finalproject
+POSTGRES_USER=postgres
+POSTGRES_PASSWORD=postgres
+POSTGRES_SCHEMA_RAW=raw
+
+DBT_SCHEMA=analytics

data/airflow/.gitignore

@@ -0,0 +1,3 @@
+.astro/config.yaml.lock
+airflow_settings.yaml
+logs/

data/airflow/Dockerfile

@@ -0,0 +1,5 @@
+# Astro runtime, the same image family you used in Week 12.
+#
+# Start locally:  astro dev start
+# Airflow UI:     http://localhost:8080
+FROM astrocrpublic.azurecr.io/runtime:3.3-2