From 9ea6e68af57e3b7f1034c5878dba49b520d350fc Mon Sep 17 00:00:00 2001 From: Francisco Schulz <22344801+fschlz@users.noreply.github.com> Date: Wed, 24 Jun 2026 17:04:32 -0400 Subject: [PATCH 1/2] chore: add contributor/agent harness Adapt the opik-examples harness to this repo: agent guardrails, Comet-accurate conventions, a starter template, and a scaffolder. Tailored to this repo's reality (comet_ml, requirements.txt + Poetry, kebab-case folders, the CI matrix) rather than copied verbatim. - .claude/settings.json: deny destructive git, gh pr merge/close/review, .env reads - AGENTS.md / CONTRIBUTING.md: Comet SDK idiom, env creds, optional offline mode, README house structure, CI-matrix instructions - templates/integration-example: minimal runnable comet_ml example - .claude/skills/scaffold-example: stamps the template, renames sentinels - readme.md: repository structure + contributing sections; fix stale tutorial links --- .claude/settings.json | 28 +++ .claude/skills/scaffold-example/SKILL.md | 81 +++++++++ .../scaffold-example/scripts/scaffold.py | 131 ++++++++++++++ .gitignore | 5 + AGENTS.md | 146 +++++++++++++++ CONTRIBUTING.md | 167 ++++++++++++++++++ readme.md | 44 ++++- templates/integration-example/README.md | 36 ++++ .../example_integration.py | 14 ++ .../integration-example/requirements.txt | 2 + 10 files changed, 646 insertions(+), 8 deletions(-) create mode 100644 .claude/settings.json create mode 100644 .claude/skills/scaffold-example/SKILL.md create mode 100644 .claude/skills/scaffold-example/scripts/scaffold.py create mode 100644 AGENTS.md create mode 100644 CONTRIBUTING.md create mode 100644 templates/integration-example/README.md create mode 100644 templates/integration-example/example_integration.py create mode 100644 templates/integration-example/requirements.txt diff --git a/.claude/settings.json b/.claude/settings.json new file mode 100644 index 0000000..8e47ae2 --- /dev/null +++ b/.claude/settings.json @@ -0,0 +1,28 @@ +{ + "permissions": { + "allow": [ + "Bash(git:*)", + "Bash(python:*)", + "Bash(python3:*)", + "Bash(pip:*)", + "Bash(poetry:*)", + "Bash(ls:*)", + "Bash(find:*)", + "Bash(grep:*)", + "Bash(jq:*)", + "Bash(echo:*)", + "Read(*)" + ], + "deny": [ + "Read(.env)", + "Bash(rm -rf /*)", + "Bash(git reset --hard:*)", + "Bash(git rebase:*)", + "Bash(git merge:*)", + "Bash(git branch -D:*)", + "Bash(gh pr merge:*)", + "Bash(gh pr close:*)", + "Bash(gh pr review:*)" + ] + } +} diff --git a/.claude/skills/scaffold-example/SKILL.md b/.claude/skills/scaffold-example/SKILL.md new file mode 100644 index 0000000..5e28dc2 --- /dev/null +++ b/.claude/skills/scaffold-example/SKILL.md @@ -0,0 +1,81 @@ +--- +name: scaffold-example +description: Scaffold a brand-new Comet example in this repo from the canonical template under templates/integration-example/. Use whenever the user wants to create, add, or start a new example, demo, or integration on Comet — phrasings like "add a Comet example for X", "scaffold an integration example", "create a new example for ", "start a hello-world for ", "set up a new example project". Do NOT hand-build the folder structure or copy files manually — this skill stamps out the example and renames it correctly. Require a name and a one-line description of what it does; refuse to invent an example from nothing. +--- + +# Scaffold a new Comet example + +This repo has one canonical example shape, a real, runnable template under +[`templates/integration-example/`](../../../templates/integration-example): a single `comet_ml` +script (`example_integration.py`), a `requirements.txt`, and a README in the house structure +(Title → Documentation → See it → Setup → Run the example). Getting it right by hand is fiddly and +easy to drift on. This skill copies the template and renames it, so you start from something that +already runs. + +## When to refuse + +You need two things: + +- A **name** (kebab-case, e.g. `pytorch-amp-example`, `keras-mnist-dnn`). It becomes the folder, the + Python module, and the Comet project name (`comet-example-`). +- A **one-line description** of what it does. + +If either is missing, ask one short question. Don't invent an example — an empty scaffold with a +guessed purpose wastes the user's time. + +## Step 1 — Generate the example + +Run the bundled generator. It copies the template, renames its sentinel identity +(`example_integration` / `example-integration`) to the new name, and refuses to overwrite an +existing folder. + +```bash +python .claude/skills/scaffold-example/scripts/scaffold.py \ + --description "" \ + --dest +``` + +`--dest` is the parent directory (default `integrations`). New examples live under +`integrations///`, so pass e.g. +`--dest integrations/model-training/pytorch`. Categories in use: `model-training`, +`model-evaluation`, `model-optimization`, `model-deployment`, `workflow-orchestration`, +`reinforcement-learning`, `llm`, `data-management`. The script prints the folder / script / project +names it chose and a next-steps checklist. + +The name is normalised: the folder and Comet project use kebab-case, the Python file uses +snake_case (so it stays importable). The description is printed for you to paste into the README +intro — there's no structured field for it. + +## Step 2 — Make it real + +The scaffold runs as-is but is generic. Fill the parts marked `TODO`: + +- **`.py`** — replace the stub loop with the real instrumentation: your framework's training + or inference, plus the `comet_ml` logging that matters (`log_parameters`, `log_metrics`, + `log_model`, …). Keep the `comet_ml.login(...)` + `comet_ml.start()` + `experiment.end()` frame. +- **`requirements.txt`** — add the framework dependency alongside the pinned `comet_ml`. +- **`README.md`** — fill the title, intro (paste the description), the docs link, and a public + Comet project link if one exists. This is required for every example (see + [AGENTS.md](../../../AGENTS.md)). + +Keep the repo conventions: credentials from env vars only, `# WHY:`-only comments, no emojis. + +## Step 3 — Verify + +From the new example directory: + +```bash +python -m pip install -r requirements.txt +python .py # logs a real run (needs COMET_API_KEY) +COMET_MODE=offline python .py # runs without an account, if practical +``` + +To get the example tested, add it to the matrix in +[`.github/workflows/test-examples.yml`](../../../.github/workflows/test-examples.yml) — the +`notebooks` list for a `.ipynb`, or the `example` list for a script. See +[CONTRIBUTING.md](../../../CONTRIBUTING.md#ci). + +## What this skill does NOT do + +- **No git commit / PR.** Leave the new files in the working tree for the user to review. +- **No domain logic.** It scaffolds; you (with the user) write the actual instrumentation, deps, and README. diff --git a/.claude/skills/scaffold-example/scripts/scaffold.py b/.claude/skills/scaffold-example/scripts/scaffold.py new file mode 100644 index 0000000..a37abe3 --- /dev/null +++ b/.claude/skills/scaffold-example/scripts/scaffold.py @@ -0,0 +1,131 @@ +#!/usr/bin/env python3 +"""Scaffold a new Comet example by copying and renaming the canonical template. + +One template lives under `templates/integration-example/`: a single `comet_ml` script, a +`requirements.txt`, and a README in the house structure. It is a real, runnable example under a +sentinel identity (`example_integration` / `example-integration`). This script copies it to the +target directory and rewrites those sentinels to the new name — a pure identifier rename, no +template language. Run it, then fill in the TODO stubs. + + python scaffold.py pytorch-amp-example --description "..." --dest integrations/model-training/pytorch +""" + +import argparse +import re +import shutil +import sys +from pathlib import Path + +TEMPLATE_DIR = ("templates", "integration-example") +PKG_SENTINEL = "example_integration" # snake_case: the .py module name +PROJECT_SENTINEL = "example-integration" # kebab-case: folder + comet project name + +EXCLUDE = shutil.ignore_patterns( + ".venv", "__pycache__", "*.pyc", ".ruff_cache", ".env", "*.log", ".tmp" +) + + +def to_snake(name: str) -> str: + s = re.sub(r"[^0-9a-zA-Z]+", "_", name.strip().lower()).strip("_") + if not s: + raise ValueError(f"cannot derive a name from {name!r}") + if s[0].isdigit(): + s = f"x_{s}" + return s + + +def to_kebab(snake: str) -> str: + return snake.replace("_", "-") + + +def is_text(path: Path) -> bool: + try: + path.read_text(encoding="utf-8") + return True + except (UnicodeDecodeError, OSError): + return False + + +def main() -> int: + parser = argparse.ArgumentParser( + description="Scaffold a new Comet example from templates/integration-example." + ) + parser.add_argument("name", help="Example name, e.g. 'pytorch-amp-example'.") + parser.add_argument( + "--description", + default=None, + help="One-line description (printed; paste into the README intro).", + ) + parser.add_argument( + "--dest", + default="integrations", + help="Parent directory for the new example (default: integrations). " + "Relative paths are resolved from the repo root; absolute paths are used as-is.", + ) + parser.add_argument( + "--repo-root", + default=None, + help="Repo root (default: inferred from this script).", + ) + args = parser.parse_args() + + repo_root = ( + Path(args.repo_root).resolve() + if args.repo_root + else Path(__file__).resolve().parents[4] + ) + template = repo_root.joinpath(*TEMPLATE_DIR) + + pkg = to_snake(args.name) + kebab = to_kebab(pkg) + + if not template.is_dir(): + print(f"error: template not found at {template}", file=sys.stderr) + return 1 + + dest_parent = ( + Path(args.dest) if Path(args.dest).is_absolute() else repo_root / args.dest + ) + dest = dest_parent / kebab + if dest.exists(): + print(f"error: {dest} already exists — refusing to overwrite.", file=sys.stderr) + return 1 + + dest.parent.mkdir(parents=True, exist_ok=True) + shutil.copytree(template, dest, ignore=EXCLUDE) + + # Rename the module file (script kind). + mod_file = dest / f"{PKG_SENTINEL}.py" + if mod_file.is_file(): + mod_file.rename(dest / f"{pkg}.py") + + # Rewrite sentinels in every text file. Snake and kebab sentinels don't overlap + # (underscore vs hyphen), so independent replacement is safe. + for path in dest.rglob("*"): + if not path.is_file() or not is_text(path): + continue + text = path.read_text(encoding="utf-8") + new = text.replace(PKG_SENTINEL, pkg).replace(PROJECT_SENTINEL, kebab) + if new != text: + path.write_text(new, encoding="utf-8") + + try: + rel = dest.relative_to(repo_root) + except ValueError: + rel = dest + print(f"Scaffolded {rel}") + print(f" folder: {kebab}/") + print(f" script: {pkg}.py") + print(f" comet project: comet-example-{kebab}") + if args.description: + print(f" description: {args.description}") + print("\nNext steps:") + print(f" 1. cd {rel} && python -m pip install -r requirements.txt") + print(f" 2. Fill in {pkg}.py (real logic) and the README sections (intro, links).") + print(f" 3. Run it: python {pkg}.py (or: COMET_MODE=offline python {pkg}.py)") + print(" 4. To test it in CI, add it to .github/workflows/test-examples.yml.") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/.gitignore b/.gitignore index 7c37f9f..b2d6926 100644 --- a/.gitignore +++ b/.gitignore @@ -108,3 +108,8 @@ data/* # pytest .pytest_cache/ MagicMock/ + +# Tooling / agent harness +.ruff_cache/ +.tmp/ +*-workspace/ # skill eval scratch (e.g. skill-creator benchmarks) — never commit diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..1e914e9 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,146 @@ +# Agent instructions for comet-examples + +This repo is a reference library of examples for [Comet](https://www.comet.com/site/), the ML +experiment-tracking, model-management, and observability platform. Examples instrument popular ML +frameworks (PyTorch, Keras, fastai, scikit-learn, XGBoost, Transformers, …) with `comet_ml`. Read +this file before generating or editing any code. + +## Repo structure + +``` +comet-examples/ +├── integrations/ # Add Comet to a framework, grouped by ML task (see below) +├── guides/ # How-to notebooks for Comet workflows +├── panels/ # Custom Comet panel (visualization) examples +├── notebooks/ # General/standalone notebooks +└── templates/ # Starter template (integration-example) +``` + +`integrations/` is grouped by ML task, then by framework, then by example: + +``` +integrations//// +``` + +Categories in use: `model-training`, `model-evaluation`, `model-optimization`, `model-deployment`, +`workflow-orchestration`, `reinforcement-learning`, `llm`, `data-management`. + +**Where does new code belong?** + +| If you are… | Put it in | +|---|---| +| Instrumenting a framework with Comet (the common case) | `integrations////` | +| Writing a how-to notebook for a Comet workflow | `guides/` | +| Building a custom Comet panel | `panels/` | + +New examples go under `integrations/`. The top-level legacy dirs (`pytorch/`, `fastai/`, +`xgboost/`, `keras/`, `tensorflow/`) are historical — **do not add new examples there.** Example +folder names are **kebab-case** (`pytorch-mnist`, `fastai-hello-world`). + +When adding an example, use the [`scaffold-example`](.claude/skills/scaffold-example/SKILL.md) +skill — it stamps [`templates/integration-example/`](templates/integration-example/) into the +target directory and renames it for you. + +## Non-negotiable conventions + +### Credentials — always from environment variables + +```python +import os + +COMET_API_KEY = os.environ.get("COMET_API_KEY") +COMET_WORKSPACE = os.environ.get("COMET_WORKSPACE") +``` + +Never hardcode keys. Never read `.env` files in example code. In CI the workspace is +`cometexamples-tests` and `COMET_API_KEY` comes from a secret. + +### The Comet SDK idiom + +Match the house style used across the repo: + +```python +import comet_ml + +comet_ml.login(project_name="comet-example-") +experiment = comet_ml.start() +# ... training / logging ... +experiment.end() +``` + +`comet_ml.login()` reads `COMET_API_KEY` from the environment. Project names follow +`comet-example--`. + +### Offline mode — the no-account path (recommended, not required) + +Most examples log a real run, so they need an API key. Where it is practical, let a reader run +without an account using Comet's offline mode — no code change required: + +```bash +COMET_MODE=offline python .py +``` + +Note this in the README's run section when it works. Don't force it onto examples that genuinely +need a live run. + +### Comments — only when the WHY is non-obvious + +These are teaching examples: keep them short and readable. Don't add comments that restate what the +code does. Use a one-line `# WHY:` comment only when a behaviour would surprise a reader (a hidden +API constraint, a non-obvious ordering requirement, a known gotcha). No type-hint or docstring +mandate — match the surrounding file. + +### Dependencies — a `requirements.txt` per example + +Every example ships a `requirements.txt` **sibling to its entry point** (script or notebook +directory). This is the single source of truth and is exactly what CI installs +(`pip install -r requirements.txt`). Pin `comet_ml` like the rest of the repo (`comet_ml>=3.44.0`) +plus the framework deps. + +Richer, multi-file projects **may** use Poetry with a committed `poetry.lock` (see +[`integrations/langgraph/`](integrations/langgraph/)). Do **not** introduce `uv`, and do **not** +gitignore `poetry.lock` — this repo commits it for reproducibility. + +## Coding best practices + +- **Principles:** DRY, KISS, YAGNI. Prefer reusing an existing helper over adding a new one. +- **Match the surrounding file's** style, naming, and comment density. +- **Git / PR safety:** + - Never `git commit` or `git push` on `master`. Cut a feature branch (`/`, e.g. + `fschlz/pytorch-amp-example`), push there, open a PR, and let a human merge. + - Commits follow **Conventional Commits**: `feat:`, `fix:`, `chore:`, `refactor:`, `docs:`. + - Never `gh pr merge`, `gh pr close`, or `gh pr review --approve` — author/reviewer actions only. + Blocked in [`.claude/settings.json`](.claude/settings.json). + - No AI-attribution footers in commit messages or PR bodies. + - **Update READMEs before opening a PR** — the example's own `README.md`, and the root + [`readme.md`](readme.md) if you add a new top-level area. + +## CI + +Tests run via [`.github/workflows/test-examples.yml`](.github/workflows/test-examples.yml), which +holds **explicit matrices** of notebooks and scripts — examples are not auto-discovered. To get a +new example covered: + +1. Make sure its `requirements.txt` is complete and the example runs from its own directory. +2. Add it to the matrix — the `notebooks` list (run with `ipython`) for a `.ipynb`, or the + `example` list (run with `python