diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml index 8e5709e3..86095586 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.yml +++ b/.github/ISSUE_TEMPLATE/bug_report.yml @@ -6,7 +6,7 @@ body: - type: markdown attributes: value: | - Thanks for taking the time to report a bug! Please fill out the sections below. + Search the existing issues first. A security weakness goes through [private reporting](https://github.com/HardMax71/Integr8sCode/security/advisories/new), not a public issue. - type: textarea id: description @@ -44,12 +44,15 @@ body: description: Which part of the system is affected? options: - Frontend (UI) - - Backend API - - Execution (K8s pods) - - Workers (Saga, DLQ, etc.) - - SSE / Real-time updates - - Authentication - - Other + - API (routes, auth, rate limits, settings) + - Running scripts (queue, sagas, pods, results) + - Kafka events (consumers, DLQ, replay) + - Live updates and notifications + - Metrics, dashboards and logs + - Deployment (compose, deploy.sh, production) + - CI and tooling + - Documentation + - Not sure validations: required: true @@ -57,12 +60,12 @@ body: id: environment attributes: label: Environment - description: Please provide relevant environment details. + description: Whatever applies. placeholder: | - - OS: [e.g., Ubuntu 22.04, macOS 14] - - Browser: [e.g., Chrome 120, Firefox 121] - - Deployment: [e.g., local dev, production] - - Python version: [e.g., 3.12] + - Where: local dev (./deploy.sh dev) or production + - Version: release tag or commit, e.g. 2026.10.3 or 7b9f060 + - Script language and version, e.g. Go 1.27 + - Browser, e.g. Firefox 145 validations: required: false diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml index 29f76427..9688e729 100644 --- a/.github/ISSUE_TEMPLATE/config.yml +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -1,6 +1,9 @@ # Discussions are off, so questions come in as issues. Neither template fits a question, hence blank issues stay on. blank_issues_enabled: true contact_links: + - name: Report a security weakness + url: https://github.com/HardMax71/Integr8sCode/security/advisories/new + about: Report it privately so it can be fixed before it's public - name: Documentation url: https://hardmax71.github.io/Integr8sCode/ about: Check the documentation for guides and reference material diff --git a/.github/ISSUE_TEMPLATE/feature_request.yml b/.github/ISSUE_TEMPLATE/feature_request.yml index 319c796f..f3b54d16 100644 --- a/.github/ISSUE_TEMPLATE/feature_request.yml +++ b/.github/ISSUE_TEMPLATE/feature_request.yml @@ -6,7 +6,7 @@ body: - type: markdown attributes: value: | - Thanks for suggesting a feature! Please describe what you'd like to see. + Describe the problem first, then what you'd like to see. - type: textarea id: problem @@ -40,12 +40,15 @@ body: description: Which part of the system would this affect? options: - Frontend (UI) - - Backend API - - Execution (K8s pods) - - Workers + - API (routes, auth, rate limits, settings) + - Running scripts (queue, sagas, pods, results) + - Kafka events (consumers, DLQ, replay) + - Live updates and notifications + - Metrics, dashboards and logs + - Deployment (compose, deploy.sh, production) + - CI and tooling - Documentation - - DevOps / CI/CD - - Other + - Not sure validations: required: true diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index a147a855..a96d289b 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -31,6 +31,7 @@ Fixes # - [ ] Frontend unit tests pass (`cd frontend && npm run test`) - [ ] Backend E2E tests pass (`docker compose exec -T backend sh -c 'uv sync --group test --no-dev --frozen --no-install-project --no-build && pytest tests/e2e'`) - [ ] Frontend E2E tests pass (`cd frontend && npm run test:e2e`) +- [ ] API spec and client regenerated if routes or schemas changed (`./deploy.sh openapi`, then `./deploy.sh types`), never edited by hand - [ ] Manual testing performed **Manual test steps:** diff --git a/.github/issue-labeler.yml b/.github/issue-labeler.yml new file mode 100644 index 00000000..6c231647 --- /dev/null +++ b/.github/issue-labeler.yml @@ -0,0 +1,21 @@ +# Area label per answer to the Component field of the issue forms, applied by the Labels workflow. +# Each pattern matches the start of a dropdown option, so keep them in step with both forms. + +"area: frontend": + - '### Component\s+Frontend' +"area: api": + - '### Component\s+API' +"area: execution": + - '### Component\s+Running scripts' +"area: events": + - '### Component\s+Kafka events' +"area: realtime": + - '### Component\s+Live updates' +"area: observability": + - '### Component\s+Metrics' +"area: deploy": + - '### Component\s+Deployment' +"area: ci": + - '### Component\s+CI and tooling' +documentation: + - '### Component\s+Documentation' diff --git a/.github/labeler.yml b/.github/labeler.yml new file mode 100644 index 00000000..3e49d1f6 --- /dev/null +++ b/.github/labeler.yml @@ -0,0 +1,99 @@ +# Area labels for pull requests by changed path, applied by the Labels workflow. +# Issues get theirs from the Component field instead; keep both in step with the label list. + +"area: frontend": + - changed-files: + - any-glob-to-any-file: frontend/** + +"area: api": + - changed-files: + - any-glob-to-any-file: + - backend/app/api/** + - backend/app/schemas_pydantic/** + - backend/app/core/middlewares/** + - backend/app/core/exceptions/** + - backend/app/core/security.py + - backend/app/services/admin/** + - backend/app/services/auth_service.py + - backend/app/services/login_lockout.py + - backend/app/services/rate_limit_service.py + - backend/app/services/saved_script_service.py + - backend/app/services/user_settings_service.py + - backend/app/settings.py + - backend/config*.toml + - docs/reference/openapi.json + +"area: execution": + - changed-files: + - any-glob-to-any-file: + - backend/app/services/execution_service.py + - backend/app/services/execution_queue.py + - backend/app/services/runtime_settings.py + - backend/app/services/saga/** + - backend/app/services/k8s_worker/** + - backend/app/services/pod_monitor/** + - backend/app/services/result_processor/** + - backend/app/runtime_registry.py + - backend/app/scripts/** + - backend/workers/run_k8s_worker.py + - backend/workers/run_pod_monitor.py + - backend/workers/run_result_processor.py + - backend/workers/run_saga_orchestrator.py + +"area: events": + - changed-files: + - any-glob-to-any-file: + - backend/app/events/** + - backend/app/dlq/** + - backend/app/domain/events/** + - backend/app/infrastructure/kafka/** + - backend/app/services/idempotency/** + - backend/app/services/event_replay/** + - backend/app/services/event_service.py + - backend/workers/run_dlq_processor.py + - backend/workers/run_event_replay.py + +"area: realtime": + - changed-files: + - any-glob-to-any-file: + - backend/app/services/sse/** + - backend/app/services/notification_service.py + - backend/app/services/notification_scheduler.py + - backend/app/api/routes/sse.py + - backend/app/api/routes/notifications.py + - frontend/src/lib/notifications/** + +"area: observability": + - changed-files: + - any-glob-to-any-file: + - backend/app/core/metrics/** + - backend/app/core/tracing/** + - backend/app/core/logging.py + - backend/app/core/middlewares/metrics.py + - backend/grafana/** + - backend/loki/** + - backend/otel-collector-config.yaml + - backend/tests/contract/** + +"area: deploy": + - changed-files: + - any-glob-to-any-file: + - docker-compose.yaml + - deploy.sh + - "**/Dockerfile*" + - cert-generator/** + - .github/workflows/docker.yml + - .github/workflows/release-deploy.yml + +"area: ci": + - changed-files: + - any-glob-to-any-file: + - .github/** + - .pre-commit-config.yaml + +documentation: + - changed-files: + - any-glob-to-any-file: + - docs/** + - "*.md" + - mkdocs.yml diff --git a/.github/labels.yml b/.github/labels.yml new file mode 100644 index 00000000..ff33ef10 --- /dev/null +++ b/.github/labels.yml @@ -0,0 +1,75 @@ +# Every label in the repo. Label Sync makes the repo match this file and deletes labels that aren't in it, +# so add or rename a label here (rename with from_name), never in the GitHub UI. + +- name: "area: api" + color: "#c5def5" + description: "FastAPI routes, auth, rate limiting, settings" +- name: "area: ci" + color: "#c5def5" + description: "GitHub workflows, test infrastructure, repo tooling" +- name: "area: deploy" + color: "#c5def5" + description: "Compose, deploy.sh, images, release workflow, production host" +- name: "area: events" + color: "#c5def5" + description: "Kafka consumers and middlewares, producer, idempotency, DLQ, replay" +- name: "area: execution" + color: "#c5def5" + description: "Queue, sagas, k8s worker, pod monitor, result processor, executor pods, runtimes" +- name: "area: frontend" + color: "#c5def5" + description: "Svelte UI and the generated API client" +- name: "area: observability" + color: "#c5def5" + description: "Metrics, dashboards, alerts, logs, traces, OTel collector" +- name: "area: realtime" + color: "#c5def5" + description: "SSE streams and notifications" +- name: "bug" + color: "#d73a4a" + description: "Something isn't working" +- name: "dependencies" + color: "#0366d6" + description: "Pull requests that update a dependency file" +- name: "docker" + color: "#21ceff" + description: "Pull requests that update docker code" +- name: "documentation" + color: "#0075ca" + description: "Improvements or additions to documentation" +- name: "duplicate" + color: "#cfd3d7" + description: "This issue or pull request already exists" +- name: "enhancement" + color: "#a2eeef" + description: "New feature or request" +- name: "github_actions" + color: "#000000" + description: "Pull requests that update GitHub Actions code" +- name: "good first issue" + color: "#7057ff" + description: "Good for newcomers" +- name: "help wanted" + color: "#008672" + description: "Extra attention is needed" +- name: "invalid" + color: "#e4e669" + description: "This doesn't seem right" +- name: "javascript" + color: "#168700" + description: "Pull requests that update javascript code" +- name: "python:uv" + color: "#2b67c6" + description: "Pull requests that update python:uv code" +- name: "question" + color: "#d876e3" + description: "Further information is requested" +- name: "security" + color: "#b60205" + description: "Hardening or a security weakness; report anything exploitable privately" +- name: "triage" + color: "#FBCA04" + description: "New report, needs a first look" +- name: "wontfix" + color: "#ffffff" + description: "This will not be worked on" diff --git a/.github/workflows/label-sync.yml b/.github/workflows/label-sync.yml new file mode 100644 index 00000000..d3c23932 --- /dev/null +++ b/.github/workflows/label-sync.yml @@ -0,0 +1,55 @@ +name: Label Sync + +on: + push: + branches: [main] + paths: + - .github/labels.yml + - .github/labeler.yml + - .github/issue-labeler.yml + - .github/ISSUE_TEMPLATE/** + - .github/workflows/label-sync.yml + pull_request: + branches: [main] + paths: + - .github/labels.yml + - .github/labeler.yml + - .github/issue-labeler.yml + - .github/ISSUE_TEMPLATE/** + - .github/workflows/label-sync.yml + schedule: + - cron: "0 3 * * 1" + workflow_dispatch: + +permissions: {} + +jobs: + sync: + name: Check and sync labels + runs-on: ubuntu-latest + permissions: + contents: read + issues: write + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + + - name: Fail on labels used but not declared + run: | + declared=$(yq '.[].name' .github/labels.yml | sort -u) + used=$({ + yq -N 'keys | .[]' .github/labeler.yml .github/issue-labeler.yml + yq -N '.labels[]' .github/ISSUE_TEMPLATE/bug_report.yml .github/ISSUE_TEMPLATE/feature_request.yml + } | sort -u) + undeclared=$(comm -13 <(echo "$declared") <(echo "$used")) + if [ -n "$undeclared" ]; then + echo "::error::Labels used in the labeler configs or issue forms but missing from .github/labels.yml:" + echo "$undeclared" + exit 1 + fi + + # Makes the repo's labels match .github/labels.yml; on PRs it only prints what it would change + - uses: crazy-max/ghaction-github-labeler@548a7c3603594ec17c819e1239f281a3b801ab4d # v6.0.0 + with: + dry-run: ${{ github.event_name == 'pull_request' }} diff --git a/.github/workflows/labels.yml b/.github/workflows/labels.yml new file mode 100644 index 00000000..32ad530d --- /dev/null +++ b/.github/workflows/labels.yml @@ -0,0 +1,34 @@ +name: Labels + +on: + pull_request_target: + types: [opened, synchronize, reopened] + issues: + types: [opened, edited] + +permissions: {} + +jobs: + pull-request: + name: Label PR by changed paths + if: github.event_name == 'pull_request_target' + runs-on: ubuntu-latest + permissions: + contents: read + pull-requests: write + steps: + # Reads .github/labeler.yml from main and the changed file list from the API; never checks out PR code + - uses: actions/labeler@bf12e9b00b37c5c0ca2b87b79b2daf7891dbda13 # v7.0.0 + + issue: + name: Label issue by component + if: github.event_name == 'issues' + runs-on: ubuntu-latest + permissions: + contents: read + issues: write + steps: + - uses: github/issue-labeler@ccca1e16fcb66762200e616fc29b57b940c1cb67 # v3.5 + with: + configuration-path: .github/issue-labeler.yml + enable-versioned-regex: 0 diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 8603a77c..45c01a95 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -137,7 +137,9 @@ test: add e2e tests for DLQ processor ## Reporting Issues -Use GitHub Issues: the Bug Report template for unexpected behavior or errors, the Feature Request template for new functionality. Include reproduction steps, environment details and relevant logs. +Use GitHub Issues: the Bug Report template for unexpected behavior or errors, the Feature Request template for new functionality. Include reproduction steps, environment details and relevant logs. The Component you pick sets the issue's `area:` label, and pull requests get theirs from the files they change. + +Report a security weakness through [private vulnerability reporting](https://github.com/HardMax71/Integr8sCode/security/advisories/new) instead of an issue. See [SECURITY.md](SECURITY.md). ## Questions? diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 00000000..be58c118 --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,7 @@ +# Security policy + +Don't report a vulnerability in a public issue. Use [private vulnerability reporting](https://github.com/HardMax71/Integr8sCode/security/advisories/new), so it can be fixed before anyone else sees it. + +Only `main` and the release deployed from it get fixes. There are no maintained release branches. + +How executor pods are isolated and what CI scans for is in the [security docs](https://hardmax71.github.io/Integr8sCode/security/policies/). diff --git a/docs/operations/cicd.md b/docs/operations/cicd.md index b61904eb..2cd64ad6 100644 --- a/docs/operations/cicd.md +++ b/docs/operations/cicd.md @@ -35,11 +35,17 @@ graph LR | Frontend CI | frontend-ci.yml | push and PRs to `main` touching `frontend/`; manual | | docs | docs.yml | push and PRs to `main` touching `docs/`, mkdocs.yml or the workflow; manual | | Generated API | generated-api.yml | push and PRs to `main`; manual | +| Labels | labels.yml | PRs opened or updated (`pull_request_target`), issues opened or edited | +| Label Sync | label-sync.yml | push and PRs to `main` touching the label files or issue forms; Mondays at 03:00; manual | The lint workflows are one job each. Ruff runs `ruff check`. MyPy runs `mypy --strict` with `SECRET_KEY` from the `TEST_SECRET_KEY` secret. Grimp runs `python -m scripts.check_orphan_modules`, which fails on any module nothing imports except `app.main` and `__init__.py` files that hold only a docstring. Security Scanning runs `uvx --no-build --python 3.14 bandit==1.9.4 -r backend/ -x backend/tests/ -ll`. Bandit parses with the Python it runs on, so on anything older it skips the files that use 3.14 syntax such as `except A, B:` without failing. Frontend CI runs ESLint, Stylelint, the Prettier check, `svelte-check --fail-on-warnings` and a production build. The SBOM workflow generates SPDX SBOMs for backend and frontend with `anchore/sbom-action`, scans them with Grype through `anchore/scan-action` (cutoff `high`, `fail-build: false`) and uploads SARIF to the Security tab. The docs workflow installs the backend dependencies, runs `./deploy.sh openapi`, builds with `mkdocs build --strict` and deploys to GitHub Pages on pushes to `main`. +Labels adds `area:` labels. On a PR, `actions/labeler` matches the changed paths against `.github/labeler.yml`; it runs on `pull_request_target` so fork PRs can be labeled too, and it never checks out the PR's code. On an issue, `github/issue-labeler` matches the Component answer of the issue form against `.github/issue-labeler.yml` and adds the label. It never removes one, so labels added by hand stay. + +Label Sync keeps labels as code. `.github/labels.yml` lists every label in the repo, and `crazy-max/ghaction-github-labeler` makes the repo match it: it creates and updates labels, renames one that has `from_name`, and deletes any label the file doesn't list, which also takes it off every issue. On a PR it only prints what it would change. Before that, a `yq` step fails the run when `.github/labeler.yml`, `.github/issue-labeler.yml` or an issue form uses a label the file doesn't declare. A new area needs its label in `.github/labels.yml`, a path rule in `.github/labeler.yml`, an option in both issue forms and a pattern in `.github/issue-labeler.yml`. + Generated API enforces that `docs/reference/openapi.json` and `frontend/src/lib/api/` only ever come from the generators. It runs `./deploy.sh openapi` and `npm run generate:api` on the checked-out code and fails if either leaves a difference, which catches a hand edit and a spec or client someone forgot to regenerate. ## Stack Tests