Community agent and starter-kit security scans and PR gates - #132
Draft
yousefi-msft wants to merge 13 commits into
Draft
Community agent and starter-kit security scans and PR gates#132yousefi-msft wants to merge 13 commits into
yousefi-msft wants to merge 13 commits into
Conversation
|
You are seeing this message because GitHub Code Scanning has recently been set up for this repository, or this pull request contains the workflow file for the Code Scanning tool. What Enabling Code Scanning Means:
For more information about GitHub Code Scanning, check out the documentation. |
github-actions
Bot
added
1p-contribution
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Comment on lines
+335
to
+348
| - name: Run starter-kit validation against PR data | ||
| id: starter_kits | ||
| if: steps.integration.outcome == 'success' | ||
| continue-on-error: true | ||
| run: | | ||
| mapfile -t added_kits < "$RUNNER_TEMP/added-kits.txt" | ||
| args=(--repo-root "$GITHUB_WORKSPACE/evaluation" --added-kits) | ||
| if (( ${#added_kits[@]} )); then | ||
| args+=("${added_kits[@]}") | ||
| fi | ||
| python trusted/.github/scripts/validate_starter_kits.py "${args[@]}" \ | ||
| | tee "$RUNNER_TEMP/shadow-validation/starter-kits.txt" | ||
|
|
||
| - name: Run removal-impact validation without write access |
Comment on lines
+348
to
+369
| - name: Run removal-impact validation without write access | ||
| id: removal_impact | ||
| if: steps.integration.outcome == 'success' | ||
| continue-on-error: true | ||
| env: | ||
| BASE_SHA: ${{ steps.target.outputs.base-sha }} | ||
| IS_FORK: ${{ steps.target.outputs.is-fork }} | ||
| PR_NUMBER: ${{ steps.target.outputs.number }} | ||
| run: | | ||
| args=( | ||
| --repo-root "$GITHUB_WORKSPACE/evaluation" | ||
| --base-sha "$BASE_SHA" | ||
| --pr-number "$PR_NUMBER" | ||
| --repo "$GITHUB_REPOSITORY" | ||
| ) | ||
| if [[ "$IS_FORK" == "true" ]]; then | ||
| args+=(--is-fork) | ||
| fi | ||
| python trusted/.github/scripts/check_agent_removal_impact.py "${args[@]}" \ | ||
| | tee "$RUNNER_TEMP/shadow-validation/removal-impact.txt" | ||
|
|
||
| - name: Summarize report-only results |
Comment on lines
+369
to
+407
| - name: Summarize report-only results | ||
| if: always() | ||
| env: | ||
| PR_NUMBER: ${{ steps.target.outputs.number }} | ||
| PR_URL: ${{ steps.target.outputs.url }} | ||
| PR_TITLE: ${{ steps.target.outputs.title }} | ||
| HEAD_REPOSITORY: ${{ steps.target.outputs.head-repository }} | ||
| IS_FORK: ${{ steps.target.outputs.is-fork }} | ||
| CHANGED_COUNT: ${{ steps.target.outputs.changed-count }} | ||
| INTEGRATION_OUTCOME: ${{ steps.integration.outcome }} | ||
| CATALOG_OUTCOME: ${{ steps.catalog.outcome }} | ||
| SCHEMA_OUTCOME: ${{ steps.schemas.outcome }} | ||
| STARTER_KIT_OUTCOME: ${{ steps.starter_kits.outcome }} | ||
| REMOVAL_OUTCOME: ${{ steps.removal_impact.outcome }} | ||
| REPOSITORY_URL: ${{ github.server_url }}/${{ github.repository }} | ||
| VALIDATOR_REVISION: ${{ github.sha }} | ||
| run: | | ||
| python .github/scripts/validate_starter_kits.py --repo-root . | ||
| # Report only: this workflow did not change or publish a check to the target PR. | ||
| python trusted/.github/scripts/render_ci_summary.py shadow \ | ||
| --pr-number "$PR_NUMBER" \ | ||
| --pr-url "$PR_URL" \ | ||
| --pr-title "$PR_TITLE" \ | ||
| --head-repository "$HEAD_REPOSITORY" \ | ||
| --is-fork "$IS_FORK" \ | ||
| --changed-count "$CHANGED_COUNT" \ | ||
| --integration-outcome "$INTEGRATION_OUTCOME" \ | ||
| --integration-log "$RUNNER_TEMP/shadow-validation/integration.txt" \ | ||
| --catalog-outcome "$CATALOG_OUTCOME" \ | ||
| --catalog-report "$RUNNER_TEMP/shadow-validation/catalog.json" \ | ||
| --schema-outcome "$SCHEMA_OUTCOME" \ | ||
| --schema-log "$RUNNER_TEMP/shadow-validation/schema-tests.txt" \ | ||
| --starter-kit-outcome "$STARTER_KIT_OUTCOME" \ | ||
| --starter-kit-log "$RUNNER_TEMP/shadow-validation/starter-kits.txt" \ | ||
| --removal-outcome "$REMOVAL_OUTCOME" \ | ||
| --removal-log "$RUNNER_TEMP/shadow-validation/removal-impact.txt" \ | ||
| --repository-url "$REPOSITORY_URL" \ | ||
| --revision "$VALIDATOR_REVISION" | ||
|
|
||
| - name: Upload shadow validation evidence |
❌ Automated Check Results — 8 issue(s) foundAll issues are listed below. Please address all of them before requesting human review.
Once all issues are resolved, push a new commit to re-trigger this check.
|
yousefi-msft
marked this pull request as draft
September 1, 2026 04:03
yousefi-msft
force-pushed
the
users/yousefi-msft/community-review-pipeline
branch
from
September 2, 2026 14:26
008ba6f to
8b57982
Compare
github-actions
Bot
removed
1p-contribution
github-actions
Bot
added
1p-contribution
❌ Automated Check Results — 8 issue(s) foundAll issues are listed below. Please address all of them before requesting human review.
Once all issues are resolved, push a new commit to re-trigger this check.
|
github-actions
Bot
added
1p-contribution
❌ Automated Check Results — 9 issue(s) foundAll issues are listed below. Please address all of them before requesting human review.
Once all issues are resolved, push a new commit to re-trigger this check.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Harden catalog contribution validation & site-wide security automation
Refactors the PR validator into a modular, content-inspecting rule engine and
adds layered, full-site security scanning. New analyzers and the weekly scan are
report-only during rollout so they do not block internal catalog contributions.
Validation engine — refactor & new architecture
validate_pr.pymonolith into a modular package(
.github/scripts/catalog_validation/):runner.py(orchestrator),contribution.py(1p/3p + docs-only/image classification),contributor_scope.py(POL-021 trusted-path gate),structural.py,schema_checks.py,policy_checks.py,documentation.py,schemas.py,findings.py.validate_pr.pyis now a thin CLI wrapper..github/scripts/rules/):base.py(Rule/Finding/Severity/Scope),
registry.py(rule discovery + run +waiver/baseline handling), and individual rule modules
pol_008,pol_014–pol_020,tag_001,tag_002.with text extensions) via new
content_sniffer.py.source-allowlist, tag taxonomy, etc.).
.github, schemas,.auto-registry,.vscode, utilities, root config).New security / analysis scripts
contact_network_validator.py+.github/policy/contact-network.json—validates publisher contact domains.
url_reputation_audit.py— URLhaus / PhishTank reputation lookups.image_inspector.py,dockerfile_parser.py,list_base_images.py+.github/policy/base-images.yaml— base-image allowlist & inspection.spellcheck_prose.py— non-blocking prose spellcheck (SPELL-001).compute_tags.py+.github/policy/tag-taxonomy.yaml— tagcomputation / taxonomy.
generate_baseline.py+.github/policy/baseline.json— ratchet baselinefor existing-catalog drift.
generate_rule_docs.py,render_ci_summary.py,run_validation_workflow.py,source_locations.py.source-allowlist.yaml,waivers.yaml,codespell-ignore.txt,.github/codeql/codeql-config.yml,.github/requirements-ci.txt,.config/dotnet-tools.json.Workflows
code-scan.yml(CodeQL + DevSkim + ApplicationInspector),weekly-deep-scan.yml(every-rule × every-file + base-image CVE + URLreputation, report-only),
dependency-review.yml.pr-review.yml(author-permission resolution, dual-checkout,spellcheck, TruffleHog verified + audit, render summary),
validate-everything.yml(full-repo + optional PR inspection),unit-tests.yml,validate-agent-schemas.yml,validate-starter-kit-schema.yml,validate-starter-kits.yml,update-registry.yml,auto-merge-on-approval.yml,check-agent-removal-impact.yml.dependabot.yml(pip + actions ecosystems) andCODEOWNERS.Schemas & docs
docs/schemas/(agent-schema-v2,tool-definition, metadata, registry, starter-kit + starter-kit-registry, new
common-schema.json).validation-rules.md,validation-rules.csv,validation-workflow-testing.md; updates toCONTRIBUTING.md,SECURITY.md,README.md,agent-authoring-guide.md.tool.yamlbase-image updates for opensta,rtl-ivlg-linter, rtl-yosys-syn, xyce;
kit.jsonfix forchip-synthesis-timing.
Tests added (~30 files; 439 passing / 3 skipped on Linux CI)
.github/tests/rules/):conftest.py(seeds real.github/policy/),test_pol_008,test_pol_014–test_pol_020,test_tag_001,test_tag_002,test_registry,test_compute_tags,test_content_sniffer.test_security_automation.py,test_schema_security.py,test_url_reputation_audit.py,test_contact_network_validator.py,test_validate_pr_pol021.py(maintainer-exemption / trusted-path).
test_generators.py,test_render_ci_summary.py,test_run_validation_workflow.py,test_source_locations.py,test_spellcheck_prose.py,test_validation_inventory.py.test_check_agent_removal_impact.py,test_validate_starter_kits.py.Net effect
Moves from a single extension-based validator to a modular, content-inspecting
rule engine with full-site (report-only) scanning, maintainer-scoped trusted
paths, and broad test coverage — while keeping new analyzers non-blocking so
internal catalog contributions are not gated.