Skip to content

Emit human-readable domain label on review findings #54

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
JesperSchulz wants to merge 1 commit into
base: main
Choose a base branch
from
+70 −39
Draft

Emit human-readable domain label on review findings #54

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions microsoft/skills/review/al-breaking-changes-review.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ Outcome selection:

## Output

Output conforms to the DO output contract. A populated example:
Output conforms to the DO output contract. Every finding this skill emits MUST set `findings[].domain` to `"Breaking Changes"`. A populated example:

```json
{
Expand All @@ -100,7 +100,8 @@ Output conforms to the DO output contract. A populated example:
"references": [
{ "path": "microsoft/knowledge/breaking-changes/do-not-change-published-procedure-signatures.md" }
],
"confidence": "high"
"confidence": "high",
"domain": "Breaking Changes"
},
{
"id": "microsoft/knowledge/breaking-changes/choose-access-modifiers-deliberately.md",
Expand All @@ -113,7 +114,8 @@ Output conforms to the DO output contract. A populated example:
"references": [
{ "path": "microsoft/knowledge/breaking-changes/choose-access-modifiers-deliberately.md" }
],
"confidence": "medium"
"confidence": "medium",
"domain": "Breaking Changes"
}
],
"suppressed": []
Expand Down
30 changes: 20 additions & 10 deletions microsoft/skills/review/al-code-review.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ For each sub-skill in the worklist, executed one at a time per the discipline ab
1. Invoke the sub-skill with the orchestrator's inputs, passing only the subset each sub-skill declares in its `inputs`.
2. Capture the sub-skill's complete findings-report verbatim and append it to `sub-results`.
3. If the sub-skill's `outcome` is `failed`, stop here for this sub-skill: its findings are not reliable per the DO contract and MUST NOT be copied into the super-skill's top-level `findings[]` or counted in `summary.counts`.
4. Otherwise, append each entry from the sub-skill's `findings[]` to the super-skill's top-level `findings[]`, setting `from-sub-skill` to the sub-skill's `skill.id`. For non-citation findings (those whose `id` is a skill-defined slug rather than a reference path), prefix `id` with `<from-sub-skill>:` to prevent collisions across sub-skills. Other finding fields are preserved.
4. Otherwise, append each entry from the sub-skill's `findings[]` to the super-skill's top-level `findings[]`, setting `from-sub-skill` to the sub-skill's `skill.id` and copying each finding's `domain` field verbatim (preserve it unchanged). For non-citation findings (those whose `id` is a skill-defined slug rather than a reference path), prefix `id` with `<from-sub-skill>:` to prevent collisions across sub-skills. Other finding fields are preserved.

### Agent self-review pass

Expand All @@ -90,6 +90,7 @@ For every candidate the agent identifies in this pass:
- Otherwise the candidate has no BCQuality coverage; emit it as a super-skill agent finding.
2. **Emit agent finding.** Per DO's *Agent findings* rules:
- `from-sub-skill: "agent"` (the super-skill itself produced it)
- `domain: "Agent"` (the display label for super-skill cross-cutting findings)
- `references: []`
- `id` is a skill-defined slug prefixed with `agent:` (for example, `agent:missing-error-handling-on-http-call`).
- `confidence` capped at `medium`.
Expand Down Expand Up @@ -139,7 +140,8 @@ Output conforms to the DO output contract, extended with `sub-results` and `skip
{ "path": "microsoft/knowledge/performance/filter-before-find.md" }
],
"confidence": "high",
"from-sub-skill": "al-performance-review"
"from-sub-skill": "al-performance-review",
"domain": "Performance"
},
{
"id": "community/knowledge/performance/call-setloadfields-before-filters.md",
Expand All @@ -153,7 +155,8 @@ Output conforms to the DO output contract, extended with `sub-results` and `skip
{ "path": "community/knowledge/performance/call-setloadfields-before-filters.md" }
],
"confidence": "high",
"from-sub-skill": "al-performance-review"
"from-sub-skill": "al-performance-review",
"domain": "Performance"
},
{
"id": "microsoft/knowledge/security/use-secrettext-for-credentials.md",
Expand All @@ -168,7 +171,8 @@ Output conforms to the DO output contract, extended with `sub-results` and `skip
{ "path": "microsoft/knowledge/security/use-secrettext-for-credentials.md" }
],
"confidence": "high",
"from-sub-skill": "al-security-review"
"from-sub-skill": "al-security-review",
"domain": "Security"
},
{
"id": "microsoft/knowledge/security/never-hardcode-secrets-in-al.md",
Expand All @@ -182,7 +186,8 @@ Output conforms to the DO output contract, extended with `sub-results` and `skip
{ "path": "microsoft/knowledge/security/never-hardcode-secrets-in-al.md" }
],
"confidence": "medium",
"from-sub-skill": "al-security-review"
"from-sub-skill": "al-security-review",
"domain": "Security"
},
{
"id": "agent:missing-error-handling-on-http-client",
Expand All @@ -195,7 +200,8 @@ Output conforms to the DO output contract, extended with `sub-results` and `skip
},
"references": [],
"confidence": "medium",
"from-sub-skill": "agent"
"from-sub-skill": "agent",
"domain": "Agent"
}
],
"suppressed": [],
Expand All @@ -220,7 +226,8 @@ Output conforms to the DO output contract, extended with `sub-results` and `skip
"references": [
{ "path": "microsoft/knowledge/performance/filter-before-find.md" }
],
"confidence": "high"
"confidence": "high",
"domain": "Performance"
},
{
"id": "community/knowledge/performance/call-setloadfields-before-filters.md",
Expand All @@ -233,7 +240,8 @@ Output conforms to the DO output contract, extended with `sub-results` and `skip
"references": [
{ "path": "community/knowledge/performance/call-setloadfields-before-filters.md" }
],
"confidence": "high"
"confidence": "high",
"domain": "Performance"
}
],
"suppressed": []
Expand All @@ -258,7 +266,8 @@ Output conforms to the DO output contract, extended with `sub-results` and `skip
"references": [
{ "path": "microsoft/knowledge/security/use-secrettext-for-credentials.md" }
],
"confidence": "high"
"confidence": "high",
"domain": "Security"
},
{
"id": "microsoft/knowledge/security/never-hardcode-secrets-in-al.md",
Expand All @@ -271,7 +280,8 @@ Output conforms to the DO output contract, extended with `sub-results` and `skip
"references": [
{ "path": "microsoft/knowledge/security/never-hardcode-secrets-in-al.md" }
],
"confidence": "medium"
"confidence": "medium",
"domain": "Security"
}
],
"suppressed": []
Expand Down
8 changes: 5 additions & 3 deletions microsoft/skills/review/al-error-handling-review.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ Outcome selection:

## Output

Output conforms to the DO output contract. A populated example:
Output conforms to the DO output contract. Every finding this skill emits MUST set `findings[].domain` to `"Error Handling"`. A populated example:

```json
{
Expand All @@ -100,7 +100,8 @@ Output conforms to the DO output contract. A populated example:
"references": [
{ "path": "microsoft/knowledge/error-handling/prefer-errorinfo-for-actionable-errors.md" }
],
"confidence": "high"
"confidence": "high",
"domain": "Error Handling"
},
{
"id": "microsoft/knowledge/error-handling/errortype-internal-vs-client-for-diagnostics.md",
Expand All @@ -113,7 +114,8 @@ Output conforms to the DO output contract. A populated example:
"references": [
{ "path": "microsoft/knowledge/error-handling/errortype-internal-vs-client-for-diagnostics.md" }
],
"confidence": "medium"
"confidence": "medium",
"domain": "Error Handling"
}
],
"suppressed": []
Expand Down
8 changes: 5 additions & 3 deletions microsoft/skills/review/al-events-review.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ Outcome selection:

## Output

Output conforms to the DO output contract. A populated example:
Output conforms to the DO output contract. Every finding this skill emits MUST set `findings[].domain` to `"Events"`. A populated example:

```json
{
Expand All @@ -117,7 +117,8 @@ Output conforms to the DO output contract. A populated example:
"references": [
{ "path": "microsoft/knowledge/events/publish-thin-onbefore-onafter-integration-events.md" }
],
"confidence": "high"
"confidence": "high",
"domain": "Events"
},
{
"id": "microsoft/knowledge/events/use-ishandled-to-make-base-behaviour-overridable.md",
Expand All @@ -130,7 +131,8 @@ Output conforms to the DO output contract. A populated example:
"references": [
{ "path": "microsoft/knowledge/events/use-ishandled-to-make-base-behaviour-overridable.md" }
],
"confidence": "high"
"confidence": "high",
"domain": "Events"
}
],
"suppressed": []
Expand Down
8 changes: 5 additions & 3 deletions microsoft/skills/review/al-interfaces-review.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ Outcome selection:

## Output

Output conforms to the DO output contract. A populated example:
Output conforms to the DO output contract. Every finding this skill emits MUST set `findings[].domain` to `"Interfaces"`. A populated example:

```json
{
Expand All @@ -100,7 +100,8 @@ Output conforms to the DO output contract. A populated example:
"references": [
{ "path": "microsoft/knowledge/interfaces/prefer-interface-over-case-branching.md" }
],
"confidence": "high"
"confidence": "high",
"domain": "Interfaces"
},
{
"id": "microsoft/knowledge/interfaces/set-defaultimplementation-on-enum.md",
Expand All @@ -113,7 +114,8 @@ Output conforms to the DO output contract. A populated example:
"references": [
{ "path": "microsoft/knowledge/interfaces/set-defaultimplementation-on-enum.md" }
],
"confidence": "high"
"confidence": "high",
"domain": "Interfaces"
}
],
"suppressed": []
Expand Down
8 changes: 5 additions & 3 deletions microsoft/skills/review/al-performance-review.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ Outcome selection:

## Output

Output conforms to the DO output contract. A populated example:
Output conforms to the DO output contract. Every finding this skill emits MUST set `findings[].domain` to `"Performance"`. A populated example:

```json
{
Expand All @@ -100,7 +100,8 @@ Output conforms to the DO output contract. A populated example:
"references": [
{ "path": "microsoft/knowledge/performance/filter-before-find.md" }
],
"confidence": "high"
"confidence": "high",
"domain": "Performance"
},
{
"id": "community/knowledge/performance/call-setloadfields-before-filters.md",
Expand All @@ -113,7 +114,8 @@ Output conforms to the DO output contract. A populated example:
"references": [
{ "path": "community/knowledge/performance/call-setloadfields-before-filters.md" }
],
"confidence": "high"
"confidence": "high",
"domain": "Performance"
}
],
"suppressed": []
Expand Down
5 changes: 3 additions & 2 deletions microsoft/skills/review/al-privacy-review.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ Outcome selection:

## Output

Output conforms to the DO output contract. A populated example:
Output conforms to the DO output contract. Every finding this skill emits MUST set `findings[].domain` to `"Privacy"`. A populated example:

```json
{
Expand All @@ -100,7 +100,8 @@ Output conforms to the DO output contract. A populated example:
"references": [
{ "path": "microsoft/knowledge/privacy/strsubstno-prebuild-breaks-error-telemetry-classification.md" }
],
"confidence": "high"
"confidence": "high",
"domain": "Privacy"
}
],
"suppressed": []
Expand Down
8 changes: 5 additions & 3 deletions microsoft/skills/review/al-security-review.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ Outcome selection:

## Output

Output conforms to the DO output contract. A populated example:
Output conforms to the DO output contract. Every finding this skill emits MUST set `findings[].domain` to `"Security"`. A populated example:

```json
{
Expand All @@ -100,7 +100,8 @@ Output conforms to the DO output contract. A populated example:
"references": [
{ "path": "microsoft/knowledge/security/use-secrettext-for-credentials.md" }
],
"confidence": "high"
"confidence": "high",
"domain": "Security"
},
{
"id": "microsoft/knowledge/security/never-hardcode-secrets-in-al.md",
Expand All @@ -113,7 +114,8 @@ Output conforms to the DO output contract. A populated example:
"references": [
{ "path": "microsoft/knowledge/security/never-hardcode-secrets-in-al.md" }
],
"confidence": "medium"
"confidence": "medium",
"domain": "Security"
}
],
"suppressed": []
Expand Down
5 changes: 3 additions & 2 deletions microsoft/skills/review/al-style-review.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ Outcome selection:

## Output

Output conforms to the DO output contract. A populated example:
Output conforms to the DO output contract. Every finding this skill emits MUST set `findings[].domain` to `"Style"`. A populated example:

```json
{
Expand All @@ -97,7 +97,8 @@ Output conforms to the DO output contract. A populated example:
"references": [
{ "path": "microsoft/knowledge/style/apply-approved-label-suffixes.md" }
],
"confidence": "high"
"confidence": "high",
"domain": "Style"
}
],
"suppressed": []
Expand Down
5 changes: 3 additions & 2 deletions microsoft/skills/review/al-ui-review.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ Outcome selection:

## Output

Output conforms to the DO output contract. A populated example:
Output conforms to the DO output contract. Every finding this skill emits MUST set `findings[].domain` to `"Accessibility"`. A populated example:

```json
{
Expand All @@ -97,7 +97,8 @@ Output conforms to the DO output contract. A populated example:
"references": [
{ "path": "microsoft/knowledge/ui/field-tooltips-start-with-specifies-and-end-with-period.md" }
],
"confidence": "high"
"confidence": "high",
"domain": "Accessibility"
}
],
"suppressed": []
Expand Down
5 changes: 3 additions & 2 deletions microsoft/skills/review/al-upgrade-review.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ Outcome selection:

## Output

Output conforms to the DO output contract. A populated example:
Output conforms to the DO output contract. Every finding this skill emits MUST set `findings[].domain` to `"Upgrade"`. A populated example:

```json
{
Expand All @@ -99,7 +99,8 @@ Output conforms to the DO output contract. A populated example:
"references": [
{ "path": "microsoft/knowledge/upgrade/enum-changes-must-be-additive-at-the-end.md" }
],
"confidence": "high"
"confidence": "high",
"domain": "Upgrade"
}
],
"suppressed": []
Expand Down
8 changes: 5 additions & 3 deletions microsoft/skills/review/al-web-services-review.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ Outcome selection:

## Output

Output conforms to the DO output contract. A populated example:
Output conforms to the DO output contract. Every finding this skill emits MUST set `findings[].domain` to `"Web Services"`. A populated example:

```json
{
Expand All @@ -100,7 +100,8 @@ Output conforms to the DO output contract. A populated example:
"references": [
{ "path": "microsoft/knowledge/web-services/set-required-api-page-properties.md" }
],
"confidence": "high"
"confidence": "high",
"domain": "Web Services"
},
{
"id": "microsoft/knowledge/web-services/expose-systemid-as-the-api-key.md",
Expand All @@ -113,7 +114,8 @@ Output conforms to the DO output contract. A populated example:
"references": [
{ "path": "microsoft/knowledge/web-services/expose-systemid-as-the-api-key.md" }
],
"confidence": "high"
"confidence": "high",
"domain": "Web Services"
}
],
"suppressed": []
Expand Down
Loading
Loading