[Schema Consistency] Schema Consistency Check - 2026-05-15

[github-actions[bot]]

Overview

Automated audit of the four-way consistency between the workflow JSON schema (pkg/parser/schemas/main_workflow_schema.json), parser/compiler code (pkg/parser, pkg/workflow), reference docs (docs/src/content/docs/reference/**), and real workflow files (.github/workflows/*.md).

Four real inconsistencies surfaced this run, ranging from a missing parser bound check (a latent bug) to a built-in engine that is unreferenced in both the schema and user-facing docs. One additional finding concerns the diff generator itself producing false positives.

Summary

• Inconsistencies found: 4 (plus 1 tooling note)
• Categories analyzed: Schema ↔ Parser, Schema ↔ Docs, Schema ↔ Workflows, Parser ↔ Docs
• Strategy used: Proven (day-of-year 135 mod 10 = 5)
• New strategies recorded: 4 (first run — empty cache)

Critical Issues

1. tracker-id maxLength: 128 is not enforced by the parser

• Schema (pkg/parser/schemas/main_workflow_schema.json:38-44): declares tracker-id as "type": "string", "minLength": 8, "maxLength": 128, "pattern": "^[a-zA-Z0-9_-]+$".
• Parser (pkg/workflow/frontmatter_extraction_metadata.go:86-122): extractTrackerID validates the lower bound (len < 8) and the character set, but never checks the upper bound.
• Impact: A tracker-id of 1000 characters compiles cleanly today, then gets embedded into every footer XML marker and GH_AW_TRACKER_ID env var. It will also fail any downstream consumer that re-validates against the published schema.
• Fix: add if len(trackerID) > 128 { return "", fmt.Errorf(...) } alongside the existing length check.

2. Built-in engines pi and opencode missing from schema description

• Code (pkg/workflow/agentic_engine.go:439-446): registers seven built-in engines — claude, codex, copilot, gemini, opencode, crush, pi.
• Schema ($defs.engine_config.oneOf[0].description): lists only five — 'claude', 'codex', 'copilot', 'gemini', 'crush'. Same shortened list is repeated on engine_config.properties.id.description.
• Docs (docs/src/content/docs/reference/engines.md:15-21): lists six — adds opencode, still omits pi.
• Impact: Users authoring workflows with engine: pi or engine: opencode see no IDE/schema completion or hover help for those values. engine: pi appears entirely undocumented to end users despite being a registered, compiling option.
• Fix options: either (a) update the two engine description strings + engines.md to enumerate all seven, or (b) confirm pi is internal-only and stop registering it as a built-in.

Documentation Gaps

3. 16 schema top-level fields are not referenced in frontmatter.md

The canonical reference docs/src/content/docs/reference/frontmatter.md omits these schema-declared top-level fields:

Full list (most are documented in other pages, but not on the main reference)

Field	Status
check-for-updates	only in frontmatter-full.md
disable-model-invocation	only in frontmatter-full.md
experiments	experiments-specification.md, frontmatter-full.md
github-app	safe-outputs.md, faq.md, frontmatter-full.md
import-schema	imports.md, frontmatter-full.md, glossary.md
infer	only in frontmatter-full.md
inlined-imports	imports.md, faq.md, frontmatter-full.md
max-runs	engines.md, glossary.md, frontmatter-full.md
mcp-servers	tools.md, faq.md, etc.
models	model-alias-specification.md, awf-reflect.md, frontmatter-full.md
rate-limit	frontmatter-full.md, weekly-update blog post
run-install-scripts	only in frontmatter-full.md
sandbox	sandbox.md, mcp-gateway.md, frontmatter-full.md
secret-masking	frontmatter-hash-specification.md, frontmatter-full.md
tracker-id	frontmatter-hash-specification.md, frontmatter-full.md
user-rate-limit	cost-management.md, rate-limiting-controls.md, frontmatter-full.md

• Impact: Users who treat frontmatter.md as the reference (its name implies it is) will not discover these knobs. frontmatter-full.md is auto-generated and not great as a discovery surface.
• Fix: add a short section or table row for each missing field to frontmatter.md, pointing at the specialised reference page where one exists.

Tooling Note (Schema Diff Generator)

4. field_gaps.in_used_not_schema is full of false positives

All 30 entries (affected_sections, has_agent_blocks, is_smoke_test, model_adequacy, prompt_phases, setup_step_proposal, task_type, verdict, score, size, ...) come from a single workflow — .github/workflows/daily-subagent-optimizer.md — and every match lives inside fenced markdown code blocks (template prompts the agent is told to output), not in the YAML frontmatter.

is_smoke_test → .github/workflows/daily-subagent-optimizer.md
    ```
    is_smoke_test: yes/no
    prompt_phases: (number)

• Impact: This category is currently noise — future runs will keep flagging the same template strings forever.
• Fix: the generator that produces /tmp/gh-aw/agent/schema-diff.json should skip lines inside fenced code blocks (``` ... ```) when extracting used_in_workflows, or only parse the top YAML document delimited by --- markers.

Schema ↔ Parser Confirmed-Consistent

While auditing, the following were verified and are consistent (recording them so future runs can skip re-checking):

• Schema engine.default: "copilot" matches constants.DefaultEngine (pkg/constants/engine_constants.go:32).
• Schema engine.version accepts string|number; parser uses stringutil.ParseVersionValue to normalise both. ✓
• Schema tracker-id.minLength: 8 and pattern: ^[a-zA-Z0-9_-]+$ are enforced in extractTrackerID. ✓

Recommendations

1. Add the missing maxLength check in extractTrackerID — a 6-line code change closes the loophole.
2. Decide on pi and opencode: either fully document and add them to schema descriptions, or hide them from the registry. Today is the worst of both worlds — they work but are invisible.
3. Backfill frontmatter.md with brief stubs + cross-links for the 16 fields that only exist in frontmatter-full.md or feature-specific pages.
4. Tighten the schema-diff generator so used_in_workflows reflects only actual frontmatter, not code-block content. This unblocks future runs from focusing on the real gaps.

Strategy Performance

Strategy	Findings	Effectiveness	Reuse
Schema field documentation coverage in frontmatter.md	16 (mostly nav gaps)	Medium	Yes
Engine registry vs schema/docs	2 critical	High	Yes
Schema string constraints vs Go extraction	1 (latent bug)	High	Yes — expand to all maxLength/pattern fields
In-frontmatter vs code-block detection	1 (tooling)	High	Yes

Strategies persisted to /tmp/gh-aw/cache-memory/strategies.json.

Next Steps

• Add maxLength: 128 check to extractTrackerID in pkg/workflow/frontmatter_extraction_metadata.go
• Update engine_config description strings to list all seven built-in engines (or drop pi/opencode from the registry)
• Add pi and opencode rows to docs/src/content/docs/reference/engines.md if both are user-facing
• Add short reference entries for 16 fields missing from docs/src/content/docs/reference/frontmatter.md
• Patch the schema-diff generator to skip content inside fenced code blocks

References:

• §25903941245

Generated by ✅ Schema Consistency Checker · ● 8.8M · ◷

• expires on May 16, 2026, 6:36 AM UTC

[github-actions[bot]]

This discussion has been marked as outdated by Schema Consistency Checker.

A newer discussion is available at Discussion #32580.