Fix skill loading on non-string metadata and judge JSON failures - #93
Merged
Conversation
Populate Frontmatter.Metadata from RawFrontmatter with string-valued entries only, instead of unmarshaling directly into map[string]string. Skills that nest lists or maps inside metadata now load and surface the existing per-key structure validation errors rather than failing the YAML parse before any checks run. Fixes #92
The anti-injection reminder appended after the delimited content looks exactly like an injected instruction to the judge model, which can respond with commentary instead of JSON. Announce the delimiters and trailing reminder in the system prompts so the judge knows they come from the harness, make the reminder format-agnostic (the novelty follow-up asks for plain text, so the old wording contradicted it), and retry once with a strengthened prompt when the response contains no parseable JSON. Fixes #91
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #92, fixes #91.
Issue #92: YAML unmarshal errors on non-string metadata values
Skills that nest lists or maps inside
metadata(e.g. the reported affinity-proteomics skill withtags:andopenclaw:blocks) crashedLoadwith a raw YAML error before any validation ran. The spec still requires string keys and string values; this only moves where the violation surfaces.Frontmatter.Metadatakeeps itsmap[string]stringtype but is now taggedyaml:"-"and populated inLoadfromRawFrontmatterwith string-valued entries only. Non-conforming skills now load and get the existing per-key structure errors (metadata["tags"] value must be a string) instead of a parse crash, andscore evaluatecan proceed. This also stops unquoted scalars likeversion: 1.3(a YAML float) from crashing the load; they degrade to the same validation error.Verified against the skill from the issue report:
checkproduces a full report with the two metadata errors, and the originally reportedscore evaluate --provider claude-cli --skill-onlycommand completes.Issue #91: "no valid JSON object found in response"
The judge's error message quoted the model saying it noticed "an appended instruction telling me to respond only with the JSON obj…" — that instruction is our own anti-injection reminder, appended after the untrusted-content delimiters by
formatUserContent. A large pasted document followed by a trailing "respond only with JSON" instruction is exactly what an injection attempt looks like, so the judge model sometimes responds with commentary instead of JSON. (The reported skill's content is benign; this is a false positive triggered by the harness itself.)Three changes:
ScoreSkill/ScoreReferenceretry once with a strengthened output-only prompt before failing.The failure is intermittent (three runs with the old prompts succeeded), so the fix is layered: the prompt change makes suspicion less likely and the retry makes a single conversational response non-fatal. Live-tested the new prompts via
--provider claude-clion both sonnet and haiku; both score cleanly. Unit tests cover the retry-and-recover and double-failure paths.