Skip to content

Fix skill loading on non-string metadata and judge JSON failures - #93

Merged
dacharyc merged 2 commits into
mainfrom
fix/issue-91-92-load-and-judge
Aug 24, 2026
Merged

Fix skill loading on non-string metadata and judge JSON failures#93
dacharyc merged 2 commits into
mainfrom
fix/issue-91-92-load-and-judge

Conversation

@dacharyc

Copy link
Copy Markdown
Member

Fixes #92, fixes #91.

Issue #92: YAML unmarshal errors on non-string metadata values

Skills that nest lists or maps inside metadata (e.g. the reported affinity-proteomics skill with tags: and openclaw: blocks) crashed Load with a raw YAML error before any validation ran. The spec still requires string keys and string values; this only moves where the violation surfaces.

Frontmatter.Metadata keeps its map[string]string type but is now tagged yaml:"-" and populated in Load from RawFrontmatter with string-valued entries only. Non-conforming skills now load and get the existing per-key structure errors (metadata["tags"] value must be a string) instead of a parse crash, and score evaluate can proceed. This also stops unquoted scalars like version: 1.3 (a YAML float) from crashing the load; they degrade to the same validation error.

Verified against the skill from the issue report: check produces a full report with the two metadata errors, and the originally reported score evaluate --provider claude-cli --skill-only command completes.

Issue #91: "no valid JSON object found in response"

The judge's error message quoted the model saying it noticed "an appended instruction telling me to respond only with the JSON obj…" — that instruction is our own anti-injection reminder, appended after the untrusted-content delimiters by formatUserContent. A large pasted document followed by a trailing "respond only with JSON" instruction is exactly what an injection attempt looks like, so the judge model sometimes responds with commentary instead of JSON. (The reported skill's content is benign; this is a false positive triggered by the harness itself.)

Three changes:

  • The judge system prompts now state up front that the delimiters and the trailing reminder in the user message come from the evaluation harness, so the judge expects them.
  • The reminder's closing sentence is format-agnostic ("respond only in the format requested by the system prompt"). The old wording hardcoded JSON, which directly contradicted the novelty follow-up prompt's "plain text only, no JSON".
  • When a response contains no parseable JSON, ScoreSkill/ScoreReference retry once with a strengthened output-only prompt before failing.

The failure is intermittent (three runs with the old prompts succeeded), so the fix is layered: the prompt change makes suspicion less likely and the retry makes a single conversational response non-fatal. Live-tested the new prompts via --provider claude-cli on both sonnet and haiku; both score cleanly. Unit tests cover the retry-and-recover and double-failure paths.

Populate Frontmatter.Metadata from RawFrontmatter with string-valued
entries only, instead of unmarshaling directly into map[string]string.
Skills that nest lists or maps inside metadata now load and surface the
existing per-key structure validation errors rather than failing the
YAML parse before any checks run.

Fixes #92
The anti-injection reminder appended after the delimited content looks
exactly like an injected instruction to the judge model, which can
respond with commentary instead of JSON. Announce the delimiters and
trailing reminder in the system prompts so the judge knows they come
from the harness, make the reminder format-agnostic (the novelty
follow-up asks for plain text, so the old wording contradicted it), and
retry once with a strengthened prompt when the response contains no
parseable JSON.

Fixes #91
@dacharyc dacharyc mentioned this pull request Aug 24, 2026
@dacharyc
dacharyc merged commit 3c17b99 into main Aug 24, 2026
3 checks passed
@dacharyc
dacharyc deleted the fix/issue-91-92-load-and-judge branch August 24, 2026 03:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

yaml: unmarshal errors no valid JSON object found in response

1 participant