Skip to content

self-development: add matrix-completeness convention to agent rules#1219

Open
kelos-bot[bot] wants to merge 1 commit into
mainfrom
kelos-config-update-20260526-1800
Open

self-development: add matrix-completeness convention to agent rules#1219
kelos-bot[bot] wants to merge 1 commit into
mainfrom
kelos-config-update-20260526-1800

Conversation

@kelos-bot

@kelos-bot kelos-bot Bot commented May 26, 2026

Copy link
Copy Markdown
Contributor

🤖 Kelos Agent @gjkim42

What type of PR is this?

/kind cleanup

What this PR does / why we need it:

Adds one new convention to AGENTS.md (with CLAUDE.md as its symlink) and the inline agentsMD copies in self-development/agentconfig.yaml and self-development/kelos-workers.yaml:

Cover the whole matrix when writing matrix-shaped tests. When a test iterates over a set of resource kinds, agent types, or subcommands (table-driven cases, for _, r := range resources, the agentConfigs parametrization), include every member of that set that supports the behavior under test. If a member is deliberately omitted, leave an inline comment naming the gate (e.g., // task and taskspawner have no create subcommand). Reviewers should not have to ask "why isn't X here?" — the comment should answer it.

Motivating review feedback

This rule is backed by repeated reviewer feedback that agent-authored matrix tests ship with partial coverage and have to be expanded after-the-fact:

  • PR Drop ValidArgsFunction from 'kelos create workspace' #1180 (Drop ValidArgsFunction from 'kelos create workspace') — three rounds of @gjkim42 review on internal/cli/completion_test.go:44 before the test was accepted:

    • 2026-05-21 (APPROVED, but with): "let's add other resources here if there aren't"
    • 2026-05-21 (CHANGES_REQUESTED): "why don't we add other resources like task, taskspawner...?"
    • 2026-05-21 (APPROVED, agent's final answer): "they don't have a create command."

    Had the agent either (a) covered every resource or (b) left an inline // task and taskspawner have no create subcommand comment on the first pass, the round-trip would have collapsed to zero turns.

  • PR Drop removed --enable skills flag from codex entrypoint #1175 (Drop removed --enable skills flag from codex entrypoint) — @gjkim42 on test/e2e/skills_test.go: "this should not be a codex specific e2e test." The agent had to rewrite the test to iterate over agentConfigs after the fact. Same shape: a test that should have been matrix-shaped from the start was shipped agent-specific.

Scope notes

  • The existing open config-update PR self-development: add CRD, e2e, agent-image, and Dockerfile-pin conventions from recent reviews #1162 codifies "parameterize e2e tests across agent types" for the agent-image angle. This PR generalizes the lesson — the same failure mode occurred for CLI resource matrices in Drop ValidArgsFunction from 'kelos create workspace' #1180, so the rule belongs in the cross-cutting test-style block (next to "Avoid vacuous substring assertions") rather than under agent-image conventions.
  • No changes to other self-development/*.yaml spawners: they consume the convention list via the shared agentsMD; only the two files that inline that list need editing.
  • CLAUDE.md is a symlink to AGENTS.md, so a single edit covers both surfaces.
  • No code or behavior change — text-only update to three files.

Which issue(s) this PR is related to:

N/A

Special notes for your reviewer:

N/A

Does this PR introduce a user-facing change?

NONE

Summary by cubic

Add a “cover the whole matrix” testing convention to AGENTS.md and the inline copies in self-development/agentconfig.yaml and self-development/kelos-workers.yaml. Tests that iterate over resources/agents/subcommands must include all supported members or explain omissions inline, improving coverage and reducing review churn.

Written for commit ebf2911. Summary will update on new commits. Review in cubic

Adds a new rule to AGENTS.md and the inline agentsMD copies in
self-development/agentconfig.yaml and self-development/kelos-workers.yaml:

> Cover the whole matrix when writing matrix-shaped tests. When a test
> iterates over resource kinds, agent types, or subcommands, include
> every member of that set that supports the behavior under test. If a
> member is deliberately omitted, leave an inline comment naming the gate.

Motivated by repeated reviewer feedback that agents ship partial-coverage
matrix tests:

- PR #1180 (Drop ValidArgsFunction from kelos create workspace): three
  rounds of CHANGES_REQUESTED on internal/cli/completion_test.go before
  the agent covered the full resource set or named the omission gate.
- PR #1175 (Drop removed --enable skills flag from codex entrypoint):
  reviewer asked the agent to parametrize the new e2e test across the
  agentConfigs matrix instead of writing a codex-only file.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant