Refresh model catalog to CLI-reachable IDs and prune dead fallback chains#1444
Conversation
🟠 Impact Analysis — PR #1444Risk tier: 🟠 HIGH 📊 Summary
🎯 Risk Factors
📦 Modules Affectedci-workflows (1 file)
copilot-config (1 file)
root (3 files)
squad-cli (9 files)
squad-sdk (8 files)
squad-state (2 files)
templates (2 files)
tests (10 files)
This report is generated automatically for every PR. See #733 for details. |
🛫 PR Readiness Check
PR Scope: 📦🔧 Mixed (product + infrastructure)
|
| Status | Check | Details |
|---|---|---|
| ❌ | Single commit | 4 commits — consider squashing before review |
| ✅ | Not in draft | Ready for review |
| ❌ | Branch up to date | dev is 6 commit(s) ahead — rebase recommended |
| ❌ | Copilot review | No Copilot review yet — it may still be processing |
| ✅ | Changeset present | Changeset file found |
| ✅ | Scope clean | |
| ✅ | No merge conflicts | No merge conflicts |
| ✅ | Copilot threads resolved | 0 active Copilot thread(s) resolved (1 outdated skipped) |
| ❌ | CI passing | 2 check(s) failing: test, samples-build |
Files Changed (36 files, +780 −288)
| File | +/− |
|---|---|
.changeset/refresh-model-catalog-1080.md |
+13 −0 |
.copilot/skills/model-selection/SKILL.md |
+11 −11 |
.github/agents/squad.agent.md |
+1 −0 |
.squad-templates/model-selection-reference.md |
+10 −10 |
.squad-templates/ralph-circuit-breaker.md |
+3 −5 |
.squad/skills/economy-mode/SKILL.md |
+6 −6 |
.squad/skills/model-selection/SKILL.md |
+3 −3 |
packages/squad-cli/src/cli-entry.ts |
+1 −1 |
packages/squad-cli/src/cli/commands/economy.ts |
+2 −2 |
packages/squad-cli/src/cli/core/command-help.ts |
+1 −1 |
packages/squad-cli/src/cli/shell/error-messages.ts |
+1 −1 |
packages/squad-cli/templates/model-selection-reference.md |
+10 −10 |
packages/squad-cli/templates/ralph-circuit-breaker.md |
+3 −5 |
packages/squad-cli/templates/skills/economy-mode/SKILL.md |
+6 −6 |
packages/squad-cli/templates/skills/model-selection/SKILL.md |
+3 −3 |
packages/squad-cli/templates/skills/squad-commands/SKILL.md |
+303 −0 |
packages/squad-sdk/src/config/models.ts |
+93 −109 |
packages/squad-sdk/src/config/schema.ts |
+4 −4 |
packages/squad-sdk/src/runtime/config.ts |
+3 −0 |
packages/squad-sdk/src/runtime/constants.ts |
+9 −8 |
packages/squad-sdk/templates/model-selection-reference.md |
+10 −10 |
packages/squad-sdk/templates/ralph-circuit-breaker.md |
+3 −5 |
packages/squad-sdk/templates/skills/economy-mode/SKILL.md |
+6 −6 |
packages/squad-sdk/templates/skills/model-selection/SKILL.md |
+3 −3 |
templates/model-selection-reference.md |
+10 −10 |
templates/ralph-circuit-breaker.md |
+3 −5 |
test/agents.test.ts |
+8 −9 |
test/catalog-refresh.test.ts |
+158 −0 |
test/compat-v041.test.ts |
+28 −6 |
test/config-integration.test.ts |
+14 −1 |
test/config-schema.test.ts |
+3 −3 |
test/economy-mode.test.ts |
+21 −21 |
test/integration.test.ts |
+1 −1 |
test/model-fallback.test.ts |
+4 −4 |
test/models.test.ts |
+15 −15 |
test/telemetry-auto-wiring.test.ts |
+7 −4 |
Total: +780 −288
This check runs automatically on every push. Fix any ❌ items and push again.
See CONTRIBUTING.md and PR Requirements for details.
|
From Earl's Squad: heads-up on the red The only failing suite is
Root cause looks like Astro dependency drift in Why it's not from this change:
Likely fixed on |
There was a problem hiding this comment.
Pull request overview
Updates the Squad SDK’s seed model catalog and default fallback behavior to use only GitHub Copilot CLI–reachable model IDs, removing deprecated IDs that would otherwise waste fallback/retry attempts. This also introduces an optional cost-ceiling classification (githubCategory) on catalog entries and refreshes affected tests/config defaults accordingly.
Changes:
- Refresh
MODEL_CATALOGto 13 currently CLI-reachable model IDs; prune dead IDs from default and runtime fallback chains and economy-mode mappings. - Add optional
githubCategory(lightweight/versatile/powerful) toModelInfoas a cost axis distinct fromtier. - Update/add tests (including new catalog invariants) to align with the refreshed catalog and optional pricing behavior.
Reviewed changes
Copilot reviewed 14 out of 14 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| test/telemetry-auto-wiring.test.ts | Adjusts pricing assertions to allow optional pricing while validating well-formed pricing when present. |
| test/models.test.ts | Updates catalog and fallback-chain expectations for the refreshed model IDs. |
| test/model-fallback.test.ts | Updates fallback behavior tests to match new premium/standard/fast chains. |
| test/integration.test.ts | Updates integration expectation for premium fallback chain head. |
| test/economy-mode.test.ts | Updates economy-mode downgrade map expectations (e.g., standard/fast mapping changes). |
| test/config-schema.test.ts | Updates default config model expectations to refreshed defaults. |
| test/config-integration.test.ts | Updates integration expectation for config default model. |
| test/compat-v041.test.ts | Updates compatibility assertions for refreshed model availability and chains. |
| test/catalog-refresh.test.ts | Adds invariants ensuring chains reference only catalog IDs and exclude known dead IDs. |
| test/agents.test.ts | Updates per-agent resolution fallback expectations to the new chains. |
| packages/squad-sdk/src/runtime/constants.ts | Refreshes runtime default model and tier fallback chains to reachable IDs. |
| packages/squad-sdk/src/config/schema.ts | Refreshes SDK config defaults (default model and tier lists). |
| packages/squad-sdk/src/config/models.ts | Refreshes catalog + fallback chains; adds githubCategory; updates economy map and preserves optional pricing behavior. |
| .changeset/refresh-model-catalog-1080.md | Adds changeset documenting the catalog refresh and githubCategory addition. |
…uard) Add gpt-5 to DEAD_MODEL_IDS (verified absent from the Copilot models API and the public github/docs pricing YAML). Add a positive invariant asserting the MODEL_CATALOG ID set exactly equals the expected curated 13-ID seed, so any unexpected reintroduction fails — including live-but-dropped-from-seed IDs such as claude-opus-4.5 (still GA + priced upstream), which are intentionally NOT listed as dead. Tighten the DEAD_MODEL_IDS doc comment accordingly. Addresses the PR bradygaster#1444 review feedback. Refs bradygaster#1080 Refs bradygaster#1183 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…backs Replace the model catalog with the 13 GitHub Copilot CLI-reachable models (copilot-cli integration set) and remove IDs no longer offered (gpt-4.1, gpt-5, gpt-5.1*, gpt-5.2*, gemini-3-pro-preview, claude-sonnet-4, claude-opus-4.5, claude-opus-4.6-fast). Update runtime + SDK fallback chains, schema defaults, and the economy-mode map to real IDs, fixing routing that referenced retired models. Add an optional githubCategory cost-ceiling field (lightweight/versatile/ powerful) sourced from the models API model_picker_category, kept as a separate axis from the existing quality tier. No hardcoded per-token pricing is added for new entries and no included/zero-credit flag is introduced. Add catalog-refresh invariant tests: no dead IDs in chains; every chain ID exists in the catalog; gpt-5 is treated as dead (verified absent from the Copilot models API and the public github/docs pricing YAML); and the MODEL_CATALOG ID set exactly equals the expected curated 13-ID seed so any unexpected reintroduction fails. Update existing model/economy/config tests. Refs bradygaster#1080 Refs bradygaster#1183 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
69364df to
f5479b0
Compare
|
@primetimetank21 — flagging a synergy with your catalog refresh: PR #1448 (per-agent context tier: |
tamirdresher
left a comment
tamirdresher
left a comment
There was a problem hiding this comment.
Approving. Catalog and fallback chains are consistent, and the invariant test is solid. The only red check is the pre-existing Astro docs build failure, which is unrelated to this change.
Non-blocking follow-up: model-selection-reference.md is now stale relative to the refreshed catalog — worth a docs pass in a separate PR.
@tamirdresher -- See #1445 (comment). TLDR is that it composes nicely! 😄 |
|
I did a focused security + correctness pass on #1444. Security verdict: PASS — no high-confidence exploitable/security-sensitive issues found. This PR does not introduce secrets, command execution, auth/network changes, or unsafe untrusted-input handling. Correctness note before merge: the code/catalog side looks consistent, but the shipped prompt/template assets still appear to reference now-dead model IDs that this PR removes from the runtime catalog/fallback chains — examples include Why this matters: generated squads/agents read those model-selection reference assets, so even after this catalog refresh lands, prompt guidance can still steer real spawns toward unreachable IDs. Suggested fix before merge, or as a tightly-linked follow-up if maintainers prefer keeping #1444 code-only:
So my current recommendation is: security looks clear; catalog/runtime code looks good; please address or explicitly split the stale-template cleanup before treating #1444 as fully complete. |
Ok, looking into this now. Will have updates later! 😄 |
…refreshed model catalog Addresses the reviewer correctness note on PR bradygaster#1444: shipped prompt/template assets still referenced model IDs removed from the runtime catalog by the catalog-refresh commit (f5479b0), meaning generated squads could steer real agent spawns toward unreachable IDs. Fixes all occurrences across canonical sources; sync scripts propagate the changes to the 10 auto-generated copies. Template/skill canonical source fixes (4 sources → 10 synced copies): - .squad-templates/model-selection-reference.md: claude-opus-4.5→4.6 (Visual role + Premium chain + spawn example), remove claude-sonnet-4 from Standard chain, remove gpt-5.1-codex-mini+gpt-4.1 from Fast chain, gemini-3-pro-preview→gemini-2.5-pro (analytical diversity switch), Valid-models section refreshed to 13-ID catalog - .squad-templates/ralph-circuit-breaker.md: remove gpt-4.1 from multiplier table, OPEN-state fallback list, JSON/PS state examples, and config reference table - .squad/skills/model-selection/SKILL.md: remove claude-opus-4.6-fast+claude-opus-4.5 from Premium chain, claude-sonnet-4 from Standard chain, gpt-5.1-codex-mini+gpt-4.1 from Fast chain (fallback chains section only — lines 119-121) - .squad/skills/economy-mode/SKILL.md: gpt-4.1→gpt-5-mini throughout economy table, claude-opus-4.5→claude-opus-4.6 (Normal Mode premium row), remove "Prefer gpt-4.1 over gpt-5-mini" paragraph Manual edit (outside sync path): - .copilot/skills/model-selection/SKILL.md: same chain fixes as model-selection SKILL plus gemini-3-pro-preview→gemini-2.5-pro, claude-opus-4.5→4.6 (Visual role + task table), gpt-5.2-codex→gpt-5.3-codex (Layer 1 example + Example 3), Valid-models section refreshed CLI user-facing display string fixes (4 files, dead token only): - packages/squad-cli/src/cli/commands/economy.ts: gpt-4.1→gpt-5-mini (×2, economy downgrade display — matches ECONOMY_MODEL_MAP already updated in f5479b0) - packages/squad-cli/src/cli/shell/error-messages.ts: gpt-4.1→gpt-5-mini (rate-limit recovery suggestion) - packages/squad-cli/src/cli-entry.ts: claude-sonnet-4→claude-sonnet-4.6 (start help example) - packages/squad-cli/src/cli/core/command-help.ts: claude-sonnet-4→claude-sonnet-4.6 (start help example) Test: expand DEAD_MODEL_IDS in test/catalog-refresh.test.ts to include claude-opus-4.5 and claude-opus-4.6-fast (live-but-dropped-from-seed for runtime chains, but dead in practice for template guidance). Add new `describe` block "template asset catalog invariants" that globs .squad-templates/**/*.md, .squad/skills/**/*.md, and .copilot/skills/**/*.md at test time, scanning each for dead IDs with a (?![-.\d]) lookahead to prevent gpt-5 from matching gpt-5-mini or gpt-5.4. All 7 catalog-refresh invariants GREEN. `npm run lint` clean. Refs bradygaster#1080, bradygaster#1183 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
🏗️ Architectural Review
Automated architectural review — informational only. |
|
From Earl's Squad: @tamirdresher — thanks for the careful pass. You're right that the correctness gap was in the shipped prompt/template assets, and I've addressed both of your suggested fixes in 1. Shipped assets synced to the refreshed catalog. Updated the canonical model-selection / economy / Ralph template + skill sources and re-ran the sync scripts so all generated copies match:
Dead IDs were remapped to live catalog entries ( 2. Added the invariant you asked for. Note on one extra file. Running the sync scripts also created Validation: |
|
Good catalog refresh — removing 12 dead model IDs that burn fallback slots is the right call. Two items to address before merge: 1. Compat passthrough is now untested (blocking) 2. \globSync\ requires Node 22 (blocking) Advisory (non-blocking):
|
…, and catalog warning
B1: add describe block in test/compat-v041.test.ts asserting getNextFallback
for removed IDs ('gpt-4.1', 'claude-sonnet-4') resolves to a live catalog
model and does not throw.
A1: add inline comments in packages/squad-sdk/src/config/models.ts and
packages/squad-sdk/src/runtime/constants.ts explaining why claude-sonnet-4.6
precedes claude-sonnet-5 in the standard fallback chain (established default
with known pricing vs newer model with no tracked pricing yet).
A3: in validateConfigDetailed (packages/squad-sdk/src/runtime/config.ts),
push a non-blocking warning when config.models.defaultModel is a non-empty
string not present in MODEL_CATALOG. Existing configs with dead IDs still
load — a warning is emitted, not an error. Test added to
test/config-integration.test.ts.
Changeset: fold A3 mention into .changeset/refresh-model-catalog-1080.md.
Addresses reviewer feedback on bradygaster#1444.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…only Both claude-sonnet-4.6 and claude-sonnet-5 carry identical cost: 5 / speed: 7 metadata. The previous comment overstated the pricing angle. The sole code-backed reason for 4.6 leading the standard chain is that it is MODELS.DEFAULT (the established primary); sonnet-5 was newly added in this catalog refresh and is first fallback. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
From Earl's Squad: @tamirdresher — thanks for the thorough review! Here's where each item lands: B1 (compat passthrough untested) — fixed. You were right that dropping the old B2 (globSync / Node 20 compatibility) — respectful pushback, no change needed. All three A1 (sonnet ordering) — comment added. Both A2 (SKILL.md unmentioned) — noted, PR description updated. That file is a sync-drift fix: the other template targets already had A3 (silent fallthrough for removed models) — warning added. All 123 tests pass and lint is clean. Fixes are in commits |
|
Thanks @dat-intellis for the thoughtful review comments and suggestions. The current changes are validated and the remaining suggestions are non-blocking, so I'm merging this PR as-is. @primetimetank21, could you please follow up with a separate PR covering the suggested catalog/documentation updates?
Keeping those updates separate will let this already-validated catalog cleanup land without expanding its scope further. Thanks both! |
tamirdresher
left a comment
tamirdresher
left a comment
There was a problem hiding this comment.
Re-reviewed at b286ddf: both prior blockers are resolved, targeted validation and CI are green, and the remaining suggestions are appropriate follow-up work.
|
From Earl''s Squad: @tamirdresher the follow-up you asked for is up: #1469. It covers all three items — prefer newest-per-series (incl. Opus 4.8), add the GPT-5.6 IDs (sol/terra/luna, CLI-reachability validated), and replace the Ralph circuit-breaker "free models" wording with usage-based framing. Kept it as a separate, self-contained PR per your suggestion; the one GPT-5.6 |
Three test files hard-coded the old fallback chain head (claude-sonnet-4.6) that was intentionally reordered in the parent commit (newest-per-series first, tamirdresher PR bradygaster#1444 follow-up). The tests remain logically correct — they now assert the NEW intended behavior. - test/compat-v041.test.ts: standard[0] → 'claude-sonnet-5' - test/models.test.ts: standard[0] → 'claude-sonnet-5' - test/agents.test.ts: standard fallback chain toEqual updated (reordered + added gpt-5.6-sol/terra/luna) Inline comment added to each noting the deliberate reorder. Pre-existing failures (cli-packaging-smoke, init-scaffolding) are EBUSY file-lock errors unrelated to these changes — confirmed on baseline. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…x Ralph free-model wording (#1469) * docs/models: prefer newest per series (Opus 4.8 + Sonnet 5), add GPT-5.6 IDs, fix Ralph free-model wording Follow-up to #1444 (merged), addressing tamirdresher's three explicit requests in #1444 (comment) (refs #1080, #1183). Item 1 — Prefer newest model per series (fallback ordering): - packages/squad-sdk/src/config/models.ts: DEFAULT_FALLBACK_CHAINS.standard now leads with claude-sonnet-5 (was claude-sonnet-4.6). - packages/squad-sdk/src/runtime/constants.ts: MODELS.FALLBACK_CHAINS.standard same reorder. Premium chains already led with claude-opus-4.8 post-#1444. - model-selection-reference.md, SKILL.md files: examples now show claude-opus-4.8 first in Premium, claude-sonnet-5 first in Standard. Item 2 — Add GPT-5.6 model IDs (gpt-5.6-sol, gpt-5.6-terra, gpt-5.6-luna): - Added to MODEL_CATALOG in models.ts (tier: standard, githubCategory: powerful — mirrors gpt-5.5 sibling entry). - Added to DEFAULT_FALLBACK_CHAINS.standard and MODELS.FALLBACK_CHAINS.standard (after claude-sonnet-4.6, before gpt-5.4). - Reachability: CLI-observed as Standard-tier reachable models (2026-07-13). - NOTE: gpt-5.6 entries in DOCS_NAME_TO_ID (cli/commands/models.ts) are deferred until draft PR #1445 merges (that file is in #1445's diff). - Updated Valid Models in all model-selection-reference.md + SKILL.md copies. Item 3 — Ralph circuit-breaker free-model wording: - .squad-templates/ralph-circuit-breaker.md: replaced multiplier table (0x / "Free — unlimited") with usage-based framing (lightweight category — lowest-cost, still billed). Updated preferredModel example to claude-sonnet-5. - Synced to all 3 template targets via sync-templates.mjs. NOTE: MODELS.DEFAULT (claude-sonnet-4.6) is intentionally left unchanged. That global default is out of scope for this PR — can be a separate decision. TDD: test/catalog-refresh.test.ts — wrote failing tests first (8 RED), then implemented changes. Final result: 16/16 tests GREEN. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(models): sync full Standard chain in doc/skill examples; add changeset Address Copilot review comments on PR #1469: - 3 doc/skill files had abbreviated Standard fallback chain (omitting gpt-5.6-terra, gpt-5.6-luna, gemini-2.5-pro). Now matches the full SDK runtime chain exactly: claude-sonnet-5 → claude-sonnet-4.6 → gpt-5.6-sol → gpt-5.6-terra → gpt-5.6-luna → gpt-5.4 → gpt-5.3-codex → claude-sonnet-4.5 → gemini-2.5-pro → (omit model param) Files: .squad-templates/model-selection-reference.md, .copilot/skills/model-selection/SKILL.md (manual), .squad/skills/model-selection/SKILL.md (sync canonical) Synced to all 3 template targets + 2 skill package targets. - Premium chain confirmed correct (claude-opus-4.8 first) — no change needed. - Add .changeset/models-gpt56-fallback-ordering.md (@bradygaster/squad-sdk patch) to satisfy Changelog Gate CI check. 16/16 catalog-refresh tests remain GREEN. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test: update stale standard-chain[0] assertions to claude-sonnet-5 Three test files hard-coded the old fallback chain head (claude-sonnet-4.6) that was intentionally reordered in the parent commit (newest-per-series first, tamirdresher PR #1444 follow-up). The tests remain logically correct — they now assert the NEW intended behavior. - test/compat-v041.test.ts: standard[0] → 'claude-sonnet-5' - test/models.test.ts: standard[0] → 'claude-sonnet-5' - test/agents.test.ts: standard fallback chain toEqual updated (reordered + added gpt-5.6-sol/terra/luna) Inline comment added to each noting the deliberate reorder. Pre-existing failures (cli-packaging-smoke, init-scaffolding) are EBUSY file-lock errors unrelated to these changes — confirmed on baseline. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(models): correct gpt-5.6 billing categories to match live Copilot API Live API (canonical) returns: sol=powerful, terra=versatile, luna=lightweight. Original seed incorrectly set all three to 'powerful' (mirroring gpt-5.5). Tier (standard) and fallback-chain membership are unchanged; only githubCategory (the cost-policy billing axis) is corrected. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Summary
Refreshes the hardcoded seed model catalog to currently-reachable GitHub Copilot model IDs and removes dead IDs from the tier fallback chains. Several IDs in the chains no longer resolve, so they silently burn retry/fallback slots before landing on a working model — this fixes that routing waste.
This is the first of a small, sequential set of changes coordinated upstream (see comment below): catalog refresh → cost policy → docs sync. Landing the catalog refresh on its own keeps each PR tight and independently reviewable.
Coordination comment: #1080 (comment)
Refs #1080
Refs #1183
What changed
gpt-4.1,gpt-5,gpt-5.1,gpt-5.1-codex,gpt-5.1-codex-max,gpt-5.1-codex-mini,gpt-5.2,gpt-5.2-codex,gemini-3-pro-preview,claude-sonnet-4,claude-opus-4.5,claude-opus-4.6-fastclaude-opus-4.7,claude-opus-4.8,claude-sonnet-5,gpt-5.5,gpt-5.4-mini,gemini-2.5-proruntime/constants.tsMODELS.FALLBACK_CHAINS,config/models.tsDEFAULT_FALLBACK_CHAINS,config/schema.tsDEFAULT_CONFIG.models, andECONOMY_MODEL_MAP) so every chain entry now resolves.githubCategorycost-ceiling field (lightweight/versatile/powerful) onModelInfo, kept as a separate axis from the existing qualitytier. No per-token pricing is hardcoded for new entries, and noincluded/ zero-credit flag is introduced.packages/squad-cli/templates/skills/squad-commands/SKILL.md(238 lines) — a template sync-drift fix; the other template targets already carried this file, and thesquad-clicopy was the only one missing it.-build.Nversion bumps.Design notes
The committed catalog is a seed / fallback by design — not a source of truth. A follow-up in this sequence adds a
squad models refreshcommand that enriches the catalog from the Copilot models API (model_picker_category) at runtime, with the publicgithub/docspricing YAML as an auth-free fallback. KeepinggithubCategoryoptional preserves existing out-of-catalog passthrough.Testing
test/catalog-refresh.test.tsinvariants: no dead ID appears in any fallback chain, and every chain ID exists inMODEL_CATALOG(covers both the SDK default chains and the runtime chains).test/compat-v041.test.tscoverage for removed-model fallback degradation (a config referencing a removed ID resolves to a live catalog model without throwing).validateConfigDetailednow emits a non-blocking warning whendefaultModelis not inMODEL_CATALOG.npm run lintand workspace build are clean; the full suite (123 tests) shows no model-related regressions.A changeset is included (
@bradygaster/squad-sdk, minor).