Skip to content

serve base_model in api.json and catalog.json - #3908

Open
arimxyer wants to merge 2 commits into
anomalyco:devfrom
arimxyer:serve-base-model
Open

serve base_model in api.json and catalog.json#3908
arimxyer wants to merge 2 commits into
anomalyco:devfrom
arimxyer:serve-base-model

Conversation

@arimxyer

Copy link
Copy Markdown

What

Provider model TOMLs already declare base_model — the edge linking an offering to its canonical models/ registry entry — but generate.ts strips the field during inheritance merging, so neither api.json nor catalog.json carries it. Consumers who want to group provider offerings by underlying model (e.g. "which providers serve anthropic/claude-opus-4-6, and at what price?") have to re-derive that linkage themselves from heuristics, even though the repository already knows the answer.

The site itself pays for this today: loadProviderBaseModelRefs in packages/web/src/render.tsx re-parses every provider TOML a second time solely to rebuild the refs the generate pipeline just dropped.

This PR retains the resolved ref on the emitted model instead:

  • generate.ts keeps base_model on the provider model after the inheritance merge validates (the ref is guaranteed to resolve at this point — mergeBaseModel throws otherwise)
  • ModelShape allows the field as an output-only optional; authored TOMLs still declare it via the BaseModel wrapper schema, and models.json (registry metadata) is unaffected
  • render.tsx derives BaseModelRefs from the generated catalog and drops the duplicate TOML scan
  • the SDK Model type documents the field (generated effect schema updated via bun run generate)

Net diff is ±25 lines.

Why

base_model is the only provider-agnostic identity signal in the dataset, and it's currently invisible to every API consumer. Serving it makes catalog.json self-contained for cross-provider grouping: group_by(model.base_model ?? id) replaces per-consumer name/id heuristics. It also removes the site's own double-parse workaround.

Verification

  • bun validate passes; 2,148 refs surface in _api.json / _catalog.json
  • the catalog-derived BaseModelRefs map is identical to the previous raw-TOML scan: 2,148/2,148 entries, 0 mismatches (so site grouping behavior is provably unchanged)
  • models.json carries no new field (0 occurrences)
  • SDK drift suite passes: bun run generate && tsc --noEmit && bun test (23 pass)

🤖 Generated with Claude Code

@github-actions

Copy link
Copy Markdown
Contributor

Action items

  • [high] [violation] packages/core/test/generate.test.ts:38 - Check: Catalog tests must match the new base_model emit contract. Why: Two tests still require provider JSON to omit base_model: base_model can factor metadata without changing provider JSON (equality + not.toHaveProperty("base_model")) and repository provider JSON strips authored metadata pointers (fails if encoded output contains base_model). This PR keeps the ref on emitted models, so those assertions are now wrong and will fail under bun test. Action: Rewrite them to assert resolved base_model is retained, factored vs inline models match after omitting that field, and only base_model_omit (plus other non-output pointers) stays stripped.
  • [medium] [violation] AGENTS.md:103 - Check: AGENTS.md is authoritative for base_model output behavior. Why: It still says the final provider JSON contains no base_model or base_model_omit fields, which this change deliberately reverses for base_model. Leaving that line would document the opposite of the served API. Action: Update that sentence to state resolved base_model is emitted on provider models and base_model_omit remains parse-only / never served.

Provider model TOMLs already declare base_model — the edge linking an
offering to its canonical models/ registry entry — but generate.ts
strips the field during inheritance merging, so no served JSON carries
it. Consumers that want to group offerings by underlying model have to
re-derive the linkage themselves; the site itself works around it by
re-parsing every provider TOML a second time (loadProviderBaseModelRefs
in render.tsx) just to rebuild the refs the pipeline dropped.

Retain the resolved ref on the emitted model instead:

- generate.ts keeps base_model on the provider model after the
  inheritance merge is validated (the ref is already guaranteed to
  resolve — mergeBaseModel throws otherwise)
- ModelShape allows the field as an output-only optional; authored
  TOMLs still declare it via the BaseModel wrapper, base_model_omit is
  still stripped, and models.json (registry metadata) is unaffected
- render.tsx derives BaseModelRefs from the generated catalog and drops
  the duplicate TOML scan
- SDK Model type documents the field; generated effect schema and
  drift tests pass
- the generate tests are updated to pin the new contract: the factored
  fixture now expects the retained ref, and the repository-wide leakage
  test now asserts base_model_omit never leaks while every served
  base_model resolves to a models/ registry entry

bun validate passes; 2,148 refs surface in _api.json/_catalog.json and
the derived refs map is verified identical to the previous raw-TOML
scan (2,148/2,148, 0 mismatches).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@arimxyer

Copy link
Copy Markdown
Author

Updated the branch to also bring the core test suite along with the contract change:

  • the factored-fixture test now expects the retained ref ({ ...direct, base_model: "lab/model" }) instead of asserting its absence
  • the repository-wide leakage test now asserts base_model_omit never leaks and that every served base_model resolves to a models/ registry entry (a stronger guarantee than before)

Note: bun test has one unrelated failure on current dev (repository open-weight model metadata includes weights linkssarvam/sarvam-105b has open_weights = true but no weights links). It pre-exists this branch; happy to fix it separately.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@arimxyer

Copy link
Copy Markdown
Author

Both review action items are addressed: the test contract was updated in the previous push (factored fixture expects the retained ref; the repo-wide test now guards base_model_omit only and additionally asserts every served ref resolves), and AGENTS.md now documents that resolved base_model is served while base_model_omit stays authoring-only.

@github-actions

Copy link
Copy Markdown
Contributor

No actionable findings.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant