fix: add modelReferences to endpoint create/update and document in runpod.toml template by jebenexer · Pull Request #281 · runpod/runpodctl

jebenexer · 2026-05-26T22:05:24Z

Summary

Adds support for modelReferences in the endpoint create/update flow.

Changes

api/endpoint.go — Added ModelReferences []string field to CreateEndpointInput; fixed missing json:"id" tag on Endpoint.Id; added modelReferences to GetEndpoints GraphQL query; added UpdateEndpointModel() function that uses saveEndpoint mutation to update models on existing endpoints
cmd/project/functions.go — deployProject reads model_refs from endpoint config and passes it as ModelReferences to CreateEndpointInput
cmd/project/tomlBuilder.go — Documents the new model_refs config key in the [endpoint] section of the runpod.toml template
cmd/endpoint/update_model.go (new) — Adds runpodctl endpoint model <endpoint-id> <model-ref> [model-ref...] CLI command to set model references on an existing endpoint
cmd/root.go — Registers the new endpoint command group

Pull request overview

Adds modelReferences support to serverless endpoint creation and introduces a new CLI command to update an existing endpoint’s cached model references, with accompanying documentation in the runpod.toml template.

Changes:

Extend endpoint API payloads/queries to include modelReferences, and fix JSON unmarshalling for endpoint IDs.
Read model_refs from runpod.toml during project deploy and pass through to endpoint creation.
Add runpodctl endpoint model ... command for updating model references on an existing endpoint.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
cmd/root.go	Registers the new top-level `endpoint` command group.
cmd/project/tomlBuilder.go	Documents `model_refs` in the generated `runpod.toml` template.
cmd/project/functions.go	Parses `model_refs` from TOML and passes them into endpoint creation.
cmd/endpoint/update_model.go	Adds CLI subcommand to set endpoint model references.
cmd/endpoint/endpoint.go	Introduces the `endpoint` command group and wires subcommands.
api/endpoint.go	Adds `ModelReferences` to create input, fixes `Endpoint.Id` JSON tag, extends queries, and adds model update mutation helper.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+		if refs, ok := endpointConfig.Get("model_refs").([]interface{}); ok {
+			for _, r := range refs {
+				modelRefs = append(modelRefs, r.(string))
+			}
+		}


+var modelCmd = &cobra.Command{
+	Use:   "model <endpoint-id> <model-ref> [model-ref...]",
+	Short: "update model references on an endpoint",
+	Long:  "set the model references (cached models) for a serverless endpoint",
+	Args:  cobra.MinimumNArgs(2),
+	RunE:  runModel,
+}
+
+func runModel(cmd *cobra.Command, args []string) error {
+	endpointID := args[0]
+	modelRefs := args[1:]
+


TimPietruskyRunPod

the modelReferences plumbing here is solid — CreateEndpointInput, the GetEndpoints query addition, UpdateEndpointModel() via saveEndpoint, and the runpod.toml model_refs key are all the right pieces. requesting changes on the CLI surface only.

don't introduce a new top-level endpoint group — use the existing serverless group.

we already have serverless (create/delete/get/list/update), and serverless create --model-reference already landed in #276. adding endpoint model alongside it gives two command namespaces for the same resource, and model would be the only thing under endpoint — so the surface fragments and --model-reference (create) vs endpoint model (update) becomes inconsistent for the same concept.

endpoint model <id> [refs...] [--clear] is functionally just a serverless update that only touches model refs. fold it into the command that already exists:

set/replace: runpodctl serverless update <id> --model-reference a --model-reference b
remove all: runpodctl serverless update <id> --clear-models

concretely:

add a repeatable --model-reference (StringArrayVar) and a --clear-models bool to cmd/serverless/update.go, matching the --model-reference verb already on serverless create
on update, set modelReferences on the saveEndpoint input: the provided refs, or [] when --clear-models is passed (same empty-list-to-clear behavior the current --clear already relies on)
keep all the api / runpod.toml plumbing from this PR as-is — only the command placement changes
drop cmd/endpoint/ (and revert the docs to extend the serverless reference page instead of a new endpoint page)

the only thing that would justify a new group is a deliberate rename serverless → endpoint (the API/console domain term is "Endpoint") with serverless kept as a hidden alias — but that's a bigger decision and should be its own PR with a deprecation plan, not introduced as a side effect of adding model refs.

tl;dr: plumbing 👍 — move endpoint model → serverless update --model-reference / --clear-models and drop the new top-level group.

TimPietruskyRunPod · 2026-06-19T11:54:44Z

to be clear on scope — the read-side additions are great, keep them:

adding modelReferences to the GetEndpoints() query selection so cached models surface in serverless get / serverless list 👍
the Endpoint.ModelReferences field + the endpoint-id json unmarshalling fix 👍
model_refs in runpod.toml + the project-deploy passthrough 👍

the only change requested is the write-side command placement: drop the new top-level endpoint group (cmd/endpoint/) and expose the update via serverless update --model-reference / --clear-models instead. everything else in the PR stays.

so: read extension to serverless get = 🔥, just not the endpoint model command.

jebenexer · 2026-06-24T17:56:26Z

done — dropped cmd/endpoint/ entirely and folded the write-side into serverless update:

runpodctl serverless update <id> --model-reference <ref> [--model-reference <ref2>...] — set/replace
runpodctl serverless update <id> --clear-models — remove all

the two are mutually exclusive (validated at runtime). the api/endpoint.go plumbing, runpod.toml model_refs, GetEndpoints query addition, and ModelReferences field are all unchanged. UpdateEndpointModels lands in internal/api/endpoints.go via saveEndpoint graphql. all unit tests pass, e2e verified against the live api.

jebenexer · 2026-06-25T17:00:53Z

Done — cmd/endpoint/ is gone and the new top-level endpoint group with it. --model-reference (repeatable, replaces existing) and --clear-models are now flags on serverless update, matching the verb already on serverless create. The model_refs TOML plumbing and the UpdateEndpointModels API helper are unchanged. History is squashed to a single commit on top of current main.

TimPietruskyRunPod

re-review — command placement is fixed, but the update path ships data loss

thanks for folding the write-side into serverless update --model-reference / --clear-models and dropping the top-level endpoint group — that part is exactly right, and the create/toml plumbing is good.

blocking: serverless update --model-reference (and --clear-models) silently resets the rest of the endpoint's config to defaults. saveEndpoint is a full replace, not a merge, and UpdateEndpointModels only sends id + name + modelReferences, so every other field reverts.

verified end-to-end against the live api on this branch:

created an endpoint with idleTimeout=42, scalerValue=9, workersMax=5
ran serverless update <id> --model-reference https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct:main
re-fetched: idleTimeout=10, scalerValue=4, workersMax=3 — all reset to defaults

this is net-new in this PR: on main, saveEndpoint is only ever called for create (no id), and the existing serverless update uses the REST PATCH (UpdateEndpoint) + the dedicated updateEndpointTemplate mutation for partial updates — neither touches saveEndpoint. the safe partial-update path was already established; the model-update code just didn't follow it. it also breaks the combined case (--workers-min 5 --model-reference X): the REST patch sets workers, then saveEndpoint immediately wipes them.

the schema confirms only name is required on EndpointInput, so the partial mutation succeeds silently with no error — which is what makes this a quiet footgun rather than a loud failure.

fix options:

round-trip the full current config — UpdateEndpointModels already fetches the endpoint for its name; fetch the rest (workersMin/workersMax/idleTimeout/scalerType/scalerValue/gpuIds/templateId/...) and echo it back alongside modelReferences, or
use/ask backend for a dedicated partial mutation (the way updateEndpointTemplate exists separately is the hint that saveEndpoint isn't safe for partial updates).

plus two cleanups inline. happy to push the round-trip fix if useful.

…od.toml - Add --model-reference flag to 'serverless update' (repeatable; replaces existing) - Add --clear-models flag to 'serverless update' to remove all model references - Wire model_refs from runpod.toml [endpoint] section into 'project deploy' - Add modelReferences to GetEndpoints query - Document model_refs in generated runpod.toml template - Add UpdateEndpointModels() to internal API client Closes SLS-94

Tim is no longer on the team and all concerns have been addressed

promptless · 2026-07-01T17:16:32Z

Promptless updated the documentation suggestion to match the final merged changes.

Triggered by PR #281

Since the standalone endpoint model command was dropped during review, the earlier suggestion was reworked to document the --model-reference and --clear-models flags on serverless update instead, plus a cross-reference from the model caching page. The stale reference page and nav entries have been removed.

Review: Document model references for Serverless endpoints

jebenexer force-pushed the ben/sls-94-modelname-cli branch from 0f09ed8 to 8a4b418 Compare June 15, 2026 18:43

deanq requested a review from Copilot June 16, 2026 09:11

Copilot started reviewing on behalf of deanq June 16, 2026 09:11 View session

Copilot AI reviewed Jun 16, 2026

View reviewed changes

TimPietruskyRunPod suggested changes Jun 19, 2026

View reviewed changes

TimPietruskyRunPod mentioned this pull request Jun 19, 2026

feat: allow --model-reference with --hub-id on serverless create #289

Merged

jebenexer force-pushed the ben/sls-94-modelname-cli branch from 1af837b to 09afb41 Compare June 24, 2026 17:46

jebenexer requested a review from TimPietruskyRunPod June 24, 2026 17:56

jebenexer force-pushed the ben/sls-94-modelname-cli branch from 09afb41 to 54c8f3b Compare June 25, 2026 16:59

TimPietruskyRunPod previously requested changes Jun 30, 2026

View reviewed changes

Comment thread internal/api/endpoints.go

Comment thread api/endpoint.go Outdated

Comment thread api/endpoint.go Outdated

jebenexer force-pushed the ben/sls-94-modelname-cli branch from 54c8f3b to 45b13fe Compare June 30, 2026 14:32

jebenexer requested a review from TimPietruskyRunPod June 30, 2026 14:34

brosenpod approved these changes Jul 1, 2026

View reviewed changes

jebenexer merged commit 171d926 into main Jul 1, 2026
1 check passed

jebenexer deleted the ben/sls-94-modelname-cli branch July 1, 2026 17:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix: add modelReferences to endpoint create/update and document in runpod.toml template#281

fix: add modelReferences to endpoint create/update and document in runpod.toml template#281
jebenexer merged 1 commit into
mainfrom
ben/sls-94-modelname-cli

jebenexer commented May 26, 2026 •

edited

Loading

Uh oh!

promptless Bot commented May 26, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

TimPietruskyRunPod left a comment

Uh oh!

TimPietruskyRunPod commented Jun 19, 2026

Uh oh!

jebenexer commented Jun 24, 2026

Uh oh!

jebenexer commented Jun 25, 2026

Uh oh!

TimPietruskyRunPod left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

promptless Bot commented Jul 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

jebenexer commented May 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Related

Uh oh!

promptless Bot commented May 26, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

TimPietruskyRunPod left a comment

Choose a reason for hiding this comment

Uh oh!

TimPietruskyRunPod commented Jun 19, 2026

Uh oh!

jebenexer commented Jun 24, 2026

Uh oh!

jebenexer commented Jun 25, 2026

Uh oh!

TimPietruskyRunPod left a comment

Choose a reason for hiding this comment

re-review — command placement is fixed, but the update path ships data loss

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

promptless Bot commented Jul 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jebenexer commented May 26, 2026 •

edited

Loading