(bug) Model picker shows a false "not available" badge for vault-managed models by mannietech15 · Pull Request #5522 · Agenta-AI/agenta

mannietech15 · 2026-07-26T16:32:24Z

Summary

The "model not available" compatibility badge in the harness picker incorrectly flagged
vault-managed custom-provider models as unavailable, even when the backend accepted and ran them
successfully.

harnessAllowsModel in connectionUtils.ts already accepts customSecrets and slug parameters
to check vault connections before concluding a model is unreachable. The function itself was correct
and unit-tested. The bug was at the call site in useModelHarness.tsx: selectedKeepsModel called
harnessAllowsModel without those arguments, so the vault path was never consulted.

Before:

// vault-hosted "custom-bedrock-model-id-123" → false → badge: "model not available"
harnessAllowsModel(capabilities, harnessValue, modelId)

After:

// vault-hosted "custom-bedrock-model-id-123" → true → badge: "supports your model"
harnessAllowsModel(capabilities, harnessValue, modelId, customSecrets, connection.slug || null)

Both customSecrets (from customSecretsAtom) and connection.slug (decoded from config.llm)
were already in scope at the call site; they just weren't being passed through.

Testing

Verified locally

pnpm test in web/packages/agenta-entity-ui — 12 test files, 210 tests, all passed.

Added or updated tests

Added "selectedKeepsModel regression: vault model flagged unavailable without secrets, available with them" to connectionUtils.test.ts. The test explicitly asserts the broken pre-fix behaviour
(no secrets passed → false) against the fixed behaviour (secrets + slug passed → true) and
includes a slug-mismatch case (a credential for a different connection must not grant availability).

QA follow-up

Add a vault custom-provider connection (Bedrock or custom kind) with a model id that does not
encode a recognizable provider family (for example a bare slug like custom-bedrock-model-id-123).
Open an agent config and go to the Model & harness section.
Select the vault model from the picker.
Expected: the harness detail panel shows "supports your model" in green. Before this fix
it showed "model not available" in amber.
Regression: catalog models (OpenAI, Anthropic, etc.) must still show correct availability
under their respective harnesses.

Demo

N/A — badge label and colour change only, no structural UI change. The QA steps above describe
where to look.

Checklist

I have included a video or screen recording for UI changes, or marked Demo as N/A
Relevant tests pass locally
Relevant linting and formatting pass locally
I have signed the CLA, or I will sign it when the bot prompts me

Contributor Resources

agenta-2.mp4

…redential failures hang instead of failing fast - fixed

vercel · 2026-07-26T16:32:30Z

@mannietech15 is attempting to deploy a commit to the agenta projects Team on Vercel.

A member of the Team first needs to authorize it.

CLAassistant · 2026-07-26T16:32:31Z

All committers have signed the CLA.

coderabbitai · 2026-07-26T16:33:20Z

📝 Walkthrough

Summary by CodeRabbit

Bug Fixes
- Improved model compatibility checks for custom-provider connections.
- Validated selected models against the matching connection and available custom-provider credentials.
- Prevented unsupported or unavailable models from being selected when explicit connection details are provided.

Walkthrough

The change extends harness model compatibility checks with custom-provider secrets and connection slugs. It validates provider reachability for custom models and adds regression tests for provider compatibility, explicit connection matching, missing secrets, and slug mismatches.

Changes

Custom-provider model reachability

Layer / File(s)	Summary
Custom-provider compatibility evaluation `web/packages/agenta-entity-ui/src/DrillInView/SchemaControls/connectionUtils.ts`, `web/packages/agenta-entity-ui/src/DrillInView/SchemaControls/agentTemplate/useModelHarness.tsx`	`harnessAllowsModel` validates custom-provider models through matching secrets, provider kinds, and connection slugs. `selectedKeepsModel` passes the custom secrets and selected slug.
Regression coverage for model gating `web/packages/agenta-entity-ui/tests/unit/connectionUtils.test.ts`	Tests cover non-standard custom-provider model IDs, explicit connection validation, missing secrets, and slug mismatches.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

Agenta-AI/agenta#4839: Extends related model-gating logic for custom vault-provider secrets and connection slugs.
Agenta-AI/agenta#4923: Modifies the same agentTemplate/useModelHarness.tsx model-selection path.
Agenta-AI/agenta#5096: Extends related harnessAllowsModel compatibility checks.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly identifies the false availability badge issue for vault-managed models.
Description check	✅ Passed	The description accurately explains the bug, code change, tests, and QA steps.

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

Actionable comments posted: 4

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 628c99f8-fa2c-4ae2-a5ff-5401798d0666

📥 Commits

Reviewing files that changed from the base of the PR and between cb095f7 and 2cf38ea.

📒 Files selected for processing (5)

CONTRIBUTING.md
fix-agenta-setup.sh
web/packages/agenta-entity-ui/src/DrillInView/SchemaControls/agentTemplate/useModelHarness.tsx
web/packages/agenta-entity-ui/src/DrillInView/SchemaControls/connectionUtils.ts
web/packages/agenta-entity-ui/tests/unit/connectionUtils.test.ts

bekossy

Hi @mannietech15, thanks for tracking this down. The root cause diagnosis is right, but the fix needs one more pass.

Ordering issue in harnessAllowsModel: the generic catalog/model check (connectionUtils.ts:369-374) runs before the slug-scoped vault check. A vault model id that collides with an existing catalog/model id (easy to hit, e.g. a bare alias like "opus") will return true without ever validating that this specific connection is reachable. When slug is provided, please resolve and validate that connection first, and only fall back to the generic check when no slug is given. A test with a colliding id would lock this in.

Please drop from this PR (unrelated to the fix, and fix-agenta-setup.sh runs sudo to rewrite /etc/docker/daemon.json and restart the Docker daemon, which we don't want in the repo):

fix-agenta-setup.sh
CONTRIBUTING.md

mannietech15 · 2026-08-04T18:39:06Z

@bekossy , sorry for the delay! Thanks for the great feedback and for catching the name collision edge case. I've updated the logic to check the slug-scoped vault first, added the collision test you requested, and cleaned up those stray files from the PR.

coderabbitai · 2026-08-10T12:07:52Z

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

coderabbitai

🧹 Nitpick comments (1)

web/packages/agenta-entity-ui/src/DrillInView/SchemaControls/connectionUtils.ts (1)

367-369: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Reduce this in-code comment to one short line.

Lines 367-369 add a multi-line explanatory comment. This is not a bug, race, or ordering constraint. Move the detail to external documentation or reduce it to one short line. As per coding guidelines, “Keep in-code comments to at most one short line; use longer comments only for genuinely surprising constraints such as bugs, races, or ordering requirements.”

Source: Coding guidelines

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: ea135f4b-689f-4d75-afe2-e8ac8e354c6a

📥 Commits

Reviewing files that changed from the base of the PR and between 8ea5d3b and 54bd923.

📒 Files selected for processing (4)

CONTRIBUTING.md
web/packages/agenta-entity-ui/src/DrillInView/SchemaControls/agentTemplate/useModelHarness.tsx
web/packages/agenta-entity-ui/src/DrillInView/SchemaControls/connectionUtils.ts
web/packages/agenta-entity-ui/tests/unit/connectionUtils.test.ts

🚧 Files skipped from review as they are similar to previous changes (2)

web/packages/agenta-entity-ui/src/DrillInView/SchemaControls/agentTemplate/useModelHarness.tsx
web/packages/agenta-entity-ui/tests/unit/connectionUtils.test.ts

- eslint --fix on connectionUtils.test.ts (prettier formatting only, no logic change) - CONTRIBUTING.md: reverted to main; the added Quickstart section told contributors to run pnpm from the repo root, which has no package.json (the workspace lives under web/) and would fail immediately

mannietech15 added 2 commits July 24, 2026 07:34

docs: update CONTRIBUTING.md with quickstart guide and guidelines

ab999e9

(bug) Model picker shows a false "not available" badge, and Bedrock c…

2cf38ea

…redential failures hang instead of failing fast - fixed

dosubot Bot added the size:L This PR changes 100-499 lines, ignoring generated files. label Jul 26, 2026

dosubot Bot added bug report Something isn't working frontend labels Jul 26, 2026

mannietech15 mentioned this pull request Jul 26, 2026

(bug) Model picker shows a false "not available" badge, and Bedrock credential failures hang instead of failing fast #5118

Open

coderabbitai Bot reviewed Jul 26, 2026

View reviewed changes

Comment thread CONTRIBUTING.md Outdated

Comment thread fix-agenta-setup.sh Outdated

Comment thread fix-agenta-setup.sh Outdated

Comment thread web/packages/agenta-entity-ui/src/DrillInView/SchemaControls/connectionUtils.ts

bekossy changed the base branch from main to release/v0.106.1 July 29, 2026 11:25

bekossy requested changes Jul 29, 2026

View reviewed changes

bekossy and others added 2 commits July 29, 2026 13:46

Merge branch 'release/v0.106.1' into mannietech15

d9fdbb6

fix(frontend): enforce vault slug matching before catalog fallback

19e90fe

dosubot Bot mentioned this pull request Aug 6, 2026

(bug) Model picker allows invalid harness/model/auth combinations without validation #5786

Open

bekossy changed the base branch from release/v0.106.1 to main August 10, 2026 12:07

Merge branch 'main' into mannietech15

54bd923

coderabbitai Bot reviewed Aug 10, 2026

View reviewed changes

dosubot Bot added size:M This PR changes 30-99 lines, ignoring generated files. and removed size:L This PR changes 100-499 lines, ignoring generated files. labels Aug 10, 2026

bekossy changed the title ~~(bug) Model picker shows a false "not available" badge, and Bedrock credential failures hang instead of failing fast - Fixed~~ (bug) Model picker shows a false "not available" badge for vault-managed models Aug 10, 2026

bekossy approved these changes Aug 10, 2026

View reviewed changes

dosubot Bot added the lgtm This PR has been approved by a maintainer label Aug 10, 2026

bekossy linked an issue Aug 10, 2026 that may be closed by this pull request

(bug) Model picker shows a false "not available" badge, and Bedrock credential failures hang instead of failing fast #5118

Open

Navigation Menu

Uh oh!

(bug) Model picker shows a false "not available" badge for vault-managed models - #5522

(bug) Model picker shows a false "not available" badge for vault-managed models#5522
mannietech15 wants to merge 6 commits into
Agenta-AI:mainfrom
mannietech15:mannietech15

mannietech15 commented Jul 26, 2026

Uh oh!

vercel Bot commented Jul 26, 2026

Uh oh!

CLAassistant commented Jul 26, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Jul 26, 2026 •

edited

Loading

Summary by CodeRabbit

Walkthrough

Changes

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bekossy left a comment

Uh oh!

mannietech15 commented Aug 4, 2026

Uh oh!

coderabbitai Bot commented Aug 10, 2026

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

mannietech15 commented Jul 26, 2026

Summary

Testing

Verified locally

Added or updated tests

QA follow-up

Demo

Checklist

Contributor Resources

Uh oh!

vercel Bot commented Jul 26, 2026

Uh oh!

CLAassistant commented Jul 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot commented Jul 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Walkthrough

Changes

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bekossy left a comment

Choose a reason for hiding this comment

Uh oh!

mannietech15 commented Aug 4, 2026

Uh oh!

coderabbitai Bot commented Aug 10, 2026

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

CLAassistant commented Jul 26, 2026 •

edited

Loading

coderabbitai Bot commented Jul 26, 2026 •

edited

Loading