Conversation
An Improve tab that turns issue detection findings into reviewed fixes through a pluggable acquire/apply/submit pipeline, building on RFC-0012's trace events. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Per review discussion: enterprise teams should be able to pull fix prompts programmatically into their own managed coding agents, not only via the UI. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
| **2. Apply a fix.** How a diagnosis becomes file changes in that checkout. Backends: OpenCode (the bundled, MIT-licensed default), Claude Code, other coding agents, or a plain script for mechanical fixes with no model calls at all. The backend is invoked in the workspace with the diagnosis and trace evidence as input, edits files, and exits. The only secret this phase ever sees is its own model access (an AI Gateway endpoint) — the interfaces on either side hold the repository credentials, so a coding agent structurally cannot touch them. | ||
|
|
||
| **3. Submit for human review.** How the changed code reaches people. Backends: a GitHub pull request (default), a GitLab merge request, a patch bundle, a pushed branch for teams with their own review tooling, or delegating to a forge's own coding agent (for example, filing a GitHub issue assigned to Copilot). **Copy prompt** is the simplest backend of all: no checkout and no branch — MLflow hands the user the diagnosis and evidence as a ready-made prompt for whatever coding agent they drive themselves, skipping phases 1 and 2 entirely. | ||
| **3. Submit for human review.** How the changed code reaches people. Backends: a GitHub pull request (default), a GitLab merge request, a patch bundle, a pushed branch for teams with their own review tooling, or delegating to a forge's own coding agent (for example, filing a GitHub issue assigned to Copilot). **Copy prompt** is the simplest backend of all: no checkout and no branch — MLflow hands the user the diagnosis and evidence as a ready-made prompt for whatever coding agent they drive themselves, skipping phases 1 and 2 entirely. Copy prompt is available both as a UI action and programmatically — an API returns the fix prompt for a diagnosis, so enterprise teams can feed it into their own managed coding agents or software-factory pipelines without going through the UI. |
There was a problem hiding this comment.
Yep - let us drive MLflow "headless". The analysis/prompt "work package" gets fed back to our systems, so it follows the blessed flows (heavy investment - need+want to keep that control). Thank @Nehanth.
| import mlflow | ||
|
|
||
| # One-time setup: connect the agent's repository to the experiment | ||
| mlflow.genai.connect_repo( |
There was a problem hiding this comment.
nit: I think this should be platform-level capability, not GenAI
There was a problem hiding this comment.
Yeah — changed it to mlflow.connect_repo in the example.
| A support agent's system prompt was written before the product's new billing system launched. Users now ask billing questions the prompt gives no guidance on, and the agent answers vaguely. Nothing errors — but the relevance and completeness scores on those conversations drop, because the judge can see from the conversation alone that the answers are not addressing what users asked. | ||
|
|
||
| 1. The threshold event fires on the rolling average and triggers the workflow. The diagnosis points at the prompt rather than the code, with the low-scoring conversations as evidence. | ||
| 2. Because MLflow already has a prompt registry and [prompt optimization](https://mlflow.org/docs/latest/genai/prompt-registry/optimize-prompts/), the fix path needs no coding agent: the low-scoring traces become the evaluation set, `optimize_prompts()` rewrites the prompt against them, and the improved version is registered with before-and-after scores shown in the diagnosis. |
There was a problem hiding this comment.
nit: Prompt optimization generally requires non-trivial number of ground truth or target function. I would let the harness directly update prompt instead of relying on optimization here.
There was a problem hiding this comment.
Makes sense, a few bad traces isn't enough for the optimizer to work with. Will change it so the harness rewrites the prompt directly, same as any other fix.
| # runs when RFC-0012's events fire; list the open ones and apply one | ||
| suggestions = mlflow.genai.list_improvements(experiment_id="12345", status="open") | ||
| job = mlflow.genai.apply_improvement( | ||
| suggestion_id=suggestions[0].id, |
There was a problem hiding this comment.
- Harness execution should be a part of Assistant, because that will be the main interface for UI users to interact with harnesses. We shouldn't introduce different execution mechanism between UI and SDK.
- With (1), this could be more generic API that trigger harness with a single prompt. It allows harnesses to handle suggestions with more flexibility, e.g., combine multiple suggestions together.
There was a problem hiding this comment.
Yeah agreed, keeping one execution path for UI and SDK is cleaner. The workflow can just build a prompt from the suggestions and hand it to the Assistant. Will update the RFC to reflect that.
|
|
||
| A team runs a support agent traced to MLflow. A provider deprecates the model identifier the agent has hardcoded, and every request starts failing. | ||
|
|
||
| 1. In the experiment's **Improve** tab, the team has already connected their repository: URL and an access token, stored through MLflow's existing secret handling, plus the harness choice. This is one-time setup. The connection lives at the experiment level, since an experiment usually tracks a single agent — a team running several agents connects each experiment to its own repository. |
There was a problem hiding this comment.
We are adding Jira-like "Issues" tab in the UI. The workflow can start from there.
There was a problem hiding this comment.
Yeah, will change it to start from the Issues tab and update the RFC.
|
|
||
| 6. On merge, the issue is resolved in MLflow, and the traces that exposed the failure are added to an evaluation dataset ([`merge_records()`](https://mlflow.org/docs/latest/genai/datasets/) accepts traces directly) — the failure is now a regression test. A pull request closed without merging means the reviewer rejected the change: the diagnosis is marked **rejected**, and later detection runs do not re-propose the same fix. | ||
|
|
||
| MLflow learns the outcome by polling the pull request through the submission backend, with backoff — checks start frequent and stretch out as the PR ages. After a bounded window, MLflow stops polling altogether and marks the diagnosis **stale**; opening it in the tab re-checks once, on demand. A PR that humans ignore therefore costs nothing in the background. Whether the forge should instead push events to MLflow is left to the detailed design: that would be a new inbound endpoint on the MLflow server (RFC-0012's webhooks only send outbound), which is a larger security decision than polling. |
There was a problem hiding this comment.
Not sure if this is the right mechanism. Github REST API has strict rate limit and a server can keep polling indefinite amount of times if some stale issue is not cleaned up. I prefer letting users to set up Github action that invokes MLflow APIs.
There was a problem hiding this comment.
Yeah fair, though the Action would be extra setup for users. Rate limits look workable: 5,000 req/hr, 15,000 for enterprise apps, and 304s from conditional requests don't count at all. Do you think we could offer both? Polling by default with the stale cutoff, and an optional GitHub Action that calls the MLflow API for teams that want instant updates.
|
|
||
| The same shortcut extends beyond prompts as the registries land: a skill is, like a prompt, mostly instructions as text, so a diagnosis rooted in a skill's instructions can become a new skill version registered through the Skill Registry and rolled out or back through the registry's own versioning. The coding harness is reserved for what actually is code. | ||
|
|
||
| ### Keep detection running on a schedule |
There was a problem hiding this comment.
This has been worked on in databricks side and we can port the same design (e.g. checkpointing) to OSS. Let's focus on the improvement workflow itself in this RFC.
There was a problem hiding this comment.
Sounds good, will trim the scheduling mechanics out and keep the RFC focused on the workflow itself.
Repository connection is a platform-level capability, per review feedback. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vz67P5bPS9UoKK9u7yJfJm
Harness execution runs through the MLflow Assistant, one mechanism shared by UI and SDK, with the workflow composing fix prompts from one or more diagnoses. The workflow starts from the Issues tab being added to the experiment UI instead of a separate queue. The harness rewrites prompts directly rather than relying on prompt optimization. Scheduling mechanics defer to the design being ported from the Databricks side. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vz67P5bPS9UoKK9u7yJfJm
|
Hey @B-Step62, thanks for the comments! Went through all of them and pushed the updates. The only open one is the polling thread, I asked a question there. Once we're aligned I can start filling in the detailed design. Also curious what you think about the webhooks RFC (#41) whenever you get a chance. |
|
@Nehanth is this still moving? |
3 participants
The second half of the #38 split, building on RFC-0012 (#41): an Improve tab where issue detection findings are pinned to the connected repository and a coding agent produces a fix as a pull request, through a pluggable pipeline (acquire source → apply a fix → submit for review) with OpenCode bundled as the open-source default. Nothing merges without human review, and fixed failures become regression datasets.
Same template and journeys-first structure; review feedback from #38 is incorporated.