Add RFC 25601: Typed Client API to Submit Issue Detection for Traces by Rohitkanithi · Pull Request #52 · mlflow/rfcs

Rohitkanithi · 2026-09-18T06:23:28Z

Summary

This RFC proposes adding a formal, typed public client API (submit_issue_detection and search_issues) to MlflowClient and formalizing the backend RPC endpoint (POST /api/2.0/mlflow/issues/invoke) for running issue detection on traces.

Tracking Issue: Add a typed client API to submit Issue Detection for traces mlflow#25601
Related PR: Add typed client API to submit Issue Detection for traces mlflow#25881
Authors: @Rohitkanithi, @adamgurary, @joshuawong-db, @tanghaoji

Signed-off-by: Rohitkanithi <63586900+Rohitkanithi@users.noreply.github.com>

…quence flows Signed-off-by: Rohitkanithi <63586900+Rohitkanithi@users.noreply.github.com>

Signed-off-by: Rohitkanithi <63586900+Rohitkanithi@users.noreply.github.com>

joshuawong-db

Thanks for making a RFC. I would suggest starting with the minimal API requirements

The routes to be added
Request/Response for each of the routes
Brief description in prose or pseudocode what this route does
Any important questions worth highlighting (eg job completion semantics).

This whole process may take a while.

Let's start with the minimal surface area and then expand from there as needed. There are also some claims in the RFC that look off and do not seem to match code

joshuawong-db · 2026-09-21T08:54:02Z

+│   mlflow.server.handlers._invoke_issue_detection_handler                        │
+│   ┌─────────────────────────────────────────────────────────────────────────┐   │
+│   │ 1. Authentication & Permission Check                                    │   │
+│   │    validate_can_update_experiment(experiment_id)                        │   │


This does not preserve the existing authorization contract. The current AJAX route requires both experiment UPDATE and USE on a caller-supplied secret_id; the proposed RPC is mapped only to validate_can_update_experiment, and that mapping also applies to the existing AJAX route. Please preserve the issue-detection validator and add tests for both REST and AJAX paths.

I've updated the RFC to preserve validate_can_invoke_issue_detection() so both experiment UPDATE and USE on secret_id remain enforced across REST and AJAX routes, and will include tests for both.

joshuawong-db · 2026-09-21T08:54:02Z

+        },
+    )
+
+    # 1. Authorization: Verify caller has write permissions on the experiment


The authorization contract also needs READ permission for every supplied trace_id, not only UPDATE on the destination experiment. Otherwise a caller who knows a trace ID from another experiment can run analysis over it and persist derived issue content into an experiment they control. Can we reuse the all-traces authorization pattern from validate_can_batch_get_traces?

Updated the spec to adopt the validate_can_batch_get_traces pattern to look up parent experiments and enforce READ permission across all supplied trace_ids.

joshuawong-db · 2026-09-21T08:54:02Z

+issues = client.search_issues(
+    experiment_id="101",
+    source_run_id=job.run_id,
+    filter_string="severity = 'high' AND status = 'pending'",


This exact example fails for two independent reasons: severity is not a supported issue-search field, and the linked implementation wraps the supplied filter in parentheses when adding source_run_id, which SearchIssuesUtils rejects

I've fixed the example to use supported search keys (status and source_run_id) with clean AND conjunctions and no parentheses.

joshuawong-db · 2026-09-21T08:54:02Z

+  // Categories of issues to inspect (e.g. "hallucination", "tool_error").
+  repeated string categories = 3;
+
+  // Optional LLM provider identifier (e.g. "openai", "anthropic", "bedrock").


What are the valid combinations and precedence rules for provider, model, secret_id, and endpoint_name? These fields currently describe an implicit union, and accepted combinations such as endpoint_name + secret_id without provider fail at runtime in #25881. Please make the two invocation modes explicit—ideally as one model URI or a typed union—and specify validation for every combination.

I made the two invocation modes explicit in the revised RFC: either AI Gateway (endpoint_name) or direct provider (provider + model, optional secret_id), with validation rejecting invalid combinations.

joshuawong-db · 2026-09-21T08:54:02Z

+```
+
+#### Implementation Details across Backends:
+* **`RestStore` & `DatabricksRestStore`**:


This overstates backend support. The linked implementation dispatches from RestStore, while DatabricksRestStore.submit_issue_detection() explicitly raises MlflowNotImplementedException. I suggest keeping Databricks explicitly out of scope for this RFC and documenting the actual support matrix, rather than implying that DatabricksRestStore implements this path.

I've updated the Out of Scope section to explicitly keep Databricks out of scope for this RFC since DatabricksRestStore raises MlflowNotImplementedException.

joshuawong-db · 2026-09-21T08:54:02Z

+┌─────────────────────────────────────────────────────────────────────────────────┐
+│                        ASYNC BACKGROUND EXECUTION ENGINE                        │
+│                                                                                 │
+│   invoke_issue_detection_job(job_id, run_id, experiment_id, trace_ids...)       │


This is not the existing function signature or launch path. invoke_issue_detection_job does not receive job_id; the handler passes the function to submit_job(), which creates that ID. Calling invoke_issue_detection_job(...) directly, as the later pseudocode does, would execute it synchronously and return a result dictionary rather than a job handle.

Updated the dispatch description to match the real server path where submit_job() creates the job_id and dispatches asynchronously.

joshuawong-db · 2026-09-21T08:54:02Z

+In `mlflow/store/tracking/abstract_store.py`:
+
+```python
+@abstractmethod


Could we model this after the existing optional issue-store methods: a concrete base method that raises MlflowNotImplementedException? That is also what #25881 implements and gives unsupported stores a defined failure instead of making the entire store hierarchy abstract.

Updated the store contract to specify a concrete base method on AbstractStore that raises MlflowNotImplementedException.

joshuawong-db · 2026-09-21T08:54:02Z

+│                                                                                 │
+│   ┌───────────────────────┐   ┌───────────────────────┐   ┌─────────────────┐   │
+│   │ 1. Load Trace Spans   ├──►│ 2. Execute LLM Judges ├──►│ 3. Aggregate &  │   │
+│   │    from Tracking Store│   │    via AI Gateway     │   │    Cluster Bugs │   │


This is not always an AI Gateway call. endpoint_name produces a gateway:/... URI, while provider + model invokes that provider directly using credentials injected into the job subprocess. Please treat these as distinct execution paths.

AI Gateway routing via gateway:/... when endpoint_name is used, vs direct provider execution using subprocess credentials.

joshuawong-db · 2026-09-21T08:54:02Z

+* **Server-Side Threading**:
+  Jobs execute inside the server's background thread/task runner. Resource limits and concurrency are governed by server configuration rather than client connection lifespan.
+* **Paginated Queries**:
+  `search_issues` supports cursor-based pagination (`page_token`, `max_results`) to ensure querying thousands of issues remains performant.


This is offset pagination encoded in an opaque page token, not cursor/keyset pagination: search_issues decodes an offset and applies query.offset(...).

Clarified that pagination uses token-encoded numeric offsets (query.offset(...)).

joshuawong-db · 2026-09-21T08:54:02Z

+│                                                                                 │
+│   ┌─────────────────────────────────┐     ┌─────────────────────────────────┐   │
+│   │ runs Table                      │     │ issues Table                    │   │
+│   │ - run_id                        │     │ - issue_id (UUID)               │   │


issue_id is stored as a string in the format iss-<uuid hex>, not as a UUID. The same diagram also omits the persisted REJECTED status and NOT_AN_ISSUE severity that the RFC lists later. Please make the schema diagram match the actual entity/database model.

Updated the schema to reflect the string iss- format and added REJECTED to status and NOT_AN_ISSUE to severity.

Signed-off-by: Rohitkanithi <63586900+Rohitkanithi@users.noreply.github.com>

Rohitkanithi · 2026-09-22T06:18:18Z

Hey @joshuawong-db, thanks for the review.

I've replied to all your inline comments and pushed an update (83f0abe) refactoring the RFC down to the minimal API surface area you suggested, with all routes, schemas, and auth contracts strictly aligned with the codebase.

Ready for review

Rohitkanithi added 3 commits September 18, 2026 01:20

Add RFC 25601: Typed Client API to Submit Issue Detection for Traces

dbe882f

Signed-off-by: Rohitkanithi <63586900+Rohitkanithi@users.noreply.github.com>

Update RFC 25601: Add Table of Contents, detailed architecture and se…

32ac35f

…quence flows Signed-off-by: Rohitkanithi <63586900+Rohitkanithi@users.noreply.github.com>

Update RFC 25601: Align with full technical design specification

8290be3

Signed-off-by: Rohitkanithi <63586900+Rohitkanithi@users.noreply.github.com>

This was referenced Sep 18, 2026

Add a typed client API to submit Issue Detection for traces mlflow/mlflow#25601

Open

Add typed client API to submit Issue Detection for traces mlflow/mlflow#25881

Closed

joshuawong-db requested changes Sep 21, 2026

View reviewed changes

Refactor RFC to minimal API surface area matching codebase

83f0abe

Signed-off-by: Rohitkanithi <63586900+Rohitkanithi@users.noreply.github.com>

Conversation

Rohitkanithi commented Sep 18, 2026

Summary

Uh oh!

joshuawong-db left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Rohitkanithi commented Sep 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants