add nld decision telemetry event by evelyn-with-warp · Pull Request #10875 · warpdotdev/warp

evelyn-with-warp · 2026-05-14T01:04:24Z

Description

Linked Issue

The linked issue is labeled ready-to-spec or ready-to-implement.
Where appropriate, screenshots or a short video of the implementation are included below (especially for user-visible or UI changes).

Testing

./script/run --features warpui/log_named_telemetry_events

I have manually tested my changes locally with ./script/run

Screenshots / Videos

ls

ls with file attached

** ls with file attached and ⌘ I**

hello in terminal mode without enabling onnx classifier

** approve after hello, should trigger AgentFollowUp whitelist**

natural language without enabling onnx classifier

hello with enabling onnx classifier, should trigger whitelisting

!hello with enabling onnx classifier, should trigger shell prefix

what's the weather should trigger nldclassifier

/plan should trigger app level heuristics

some video on latest commit
https://www.loom.com/share/480d640e565d4d28a40e1a5d15fc709c

Agent Mode

Warp Agent Mode - This PR was created via Warp's AI Agent Mode

oz-for-oss · 2026-05-14T01:04:36Z

@evelyn-with-warp

I'm starting a first review of this pull request.

You can view the conversation on Warp.

I completed the review and no human review was requested for this pull request.

Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).

Powered by Oz

oz-for-oss

Overview

This PR adds input-decision source tracking through the NLD classifiers and includes decision metadata plus optionally collected input text on InputBufferSubmitted telemetry.

Concerns

Locked config updates that go through the compatibility set_input_config path can retain a stale prior input_decision_source, so later submissions can report an unrelated source.
The image attachment path sets AttachmentForcedAi and then immediately calls the default helper, which overwrites the source back to ManualToggle.

Verdict

Found: 0 critical, 2 important, 0 suggestions

Request changes

Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).

Powered by Oz

oz-for-oss · 2026-05-14T01:17:23Z

        self.input_config = new_config;
+        if let Some(decision_source) = decision_source {
+            self.input_decision_source = Some(decision_source);
+        } else if !new_config.is_locked {


⚠️ [IMPORTANT] When decision_source is None and the new config is locked, this preserves the previous source. Existing callers that still use set_input_config can leave stale values like ShellPrefix or AttachmentForcedAi, so later submit telemetry can report a source from an unrelated prior transition. Clear the source when no source is provided, or require every locked transition to pass an explicit source.

oz-for-oss · 2026-05-14T01:17:23Z

+            InputDecisionSource::AttachmentForcedAi,
+            ctx,
+        );
        self.ensure_agent_mode_for_ai_features(true, ctx);


⚠️ [IMPORTANT] This second helper call overwrites the AttachmentForcedAi source set just above because the config is already in AI mode and set_input_config_internal_with_source still updates the source for unchanged configs. Remove this duplicate call so image submissions keep the attachment-specific decision source.

evelyn-with-warp · 2026-05-18T17:19:45Z

@szgupta I occasionally saw decision source is Null, but it did not reproduce robustly.

09:48:39.634 [INFO] [warpui_core::telemetry::event_store] Recorded telemetry event: Event {
    payload: NamedEvent {
        user_id: Some(
            "paKnCQIb4NVDXhTQambFVn6dnej1",
        ),
        anonymous_id: "9a826897-8e46-4716-afa7-3fcfc7bc1c42",
        name: "AgentMode.NaturalLanguageDetection.InputBufferSubmitted",
        value: Some(
            Object {
                "decision_source": Null,
                "input": String("approve"),
                "input_type": String("Shell"),
                "is_locked": Bool(false),
                "was_lock_set_with_empty_buffer": Bool(false),
            },
        ),
    },
    session_created_at: 2026-05-18T16:34:43.407171Z,
    timestamp: 2026-05-18T16:48:39.633855Z,
    contains_ugc: true,

Despite our code does imply if no NLD decision applied when is_locked is False the decision source will be Null, it's not clear to me why there's no NLD decision for unlocked sessions.
I'm okay with current implementation having coverage less than 100%, as long as the logged decision source is accurate

oz-for-oss · 2026-05-18T17:20:53Z

@evelyn-with-warp

I'm starting a first review of this pull request.

You can view the conversation on Warp.

I completed the review and no human review was requested for this pull request.

Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).

Powered by Oz

oz-for-oss

Overview

This PR adds a decision-source enum for natural language detection, threads that source through input mode state, and expands InputBufferSubmitted telemetry to include gated UGC plus the final decision source. The implementation preserves UGC routing via contains_ugc() and includes targeted tests for manual toggles, shell prefix handling, and attachment-forced AI.

Concerns

Suggestion: in crates/input_classifier/src/onnx/mod.rs, the detect_input_decision error path reports decision_source as NldClassifier while falling back to context.current_input_type. If inference returns an error, telemetry will look like a successful model decision and mask classifier failures; consider using the heuristic fallback path or adding a distinct source for current-input fallback.

Verdict

Found: 0 critical, 0 important, 1 suggestions

Approve with nits

Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).

Powered by Oz

evelyn-with-warp · 2026-05-18T17:58:31Z

/oz-review

oz-for-oss · 2026-05-18T17:58:44Z

@evelyn-with-warp

I'm re-reviewing this pull request in response to a review request.

You can view the conversation on Warp.

I completed the review and no human review was requested for this pull request.

Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).

Powered by Oz

oz-for-oss

Overview

This PR adds decision-source tracking to input classification and includes that metadata in the InputBufferSubmitted telemetry event, with the submitted input attached only through the existing AI UGC telemetry gate.

Concerns

No blocking correctness or security concerns found in the annotated diff.

Verdict

Found: 0 critical, 0 important, 0 suggestions

Approve

Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).

Powered by Oz

szgupta

Initial thoughts

szgupta · 2026-05-19T16:50:05Z

+    /// The source of the final input decision that is currently reflected in `input_config`.
+    input_decision_source: Option<InputDecisionSource>,


When is this None?

like the case i got #10875 (comment), although not robustly reproducible;
i'd rather leave it as none (trade coverage for accuracy) if there's some decision path that I am not aware of.

also for some case if it's set to enable auto detection, then it's okay to be None for NLD to fill in later e.g. in enable_auto_detection

szgupta · 2026-05-19T16:51:13Z

+    /// The source of the final input decision that is currently reflected in `input_config`.
+    input_decision_source: Option<InputDecisionSource>,


"input_decision" is vague - can we be more clear here?

agreed on this. maybe nld_decision_source is better?

szgupta · 2026-05-19T16:51:55Z

+#[cfg_attr(not(target_family = "wasm"), async_trait)]
+#[cfg_attr(target_family = "wasm", async_trait(?Send))]
+impl InputClassifier for OnnxClassifier {
+    async fn detect_input_decision(


Does this mean checking the input decision happens separately from actually running NLD? Feel like the detect_input_type API ought to return the type and why that decision was made

it was mostly a rename. we did append source besides decision. Reverted it back to avoid confusion 557db91

vorporeal · 2026-05-19T19:48:56Z

    /// The timestamp of the last time the input mode was switched, if the switch was to AI mode and
    /// it was autodetected. Else, `None`.
    last_ai_autodetection_ts: Option<Instant>,
+    /// The source of the final input decision that is currently reflected in `input_config`.


if this is tied to some data in InputConfig, why is it not part of InputConfig?

also seems like it might simplify some other code

@vorporeal it was intended to say the source will be set in set_input_config_internal. Will update the comment to be less confusing; The decision source is dependent on user input text, and will be decided in runtime (e.g if there's ONNX panic, the source will be NLDHeuristicClassifier instead of NLDClassifier..etc), and therefore i think it's not appropriate to put it in input config, which aims to track the mode state (it's logically possible to have different decision source in one mode state)
i also didn't get why put nld_decision_source in inputConfig will simplify this implementation, as we will still need to update everywhere we call set_input_config.
Pls lmk if i oversee some potential shortcut here

szgupta · 2026-05-20T16:20:45Z

    /// it was autodetected. Else, `None`.
    last_ai_autodetection_ts: Option<Instant>,
+    /// The source of the final input decision upon submission
+    nld_decision_source: Option<NldDecisionSource>,


nit: should we call this last_ai_autodetection_decision_source to be consistent with the property above? even better, wrap the two into a mini struct!

szgupta · 2026-05-20T16:21:18Z

-        self.set_input_config_internal(current_config.with_input_type(input_type), ctx);
+        self.set_input_config_internal(
+            current_config.with_input_type(input_type),
+            Some(NldDecisionSource::ManualToggle),


hrm, how are we sure this is a manual toggle? this API is in a model, not a ui view

szgupta · 2026-05-20T16:22:21Z

    let index = HINT_INDEX.fetch_add(1, Ordering::Relaxed) % AGENT_MODE_HINT_OPTIONS.len();
    AGENT_MODE_HINT_OPTIONS[index]
 }
+fn submitted_nld_decision_source(input_model: &BlocklistAIInputModel) -> Option<NldDecisionSource> {


What is this doing?

removed; it seems not necessary b6f6fe2

szgupta · 2026-05-20T16:23:02Z


                            self.ai_input_model.update(ctx, |ai_input_model, ctx| {
-                                ai_input_model.set_input_config(
+                                ai_input_model.set_input_config_with_source(


nit: i'd leave this API as set_input_config and make NldDecisionSource required. There shouldn't be a single callsite that can't specify why we chose to change the input type, right?

szgupta · 2026-05-20T16:24:45Z


+/// The source of the final input type decision applied to the user input.
+#[derive(Debug, Copy, Clone, PartialEq, Eq, Serialize, Deserialize)]
+pub enum NldDecisionSource {


I think we should decouple the decision source in the input_classifier crate vs the application level decision source. The one coming from this crate should only include the sources that it can actually produce (stuff in the NLD pipeline), whereas the one in the application should be like

enum AiAutodetectionSource { ManualToggle, ..., ClassifierSource(ClassifierSource) }

(naming TBD)

szgupta · 2026-05-20T16:24:57Z

-    }
-
-    async fn classify_input(
+    async fn classify_input_with_source(


nit: let's keep the name as classify_input

szgupta · 2026-05-20T16:25:12Z

-        _context: &Context,
-    ) -> anyhow::Result<ClassificationResult> {
+        context: &Context,
+    ) -> anyhow::Result<ClassificationWithSource> {


Can we just add the source to the existing ClassificationResult?

Co-Authored-By: Oz <oz-agent@warp.dev>

… failed cases

Co-Authored-By: Oz <oz-agent@warp.dev>

szgupta · 2026-05-21T21:50:22Z


+/// Sources produced by the NLD pipeline.
+#[derive(Debug, Copy, Clone, PartialEq, Eq, Serialize, Deserialize)]
+pub enum NldDecision {


nit: InputClassifierDecisionSource

let's avoid "nld" since we don't really use that term in the codebase today

szgupta · 2026-05-21T21:51:46Z

+    NldClassifierFallbackHeuristic,
+    NldClassifierFallbackCurrentInput,
+    Denylist,
+    HistoryMatch,


The input_classifier crate doesn't check for history

szgupta · 2026-05-21T21:53:08Z

+    NldClassifierFallbackCurrentInput,
+    Denylist,
+    HistoryMatch,
+    OneOffWhitelist,


Can you break this out into the two allowlists: "ShellCommandAllowlist" vs "NaturalLanguageAllowlist" vs "NaturalLanguageFollowupAllowlist"

szgupta · 2026-05-21T21:53:17Z

+    NldClassifier,
+    NldClassifierFallbackHeuristic,
+    NldClassifierFallbackCurrentInput,
+    Denylist,


Which denylist?

AISettings.autodetection_command_denylist to block auto detection run ; also moved to app level tracking now

szgupta · 2026-05-21T21:53:28Z


+/// Sources produced by the NLD pipeline.
+#[derive(Debug, Copy, Clone, PartialEq, Eq, Serialize, Deserialize)]
+pub enum NldDecision {


All of these variants should be more specific and documented please!

szgupta · 2026-05-21T21:54:03Z

+        &self,
+        input: ParsedTokensSnapshot,
+        context: &Context,
+    ) -> (InputType, NldDecision) {


Instead of a tuple, can you create a new return type (struct) with these two properties?

szgupta · 2026-05-21T21:55:44Z

+    /// App level settings from explicit user actions.
+    AppLevelOverride(AppLevelOverride),
+    /// App level implicit / side-effect transitions from other UI flows.
+    AppLevelHeuristic(AppLevelHeuristic),


Hrm, I feel like we should collapse these into one. In fact, I would change this type to

enum InputTypeAutoDetectionSource { InputClassifier(InputClassifierDecisionSource), HistoryMatch, ... // all the other app types inlined here }

szgupta

Awesome, thank you for iterating!

szgupta · 2026-05-22T14:59:02Z

+
+/// The source of the final input type decision applied to the user input.
+#[derive(Debug, Copy, Clone, PartialEq, Eq, Serialize, Deserialize)]
+pub enum InputTypeAutoDetectionSource {


nit: this is more like InputTypeDecisionSource - because a lot of these don't go through "auto-detection" (e.g. voice input)

szgupta · 2026-05-22T14:59:37Z

+    /// Image / file attachment in progress force-locked AI mode.
+    AttachmentForcedAi,


nit: I would just call this "AttachedFile"

szgupta · 2026-05-22T15:00:02Z

+    /// First token matched the autodetection command denylist.
+    Denylist,


Can you make this more specific (i.e. which denylist)

szgupta · 2026-05-22T15:01:35Z

        } else {
            AISettings::as_ref(ctx).is_ai_autodetection_enabled(ctx)
        };
+        let initial_decision_source = None;


Can you just inline this? no need for variable declaration

szgupta · 2026-05-22T15:01:41Z

    pub fn input_config(&self) -> InputConfig {
        self.input_config
    }
+    pub fn last_ai_autodetection_source(&self) -> Option<InputTypeAutoDetectionSource> {


nit: newline above

szgupta · 2026-05-22T15:04:30Z

+/// The source of the final input type decision applied to the user input.
+#[derive(Debug, Copy, Clone, PartialEq, Eq, Serialize, Deserialize)]


Can you move this down, past the imports

szgupta · 2026-05-22T15:05:16Z

            TelemetryEvent::AgentExitedShellProcess { .. } => true,
            TelemetryEvent::CreateProjectPromptSubmitted { .. } => false,
            TelemetryEvent::CreateProjectPromptSubmittedContent { .. } => true,
+            TelemetryEvent::InputBufferSubmitted { .. } => false,


revert this? it was fine before

szgupta · 2026-05-22T15:07:24Z

-                return InputType::Shell;
+                return InputClassificationResult::new(
+                    InputType::Shell,
+                    InputClassifierDecisionSource::ShellHeuristic,


Should we break this up into two types rather than use the same one here and in is_likely_shell_command?

szgupta · 2026-05-22T15:07:48Z

+    /// The classifier source that produced this classification.
+    pub source: InputClassifierDecisionSource,


nit: instead of making this pub, define a getter

szgupta · 2026-05-22T15:08:22Z

+#[cfg(test)]
+mod tests {


nit: move to mod_tests file

cla-bot Bot added the cla-signed label May 14, 2026

oz-for-oss Bot reviewed May 14, 2026

View reviewed changes

evelyn-with-warp marked this pull request as draft May 14, 2026 19:49

evelyn-with-warp marked this pull request as ready for review May 18, 2026 17:20

evelyn-with-warp requested a review from szgupta May 18, 2026 17:20

oz-for-oss Bot reviewed May 18, 2026

View reviewed changes

szgupta reviewed May 19, 2026

View reviewed changes

vorporeal reviewed May 19, 2026

View reviewed changes

evelyn-with-warp requested review from szgupta and vorporeal May 19, 2026 22:19

szgupta reviewed May 20, 2026

View reviewed changes

evelyn-with-warp and others added 8 commits May 20, 2026 10:40

add nld decision telemetry event

ee63712

Co-Authored-By: Oz <oz-agent@warp.dev>

staging cp for logic improvement

db067a0

adding NldClassifierFallbackCurrentInput for transient NLD classifier…

6baa915

… failed cases

rename variables

096f507

rename to NldDecisionSource

c03ea0a

Add block ID to InputBufferSubmitted telemetry event

62db581

clean up comment

13a4536

rename

5c54c6b

Co-Authored-By: Oz <oz-agent@warp.dev>

evelyn-with-warp force-pushed the evelyn/add-nld-submission-telemetry branch from 557db91 to 5c54c6b Compare May 20, 2026 17:41

evelyn-with-warp added 4 commits May 20, 2026 12:47

rm submitted_nld_decision_source

b6f6fe2

refactor nlddecisionsource

9a7762f

rearrange source category

d18a1f2

refactor: split input type decision source to app and nld

bcea434

evelyn-with-warp added 5 commits May 20, 2026 17:56

add app level heuristic input type decision source

20ed3c8

add input_type_decision_source to enter_ai_mode

416ed50

fix compile error

bc82e03

add decision source to ensure_agent_mode_for_ai_features

e324526

updated a few heuristics

a67c7ad

evelyn-with-warp requested a review from szgupta May 21, 2026 17:36

szgupta reviewed May 21, 2026

View reviewed changes

evelyn-with-warp added 9 commits May 21, 2026 15:18

cp checking

28f7d38

cp

9643ea5

check point for reorg inputtypeautodetectionsource

4e5db21

fix build error

2167e9c

use input classifier to replace nld

bdff07c

rename allowlist telemetry

d9c5dc4

add comment to InputClassifierDecisionSource

66845db

create struct InputClassificationResult

8ad7133

cargo fmt

93d972b

evelyn-with-warp requested a review from szgupta May 22, 2026 06:47

szgupta approved these changes May 22, 2026

View reviewed changes

		/// The source of the final input decision that is currently reflected in `input_config`.
		input_decision_source: Option<InputDecisionSource>,

		/// Image / file attachment in progress force-locked AI mode.
		AttachmentForcedAi,

		/// First token matched the autodetection command denylist.
		Denylist,

		/// The source of the final input type decision applied to the user input.
		#[derive(Debug, Copy, Clone, PartialEq, Eq, Serialize, Deserialize)]

		/// The classifier source that produced this classification.
		pub source: InputClassifierDecisionSource,

		#[cfg(test)]
		mod tests {

Conversation

evelyn-with-warp commented May 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Linked Issue

Testing

Screenshots / Videos

Agent Mode

Uh oh!

oz-for-oss Bot commented May 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

oz-for-oss Bot left a comment

Choose a reason for hiding this comment

Overview

Concerns

Verdict

Uh oh!

oz-for-oss Bot May 14, 2026

Choose a reason for hiding this comment

Uh oh!

oz-for-oss Bot May 14, 2026

Choose a reason for hiding this comment

Uh oh!

evelyn-with-warp commented May 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

oz-for-oss Bot commented May 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

oz-for-oss Bot left a comment

Choose a reason for hiding this comment

Overview

Concerns

Verdict

Uh oh!

evelyn-with-warp commented May 18, 2026

Uh oh!

oz-for-oss Bot commented May 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

oz-for-oss Bot left a comment

Choose a reason for hiding this comment

Overview

Concerns

Verdict

Uh oh!

szgupta left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

evelyn-with-warp May 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

evelyn-with-warp May 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

evelyn-with-warp May 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

evelyn-with-warp commented May 14, 2026 •

edited

Loading

oz-for-oss Bot commented May 14, 2026 •

edited

Loading

evelyn-with-warp commented May 18, 2026 •

edited

Loading

oz-for-oss Bot commented May 18, 2026 •

edited

Loading

oz-for-oss Bot commented May 18, 2026 •

edited

Loading

evelyn-with-warp May 19, 2026 •

edited

Loading

evelyn-with-warp May 20, 2026 •

edited

Loading

evelyn-with-warp May 19, 2026 •

edited

Loading

evelyn-with-warp May 22, 2026 •

edited

Loading

szgupta May 21, 2026 •

edited

Loading