Skip to content

test: add e2e regression tests for issue #1138 — AI insert between existing lines#1270

Open
svarlamov wants to merge 1 commit into
mainfrom
devin/1777998996-test-issue-1138-ai-insert-between
Open

test: add e2e regression tests for issue #1138 — AI insert between existing lines#1270
svarlamov wants to merge 1 commit into
mainfrom
devin/1777998996-test-issue-1138-ai-insert-between

Conversation

@svarlamov
Copy link
Copy Markdown
Member

@svarlamov svarlamov commented May 5, 2026

Summary

Adds three e2e tests using the TestRepo harness with custom checkpoints to verify whether #1138 still reproduces.

Finding: The bug reproduces. When AI inserts content at multiple positions in a single edit, the line ranges for later insertion blocks are truncated, causing the tail lines to be misattributed as unknown/untracked instead of AI.

Tests added

Test Result Description
test_ai_insert_between_existing_methods_issue_1138 ✅ Pass Single AI method block inserted between two existing methods. All lines correctly AI-attributed.
test_ai_insert_single_line_between_existing_lines_issue_1138 ✅ Pass Single AI line inserted between existing lines. Correctly attributed.
test_ai_insert_multiple_blocks_between_existing_issue_1138 ❌ Fail (ignored) Two AI method blocks inserted at different positions in a single edit. Lines 19-20 of the second block are misattributed as Test User instead of mock_ai, confirming the lineRanges bug.

Blame output from the failing test

Lines 16-18: mock_ai     ← correct (first 3 lines of second AI block)
Line  19:    Test User   ← WRONG (closing brace of helperBC)
Line  20:    Test User   ← WRONG (trailing blank line)

The third test is marked #[ignore] so CI passes. Remove it once the underlying line-range calculation is fixed.

Review & Testing Checklist for Human

  • Run the ignored test to confirm the reproduction: task test TEST_FILTER=test_ai_insert_multiple_blocks EXTRA_TEST_BINARY_ARGS="--ignored" NO_CAPTURE=true
  • Verify the two passing tests cover the expected single-block insertion scenarios

Notes

  • All three tests use explicit fs::write + manual checkpoint human / checkpoint mock_ai calls (not set_contents) to precisely replicate real AI agent checkpoint flows.
  • The bug appears specific to multi-position insertions in a single edit — single-block insertions work correctly.

Link to Devin session: https://app.devin.ai/sessions/c78741bb99c14c0ca7ce1536292df196
Requested by: @svarlamov


Open in Devin Review

Three tests using the TestRepo harness with custom checkpoints:

1. test_ai_insert_between_existing_methods_issue_1138 — PASSES
   AI inserts a single method block between two existing methods.
   All inserted lines are correctly attributed as AI.

2. test_ai_insert_single_line_between_existing_lines_issue_1138 — PASSES
   AI inserts a single line between existing lines.
   Correctly attributed.

3. test_ai_insert_multiple_blocks_between_existing_issue_1138 — FAILS (ignored)
   AI inserts two method blocks at different positions in a single edit.
   The tail lines of the second insertion block are misattributed as
   'Test User' instead of 'mock_ai', confirming the lineRanges bug
   described in #1138.

The third test is marked #[ignore] since it reproduces the bug.
Refs: #1138

Co-Authored-By: Sasha Varlamov <sasha@sashavarlamov.com>
@devin-ai-integration
Copy link
Copy Markdown
Contributor

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

@CLAassistant
Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

Copy link
Copy Markdown
Contributor

@devin-ai-integration devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 3 additional findings.

Open in Devin Review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants