Skip to content

Add cancellation handling to prevent UnobservedTaskException in Watcher#1827

Draft
tg123 wants to merge 4 commits into
kubernetes-client:masterfrom
tg123:pr-1814
Draft

Add cancellation handling to prevent UnobservedTaskException in Watcher#1827
tg123 wants to merge 4 commits into
kubernetes-client:masterfrom
tg123:pr-1814

Conversation

@tg123

@tg123 tg123 commented Jul 1, 2026

Copy link
Copy Markdown
Member

follow up copilot ver #1814
fix #1813

Cancelling a watch's CancellationToken while ReadLineAsync() is in flight left the original read task orphaned. When the transport tore the connection down, that task faulted with an unobserved IOException, surfacing as a TaskScheduler.UnobservedTaskException at finalization (e.g. during app shutdown).

Root cause: AttachCancellationToken in CreateWatchEventEnumerator wraps the read via task.ContinueWith(..., cancellationToken). On cancellation the continuation is cancelled immediately while the original ReadLineAsync() keeps running and later faults — with no one observing it.

Copilot AI review requested due to automatic review settings July 1, 2026 21:39
@kubernetes-prow

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tg123

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubernetes-prow
kubernetes-prow Bot requested a review from brendandburns July 1, 2026 21:39
@kubernetes-prow kubernetes-prow Bot added approved Indicates a PR has been approved by an approver from all required OWNERS files. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Jul 1, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes a watch-cancellation edge case where cancelling while an async read is in-flight can orphan the original task and later surface as TaskScheduler.UnobservedTaskException during finalization/shutdown.

Changes:

  • Update Watcher<T>.CreateWatchEventEnumerator to ensure faulted orphan tasks are observed, preventing UnobservedTaskException.
  • On newer TFMs, pass the CancellationToken directly into ReadLineAsync(...) to cancel the read itself.
  • Add a regression test that reproduces the orphaned-task scenario and asserts no UnobservedTaskException is raised.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
src/KubernetesClient/Watcher.cs Ensures faulted orphan tasks are observed; uses cancellation-aware read on newer TFMs.
tests/KubernetesClient.Tests/WatchTests.cs Adds regression coverage to detect UnobservedTaskException during watch cancellation/fault scenarios.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +1040 to +1044
var unobservedExceptions = new List<Exception>();
void Handler(object sender, UnobservedTaskExceptionEventArgs e)
{
unobservedExceptions.Add(e.Exception);
}
await Task.Delay(50).ConfigureAwait(true);
}

Assert.Empty(unobservedExceptions);
Copilot AI review requested due to automatic review settings July 25, 2026 08:50
@kubernetes-prow kubernetes-prow Bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jul 25, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

tests/KubernetesClient.Tests/WatchTests.cs:1044

  • TaskScheduler.UnobservedTaskException is raised on the finalizer thread, so this handler can be invoked concurrently with the test thread. Using List<Exception> here is not thread-safe, and not calling e.SetObserved() can allow the runtime to escalate the unobserved exception (environment-dependent), making the test flaky or terminating the process.
            var unobservedExceptions = new List<Exception>();
            void Handler(object sender, UnobservedTaskExceptionEventArgs e)
            {
                unobservedExceptions.Add(e.Exception);
            }

@tg123
tg123 marked this pull request as draft July 25, 2026 08:54
@kubernetes-prow kubernetes-prow Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 25, 2026
Copilot AI review requested due to automatic review settings July 25, 2026 09:01

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

tests/KubernetesClient.Tests/WatchTests.cs:1044

  • TaskScheduler.UnobservedTaskException is raised on the finalizer thread; mutating a List<Exception> from that handler is not thread-safe. Also, if an exception does occur, not calling e.SetObserved() can destabilize the test run depending on runtime settings. Prefer a thread-safe collection and mark the exception as observed while still asserting that the queue stays empty.
            var unobservedExceptions = new List<Exception>();
            void Handler(object sender, UnobservedTaskExceptionEventArgs e)
            {
                unobservedExceptions.Add(e.Exception);
            }

Comment on lines +24 to +41
if (cancellationToken.IsCancellationRequested)
{
return Task.FromCanceled<TResult>(cancellationToken);
}

// Observe any exception from the original task to prevent an
// UnobservedTaskException when the continuation below is cancelled
// before the original task faults (e.g. the transport tears down the
// connection after cancellation).
_ = task.ContinueWith(
static t => { _ = t.Exception; },
CancellationToken.None,
TaskContinuationOptions.OnlyOnFaulted | TaskContinuationOptions.ExecuteSynchronously,
TaskScheduler.Default);

// here to pass cancellationToken into task
return task.ContinueWith(t => t.GetAwaiter().GetResult(), cancellationToken);
}
Copilot AI review requested due to automatic review settings July 25, 2026 09:32

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (3)

tests/KubernetesClient.Tests/WatchTests.cs:1044

  • The UnobservedTaskException handler mutates a List<Exception> from the finalizer thread, which is a data race with the test thread's later Assert.Empty. Also, if this event fires unexpectedly, not calling e.SetObserved() can still impact the process under some configurations.
            var unobservedExceptions = new List<Exception>();
            void Handler(object sender, UnobservedTaskExceptionEventArgs e)
            {
                unobservedExceptions.Add(e.Exception);
            }

src/KubernetesClient.Classic/CancellationExtensions.cs:40

  • WaitAsync polyfill uses ContinueWith(...) without specifying TaskScheduler.Default / ExecuteSynchronously. This can capture a caller's current scheduler (e.g., UI/ASP.NET sync context) and run the continuation there, which is surprising for a general-purpose polyfill and can contribute to deadlocks.
            // here to pass cancellationToken into task
            return task.ContinueWith(t => t.GetAwaiter().GetResult(), cancellationToken);

src/KubernetesClient.Classic/CancellationExtensions.cs:27

  • WaitAsync returns a canceled task immediately when cancellationToken.IsCancellationRequested, but in that branch it never observes exceptions from the original task. If the original task later faults, this can still surface as TaskScheduler.UnobservedTaskException—the exact scenario this helper is meant to prevent.
            if (cancellationToken.IsCancellationRequested)
            {
                return Task.FromCanceled<TResult>(cancellationToken);
            }

Comment on lines 107 to 110

<Compile Include="..\KubernetesClient\Watcher.cs" />
<Compile Include="..\KubernetesClient\WatcherExt.cs" />
<Compile Include="..\KubernetesClient\LineSeparatedHttpContent.cs" />

Comment on lines +161 to +166
using var streamReader = await streamReaderCreator().WaitAsync(cancellationToken).ConfigureAwait(false);

for (; ; )
{
// ReadLineAsync will return null when we've reached the end of the stream.
var line = await AttachCancellationToken(streamReader.ReadLineAsync()).ConfigureAwait(false);
var line = await streamReader.ReadLineAsync(cancellationToken).ConfigureAwait(false);
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

UnobservedTaskException from orphaned ReadLineAsync task in Watcher<T>.CreateWatchEventEnumerator during cancellation

2 participants