Repository navigation
stream: keep a pending FIN schedulable after data is acknowledged - #2804
Conversation
|
@lixing-lx Thanks for offering to help validate! The fix is up in this PR — would you mind running it against your TrustTunnel test setup to confirm the stranded FIN case is resolved? |
|
Thanks @akasakariko. I validated Deterministic comparison (same dependency lock, BoringSSL 5.2.0, Rust 1.99.0): I appended the two original Issue #2800 reproducer functions unchanged in behavior to separate base/head checkouts. On base TrustTunnel downstream validation: I compiled both the client and the server against this exact PR's quiche production sources, checking the changed production files against the commit. Client/server Cargo target directories are separate. The server uses the source from TrustTunnel's dependency-upgrade PR #155, H2/H3 smoke tests pass, covering TCP/UDP, authentication and certificate rejection, half-close and concurrent byte-checked transfers. The sustained H3 run uses client BBRv2 with a 32-packet initial window and the server's unchanged default congestion settings, in a Linux arm64 Docker namespace with
Total: 8,817 content-and-EOF-verified transfers / 577,830,912 bytes, no missing EOF or transfer timeout. The same run passes 20 session recreations, cancellation of an old read during a blackhole, bounded blackhole dial failure and subsequent recovery. Kernel UDP receive-buffer/send-buffer error counters remain zero; the acceptance validator checks all phase durations/counts and recreation cases, not just the test harness exit code. The latency/loss relay uses approximately 80 ms ± 10 ms delay per direction and a 10 Mbps cap per direction. The fault conditions, source fingerprints, binary SHA256 values and per-run logs are retained locally. This confirms the standalone scheduling failures and the controlled TrustTunnel path on this head; it is not a claim of public-network or production-server acceptance. |
|
Thanks a lot @lixing-lx for the thorough validation, especially the downstream TrustTunnel run under loss and latency. Really appreciate it Maintainers, this is ready for review whenever you have a chance. Happy to address any feedback |
|
Updated in
Local validation on macOS arm64
GitHub now reports the PR as mergeable without conflicts |
…-ack tracking Port of cloudflare#2804 (head 38880fe) by akasakariko, fixing cloudflare#2800: a pending FIN could not be sent after its data was acknowledged and the frame carrying the FIN was lost. Replaces the local fin_sent/fin_lost/fin_only_pending approach from c3d2192 with the upstream fin_pending state. Also ports the prerequisite upstream commit af8788e (track FIN acknowledgements separately), since the 0.30.0 base completes the send side once data is acked, which would collect the stream before a lost FIN can be resent. In the fork the "peer stopped sending" guard is SendBuf::error, as 0.30.0 has no PeerStopSendingState. Covers additionally: lost PTO probe carrying FIN, FIN after retransmission that ends before an acked tail, no STREAM frame after RESET_STREAM. The c3d2192 regression tests are kept.
|
Test data for this PR, from a hysteria2 client built on quiche (CUBIC/BBRv2). Scenario: bulk upload of 128 MiB over a single HTTP/3-style stream (hysteria2 TCP proxy stream) to a sing-box server, with 20 ms added delay and 5% random packet loss on the server's inbound UDP path (netem). The application writes all the data, then the FIN afterwards. When earlier lost data is retransmitted before the pending FIN has been sent, the FIN is dropped from the send queue and never goes out, so the peer never sees end-of-stream and the upload sits until the 150 s timeout. Numbers (150 s timeout counts as a hang):
Our regression tests in the fork ( |
|
Friendly ping. I have updated the branch onto the latest master, and it merges cleanly. Since the last update, a downstream user has also posted independent test results on this PR and adopted this fix in their fork. Happy to adjust anything if reviewers have feedback. Thank you. |
ghedo
left a comment
There was a problem hiding this comment.
Thanks a lot for the fix, and sorry it took a while to review this.
Tracking the pending FIN in SendBuf looks like the right approach. The extra state is needed to send the FIN when retransmitted data ends before an acknowledged tail, without resending a FIN that is already in flight. I also confirmed that the two regression tests fail on master and pass with this change.
Most of the requested changes are about keeping the PR minimal.
- The reset checks in
retransmit_fin(),stream_lost_fin_is_not_retransmitted_after_resetand the related unit test assertions became redundant with the latest merge from master, which already drops lost STREAM frames on reset streams. For the same reason, please drop the "no STREAM frame after RESET_STREAM" item from the description and commit message. - The other new tests can mostly be folded into existing ones, as described inline.
I also left a couple of smaller suggestions inline.
7ff3c03 to
c84374e
Compare
|
All review points are addressed in c84374e
The branch is rebased onto current master and is a single commit Local validation on macOS arm64 with
Both folded regressions still fail on master and pass with this change |
ghedo
left a comment
There was a problem hiding this comment.
Thanks for the quick update. This looks much better. The FinState enum maps exactly onto the previous booleans, and I checked that the folded regression tests still fail on master and that the fin_with_tail case catches the simpler fix from #2800.
When I checked the folded tests with a few mutations, I found that the tests no longer check that a lost FIN stays acked, that writing the FIN again doesn't schedule another one, or that a reset clears the pending FIN. Sorry, my earlier comments weren't clear that some steps of the removed unit tests had no equivalent in the existing ones. The inline suggestions restore that coverage with a few lines in the existing tests.
I also think the reserve_for_write() change can be dropped, as described inline.
The FIN was never sent in two cases: - a lost STREAM frame carrying FIN whose data was acknowledged in another packet, since only lost zero-length FINs were requeued - data retransmitted before an already acknowledged tail, since the stream stopped being flushable once that data was sent Track the final size lifecycle in SendBuf as a single state, which is only meaningful once the final size is known: - it starts as pending when the final size becomes known - it becomes sent when a STREAM frame carrying FIN is emitted, or when the final size is carried by RESET_STREAM instead - it becomes acked when a STREAM frame carrying FIN is acknowledged - a lost FIN-bearing frame makes it pending again, unless it is acked A stream is flushable when only its pending FIN is left to send This replaces the zero-length FIN special cases and avoids resending a FIN that is still in flight Extend the existing tests to retain coverage of repeated FIN writes, lost acknowledged FINs, and pending FIN cleanup on reset Fixes cloudflare#2800
c84374e to
4393dcf
Compare
|
Updated in All four follow-up comments are addressed and replied to inline, and the branch remains a single commit Local validation on macOS arm64
The four original fault cases still fail on base The new Stable, Nightly and Semgrep workflows require maintainer approval and have not executed any jobs on this head |
ghedo
left a comment
There was a problem hiding this comment.
Thanks for addressing everything so quickly. This looks good to me, with two small test suggestions inline.
Check that an acked FIN stays acked when a stale queue entry emits it, and end the lost-probe test on the retransmitted FIN frame.
Summary
A FIN can remain unsent when its DATA is acknowledged in another packet or when retransmitted DATA ends before an acknowledged tail
SendBufas a singleFinStateTesting
Local checks use a generated
Cargo.lockandRUSTFLAGS="-D warnings"cargo +nightly fmt -- --checkcargo clippy --locked --features=async,ffi,qlog,rpk,congestion_window_unchecked_available --workspace --all-targets -- -D warningscargo test --locked --all-targets --features=async,ffi,qlog,rpk,congestion_window_unchecked_available --workspacecargo test --locked --doc --features=async,ffi,qlog,rpk,congestion_window_unchecked_available --workspaceFixes #2800