fix(libpeer): prevent DTLS receive stalls during stream startup

[zortos293]

Addresses a reproducible transport stall relevant to #36. This does not claim hardware confirmation or resolve the separate touch-mapping report.

The DTLS receive callback returned the same cached UDP datagram repeatedly, while dtls_srtp_read retried WANT_READ/WANT_WRITE without yielding. The network worker holds the peer mutex across this path, preventing UI polling and the startup watchdog from progressing when the read stalls.

• Consume each cached DTLS datagram once and return retry statuses to the peer loop.
• Keep established-session socket reads in the peer loop so DTLS cannot consume RTP/RTCP packets intended for the media path.
• Preserve an unconsumed datagram across retries and drain Mbed TLS's buffered records before fetching another packet. If buffered processing needs fresh input, continue socket polling rather than starving media.
• Keep pending receive storage bounded to the existing one-datagram buffer. TLS verification, SCTP reliability, and input packet formats are unchanged.

Both original regression tests fail on the old code and pass with the fix. Added source-level tests cover repeated receive calls, nonblocking read results, retained datagrams through 100 retry iterations, buffered TLS records, interleaved RTP, and close/error handling using controlled TLS and UDP I/O.

Validation: all 65 streaming/transport checks pass normally and under GCC ASan/UBSan, including the real-packet SCTP reliability suite. The supported pinned-container Switch build recompiles both modified dependency objects and produces SwitchNOW.nro.

Hardware acceptance still needs the affected Switch to retry negotiation and first-video startup. No touch-coordinate changes are included.

Summary by CodeRabbit

• Bug Fixes

  • Improved DTLS handling for non-blocking reads and buffered data.
  • Prevented unnecessary blocking while receiving DTLS traffic.
  • Improved processing of pending DTLS, SCTP, and application data.
  • Added handling for close notifications, invalid records, and receive buffer limits.

• Tests

  • Added coverage for non-blocking DTLS reads, peer reception, handshake processing, idle retries, and error scenarios.
  • Added streaming-host checks for DTLS behavior.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 1d7b27f3-8536-4a71-86c7-ccd8c1ed0e35

📥 Commits

Reviewing files that changed from the base of the PR and between e950d8d and 75bc9e5.

📒 Files selected for processing (6)

• extern/libpeer/src/dtls_srtp.c
• extern/libpeer/src/peer_connection.c
• scripts/test-streaming-host.sh
• tests/dtls_nonblocking_read_test.c
• tests/peer_dtls_loop_test.c
• tests/peer_dtls_receive_test.c

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

---

📝 Walkthrough

Walkthrough

The DTLS path now performs non-blocking single reads, tracks pending data, drains buffered input in peer_connection_loop, and handles DTLS data, errors, and close notifications through a shared helper. New tests cover read results, receive behavior, loop processing, and state transitions.

Changes

DTLS receive flow

Layer / File(s)	Summary
Single-read DTLS contract extern/libpeer/src/dtls_srtp.c, tests/dtls_nonblocking_read_test.c	dtls_srtp_read performs one mbedtls_ssl_read call and returns its result unchanged. The test covers error, close, empty, and positive-read results.
Buffered DTLS receive extern/libpeer/src/peer_connection.c, tests/peer_dtls_receive_test.c	peer_connection_dtls_srtp_recv serves buffered bytes, reports oversized-buffer errors, returns WANT_READ for completed transport, and uses non-blocking agent reads otherwise.
Pending DTLS loop processing extern/libpeer/src/peer_connection.c, tests/peer_dtls_loop_test.c, scripts/test-streaming-host.sh	peer_connection_loop tracks pending DTLS data, drains pending SSL input, routes application data to SCTP, handles errors and close notifications, and runs the new C tests.

Priority: ⬇️ Low

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Bug fix

Sequence Diagram(s)

sequenceDiagram
  participant peer_connection_loop
  participant peer_connection_read_dtls
  participant dtls_srtp_read
  participant SCTP
  peer_connection_loop->>peer_connection_read_dtls: process pending or checked SSL input
  peer_connection_read_dtls->>dtls_srtp_read: perform one DTLS read
  dtls_srtp_read-->>peer_connection_read_dtls: return bytes or status
  peer_connection_read_dtls->>SCTP: pass application data
  peer_connection_read_dtls-->>peer_connection_loop: update pending data or peer state

Loading

Merge Risk: ⚪ Minimal · up to 75bc9

The DTLS receive-flow change has targeted coverage for retries, retained datagrams, buffered records, and connection state transitions. No actionable merge-blocking risk remains.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 38 functions across 6 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: preventing DTLS receive stalls during stream startup.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

• Fix all pre-merge checks with AI

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch capy/investigate-hardware-startup

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}