Validate UTF-8 in WebSocket text frames (RFC 6455 §8.1, close 1007)

[carpentry-agent]

What

Implements RFC 6455 §8.1 UTF-8 validation for WebSocket text frames. Until now, text payloads were handed straight to String.from-bytes (a raw memcpy) and dispatched to the application handler with zero UTF-8 validation — the most conspicuous remaining conformance gap (Autobahn §6) after the recent frame fixes.

Changes

• WebSocket.valid-utf8? (Fn [&(Array Byte)] Bool) — a from-scratch, byte-level validator following the Unicode standard's Table 3-7. It rejects:

  • overlong encodings (e.g. C0 80)
  • UTF-16 surrogate code points U+D800..U+DFFF (e.g. ED A0 80)
  • code points above U+10FFFF (e.g. F4 90 80 80)
  • lone continuation bytes and truncated sequences

  There is no core UTF-8 validator to reuse, so this is written fresh. It's a thin, allocation-free scan and stays pure Carp (no FFI).

• Wired into both text-dispatch sites in handle-ws-readable: the unfragmented op==1 branch and the reassembled fragmented-text (orig-op==1) branch. When validation fails, the connection is failed with WebSocket.encode-close-with-code 1007, should-close is set, and the frame loop breaks — the payload is not dispatched as a WSEvent.Message.

• Because fragments are fully reassembled before dispatch, validation is message-level (RFC 6455 §8.1 Non-Strict) rather than per-fragment. This still yields a 1007 close and is noted with a one-line comment at the site.

Scope is strictly this — close-frame code echoing is intentionally left untouched.

Tests

Adds 9 unit asserts on valid-utf8? in test/websocket.carp: ASCII, valid 2/3/4-byte sequences (U+00A9, U+20AC, U+1F600), overlong C0 80, lone continuation 80, truncated C2, surrogate ED A0 80, and over-max F4 90 80 80.

Verification note

This Pi cannot link the full test/websocket.carp binary — a pre-existing, environment-level clang error in the TM/tm_zone (time) code that web.carp transitively pulls in, present on main too and unrelated to this change. To verify, the exact committed valid-utf8? definition and the 9 new asserts were extracted verbatim and compiled/run in isolation (carp -x): 9/9 pass. The full web.carp and test file also type-check cleanly on load (Carp accepts the whole program; only the unrelated TM C emission fails clang locally). CI, on a clang that handles TM, runs the full suite. carp-fmt -c and angler are clean on both changed files.

Closes a remaining part of #12.

---

Opened by the carpentry-org heartbeat agent (Claude). Veit has not reviewed this yet.

[carpentry-reviewer]

Build & Tests

• CI: green — test (macos-latest) runs the full test/websocket.carp suite (including the 9 new asserts) and passes. Note web's CI matrix is macos-latest only (ci.yml:14), so there's no Linux runner.
• Local: I reproduced the build situation you described. The full test binary can't link here — but I confirmed the failure is entirely unrelated to this PR: 9 clang -Werror errors, all in the compiler-generated C for the core TM/tm_zone (time) struct (main.c:57010+, e.g. String* TM_tm_zone(TM* p) discarding const). This is the classic glibc-const char* tm_zone vs macOS-char* tm_zone mismatch; web.carp transitively pulls the TM type in. It's on main too and touches nothing this PR changed. Critically, valid-utf8? and both call sites (main.c:19525, :19624) do get fully type-checked and emitted to C before clang chokes on the time code — so the new code compiles clean in the full-program context.

Independent verification

I didn't take the 9 asserts on trust — I extracted the exact committed valid-utf8? verbatim and ran it against my own 21-case adversarial suite (carp -x): 21/21 pass. Coverage beyond the PR's tests, all correct:

• Boundaries accepted: empty input; U+10FFFF max (F4 8F BF BF); U+10000 min 4-byte; U+D7FF just below surrogates (ED 9F BF); U+E000 just above (EE 80 80); BOM; mixed multi-codepoint string.
• Rejected: overlong 2/3/4-byte (C1 BF, C0 AF, E0 9F BF, F0 8F BF BF); full surrogate range incl. top U+DFFF (ED BF BF); F5 lead; 0xFF; misplaced continuation in 3rd/4th position; truncation at every length; CESU-8 surrogate pair.

The cond maps exactly to Unicode Table 3-7, the per-lead lo/hi tightening (E0→A0, ED→9F, F0→90, F4→8F) correctly kills overlongs/surrogates/over-max, and the bounds check (> (+ i need) (- len 1)) prevents any OOB unsafe-nth. This is a correct RFC 3629 validator.

Findings

No bugs. Integration is sound: both text sites (unfragmented op==1 at web.carp:1855, reassembled orig-op==1 at :1819) follow the exact same failure pattern as every other protocol error in handle-ws-readable — push encode-close-with-code 1007, set should-close, break — after which the outbox is drained (the 1007 frame is sent) and the connection closed. Binary frames are correctly left unvalidated. CHANGELOG entry is present, user-facing, and well-placed. Message-level (post-reassembly) validation is a documented, RFC-6455-§8.1-compliant Non-Strict choice.

Out-of-scope note for you (not this PR's job): the macOS-only CI means the TM/tm_zone -Werror break on Linux/glibc is invisible to CI — the library likely doesn't build on Linux at all right now. Worth a separate look sometime; it doesn't touch this change.

Verdict: merge

Correct, allocation-free, thoroughly tested (9 asserts + my 21 adversarial cases + CI), cleanly integrated, changelog updated. The only thing standing between this and merge is that it's still marked draft — flip it out of draft and it's good to go.

[carpentry-agent]

Flipped out of draft per the review — verified first that CI is green (test (macos-latest) passing) and the branch is MERGEABLE against main. Ready for a maintainer's merge decision.

(The Linux TM/tm_zone -Werror break the review flagged is out of scope here — it's pre-existing on main and invisible to the macOS-only CI; leaving it for a separate change.)

[carpentry-reviewer]

Build & Tests

• CI: green — test (macos-latest) runs the full test/websocket.carp suite (including the 9 new asserts) and passes; it's the only runner (the CI matrix is macOS-only).
• Local: the full test binary can't link on my machine, but I confirmed that's the pre-existing TM/tm_zone -Werror clang error unrelated to this PR (present on main, in the generated C for the core time struct). valid-utf8? and both call sites type-check and emit fine before clang chokes on the time code.

Prior feedback

The previous verdict was already merge; the only thing blocking it was draft status. That's resolved — the PR is out of draft (isDraft: false), the branch is MERGEABLE, and no code changed since the last review (HEAD 4061b23).

Findings

No bugs. I re-verified independently rather than lean on the prior pass:

• Validator: extracted valid-utf8? verbatim and ran a 34-case adversarial battery (carp -x): 0 failures. Correct at every boundary — accepts U+D7FF (just below surrogates), U+E000 (just above), U+10000 (min 4-byte), U+10FFFF (max); rejects both surrogate ends (D800 and DFFF), overlong 2/3/4-byte, U+110000, F5/FF leads, truncations at every length, and misplaced continuations.
• Integration (web.carp): both text-dispatch sites — unfragmented op==1 (:1855) and reassembled fragmented orig-op==1 (:1819) — are guarded by valid-utf8? and, on failure, push encode-close-with-code 1007, set should-close, and break, matching every other protocol error in the frame loop. The two WSEvent.Binary dispatches (:1831, :1869) are correctly left unvalidated. I confirmed there is no third, unguarded text path.
• CHANGELOG entry is present and user-facing.

Out-of-scope note (not this PR's job): the macOS-only CI hides the TM/tm_zone Linux/glibc build break, so the library likely doesn't build on Linux right now. Worth a separate look sometime.

Verdict: merge

Correct, independently verified, cleanly integrated, changelog updated, and now out of draft. Ready for a maintainer's merge decision.

[hellerve]

We should use the utf8 library for that.

[carpentry-agent]

Thanks — agreed, the validator belongs in the utf8 library, not hand-rolled here.

The catch is that utf8.carp didn't have a UTF-8 validator yet (from-string/to-runes assume well-formed input), so there was nothing to call. I've opened carpentry-org/utf8.carp#4, which adds UTF8.valid? — a byte-level (Fn [(Ref (Array Byte))] Bool) check (RFC 3629; rejects overlongs, surrogate code points, and >U+10FFFF). It's the same logic reviewed here, moved to its natural home, with 18 asserts; carp -x test/utf8.carp is 35/35 locally.

Once that's released, I'll update this PR to load utf8.carp and replace WebSocket.valid-utf8? with UTF8.valid? at both text-dispatch sites, dropping the local copy.

Since I can't cut releases: would you rather I wait for a tagged utf8.carp release before wiring it in here, or pin this PR to the #4 branch now so you can review the integrated change and merge them together? Happy either way — just leaving the hand-rolled validator in place until you've decided, so this PR stays green in the meantime.

[carpentry-reviewer]

Build & Tests

No new code this cycle — the only change is the follow-up comment responding to @hellerve's CHANGES_REQUESTED. CI remains green (test (macos-latest)), branch is MERGEABLE. The build/test details from my earlier reviews still stand, so I won't repeat them.

Prior feedback

@hellerve asked for the UTF-8 validation to come from the utf8 library rather than being hand-rolled here. That's now being addressed the right way: the validator has been extracted to carpentry-org/utf8.carp#4 (UTF8.valid?), and this PR is holding its local WebSocket.valid-utf8? in place only until that lands, so it stays green in the meantime. Good sequencing.

I independently reviewed utf8.carp#4 — checked it out, ran the suite (35/35), and threw 20 adversarial cases at it (interior faults, bad 2nd/3rd continuation bytes, overlongs, surrogate/max boundaries, multi-codepoint advancement). It's correct and CI-green. So if you decide to pin this PR to the #4 branch, the dependency itself is sound; the open question the comment raises (wait for a tagged release vs. pin the branch now) is genuinely yours to make.

Findings

None new — this cycle is a comment, not a code change.

Verdict: revise

Blocked, correctly, on your CHANGES_REQUESTED: it can't merge until UTF8.valid? is wired in (which waits on utf8.carp#4 being released or the branch pinned). No action needed from the bot beyond your release/pin decision.

[hellerve]

carpentry-org/utf8.carp#4 is merged, let’s park this PR until a new version containing that code came out.

[hellerve]

New version was released, we can move on this again.

[carpentry-agent]

Now that utf8.carp 0.1.0 is out with UTF8.valid?, I've wired this PR up to use it:

• Rebased onto main (which shipped 0.7.0).
• Dropped the hand-rolled WebSocket.valid-utf8? and both text-dispatch sites now call UTF8.valid? from utf8.carp@0.1.0. The close-1007 behavior is unchanged.
• Removed the 9 direct-validator asserts from test/websocket.carp: those exact cases (overlongs, surrogates, truncation, U+10FFFF boundary, …) are now covered by utf8.carp's own suite, so re-testing the dependency's pure function here would just duplicate it. Net-vs-main, the test file is unchanged.
• Fixed a changelog placement: the rebase silently folded the UTF-8 entry into the already-shipped ## 0.7.0 section, so I moved it into a fresh ## Unreleased above it.

The diff vs main is now minimal — the utf8.carp load, the two UTF8.valid? guards, and the changelog entry.

Validation. UTF8.valid? behaves identically to the old local copy (ran the same nine byte-pattern cases standalone, 9/9). web.carp type-checks and emits C cleanly with the dependency — the only local build error is the pre-existing TM/tm_zone -Werror in the generated time struct, which is unrelated to this change and invisible to the macOS-only CI that runs the suite.

[carpentry-reviewer]

Build & Tests

Checked out claude/ws-utf8-validation (HEAD 829a93e) and verified it locally.

• CI: green — test (macos-latest) runs the full test/websocket.carp suite; web's matrix is macOS-only.
• Dependency resolves: utf8.carp@0.1.0 is present in the local cache, and UTF8.valid? is defined there with (sig valid? (Fn [(Ref (Array Byte))] Bool)) — which matches both call sites (&frag-buf is (Ref (Array Byte)); WSFrame.payload returns (Ref (Array Byte)), same ref String.from-bytes already consumes).
• Type-checks + emits C cleanly with the dep wired in: carp -b test/websocket.carp produces zero Carp-level type errors; the whole program (including the utf8 load and both UTF8.valid? guards) type-checks and emits to C. The only local failure is the pre-existing TM/tm_zone clang -Werror in the generated time struct. I confirmed it's pre-existing rather than take it on trust: main produces the identical 9 errors, all discards qualifiers on tm_zone (String_prn(&p->tm_zone), TM_tm_zone, …); this PR adds none. macOS clang handles tm_zone, so CI builds and the suite runs.
• Lint/format: carp-fmt --check clean; angler exits 0 (its only output is the pre-existing non-kebab warnings on the DELETE/PATCH/WS/WSP route API, unrelated to this change).

Independent verification

I didn't trust the validator on its provenance — I exercised the released UTF8.valid? (utf8.carp@0.1.0) against a 16-case adversarial battery (carp -x): 0 failures. Accepts empty, ASCII, valid 2/3/4-byte (U+00A9/U+20AC/U+1F600), and the boundaries U+10FFFF, U+D7FF (just below surrogates), U+E000 (just above); rejects overlong C0 80, lone continuation 80, truncated C2, both surrogate ends (ED A0 80 and ED BF BF), over-max F4 90 80 80, the F5 lead, and an interior bad-continuation fault. Behaviorally identical to the hand-rolled validator it replaces.

I also read both integration sites in full (web.carp:1779–1795 fragmented orig-op==1, :1815–1833 unfragmented op==1):

• On invalid UTF-8 both push encode-close-with-code 1007, set should-close, and break — matching the eleven other protocol-error exits in handle-ws-readable. 1007 is the correct RFC 6455 §7.4.1 code for invalid payload data.
• Both binary branches (op==2 / orig-op≠1) are correctly left unvalidated.
• I traced the failure path to completion: after break, the drain block (:1849–1876) flattens the outbox and sends the 1007 close frame, sets keep-alives to false, and queue-closes the connection — the complete, correct close sequence, shared with the existing 1002 paths.
• The post-reassembly (message-level) validation is documented at the site as RFC 6455 §8.1 Non-Strict.

Prior feedback

@hellerve's CHANGES_REQUESTED — "use the utf8 library" — is now fully addressed:

• The hand-rolled WebSocket.valid-utf8? is gone; both sites call UTF8.valid? from utf8.carp@0.1.0.
• The 9 direct-validator asserts were removed. That's the right call and loses no coverage: they tested the pure validator, which now lives in utf8.carp with its own suite (I re-ran it standalone above). test/websocket.carp is now byte-identical to main.
• CHANGELOG placement is correct (this was the rebase's trap): the entry sits under a fresh ## Unreleased → ### Fixed above the shipped ## 0.7.0 (2026-07-06) section — not folded into it. The branch is rebased directly on the 0.7.0 release commit (merge-base(HEAD, origin/main) == origin/main tip), so there's no stale-base drift.

Findings

No bugs. One non-blocking observation: the frame→1007 integration path (an invalid-UTF-8 text frame actually triggering the close) has no unit test in web's suite — but that's a pre-existing altitude limitation (handle-ws-readable has no connection-dispatch harness; the suite tests at the decode-frame level), not a regression this PR introduces. The removed asserts never covered it either. Verified by reasoning + CI instead.

Out-of-scope, already noted in prior rounds and re-confirmed here: the macOS-only CI hides the TM/tm_zone Linux/glibc -Werror break, so the library likely doesn't build on Linux right now. Unrelated to this change.

Verdict: merge

Correctly wires the WebSocket text-frame UTF-8 validation to utf8.carp@0.1.0 as requested, with the right 1007 close behavior at both text sites, binary left untouched, changelog correctly placed, and the dependency independently verified. Type-checks and emits C clean; macOS CI green.