Skip to content

[MessageHandling] Decode valid JSON surrogate pairs - #3464

Merged
rintaro merged 1 commit into
swiftlang:mainfrom
Hokila:fix-json-surrogates
Oct 8, 2026
Merged

rintaro merged 1 commit into
swiftlang:mainfrom
Hokila:fix-json-surrogates

Conversation

@Hokila

@Hokila Hokila commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

This PR adds support to decode valid JSON surrogate pairs in string escapes. The decoder now checks for a trail surrogate after a lead surrogate. The decoder still rejects unpaired surrogates. I verified this by running the local tests with swift test --filter JSONTests.

Example

The decoder rejected some characters, such as emojis, when JSON used surrogate pairs.

// Rocket emoji (🚀) encoded as a surrogate pair
let jsonString = #""\uD83D\uDE80""#

// Before: The decoder throws an error. It treats \uD83D as an invalid lone surrogate.
// After: The decoder successfully returns the "🚀" character.
let decoded = try jsonString.withUTF8 {
    try JSON.decode(String.self, from: $0)
}

@rintaro rintaro left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!
Although this has been unreachable as long as both the compiler and the plugin are using this JSON encoder/decoder, this change is good for spec compliance!

Just a few nit-picks, and please fix the format as https://github.com/swiftlang/swift-syntax/blob/main/CONTRIBUTING.md#formatting

let decoded = try json.withUTF8 { try JSON.decode(String.self, from: $0) }
XCTAssertEqual(decoded, "𐐷")
} catch {
XCTFail("Decoding valid surrogate pair failed")

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this should be wrapped with do {} catch {}. the test method can be throws

// Valid '𐐷' (U+10437) character
assertRoundTrip(
of: "𐐷",
expectedJSON: #""𐐷""#

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since JSON.encode() always emits this as a bare UTF8 sequence, this test would not hit the escaped UTF16 sequence code path. Since equivalent path is covered by testComplexStruct (🛑 emoji), I don't think this test is needed.

@Hokila
Hokila force-pushed the fix-json-surrogates branch from f0f81f9 to d9a8db7 Compare October 7, 2026 05:52
@Hokila
Hokila force-pushed the fix-json-surrogates branch from d9a8db7 to 68f8cb1 Compare October 7, 2026 05:53
@Hokila

Hokila commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor Author

@rintaro

Thank you for review. I am new to SwiftSyntax. I looked for contribution opportunities by checking FIXME comments in the source code and reviewing recent issues. This PR addresses the surrogate-pair decoding gap I found.

Did the following change:

  • Ran swift-format on the changed files. I will include formatting in my checks for future PRs.
  • Changed testStringSurrogatePairDecoding to throw errors directly and removed the do/catch block.
  • Removed the redundant round-trip check for raw UTF-8. testComplexStruct already covers that path. The test still checks the escaped surrogate pair directly.

I amended the commit and rebased the branch onto latest main, also ran the focused test, and it passed.

@Hokila

Hokila commented Oct 7, 2026

Copy link
Copy Markdown
Contributor Author

@rintaro I noticed that the existing tests use XCTest. Is there a plan to migrate swift-syntax to Swift Testing, or should new tests continue to use XCTest?

@rintaro

rintaro commented Oct 7, 2026

Copy link
Copy Markdown
Member

@swift-ci Please test

@rintaro

rintaro commented Oct 7, 2026

Copy link
Copy Markdown
Member

I noticed that the existing tests use XCTest. Is there a plan to migrate swift-syntax to Swift Testing, or should new tests continue to use XCTest?

This repository still supports Swift 5.9 toolchain, which I don't think includes swift-testing, not 100% sure.

// swift-tools-version: 5.9

@rintaro rintaro left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you Hokila!

@rintaro

rintaro commented Oct 8, 2026

Copy link
Copy Markdown
Member

@swift-ci Please test

@rintaro
rintaro merged commit 4a71c31 into swiftlang:main Oct 8, 2026
42 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants