Start JDT LS with a UTF-8 locale when the environment selects ASCII - #4473
Start JDT LS with a UTF-8 locale when the environment selects ASCII#4473wenytang-ms wants to merge 1 commit into
Conversation
The JVM derives sun.jnu.encoding - the charset it uses to decode and encode file names on POSIX systems - from the process locale at startup, and silently ignores -Dsun.jnu.encoding, so neither java.jdt.ls.vmargs nor -Dfile.encoding can influence it. Containers routinely start with an unset or C/POSIX locale, which makes the JVM fall back to ASCII. Any workspace file whose name contains non-ASCII characters is then decoded into U+FFFD and can no longer be turned into a java.nio.file.Path, so importing the workspace fails with "InvalidPathException: Malformed input or input contains unmappable characters" and every project after the failing one is missing from the Java Projects view. This is why the reported project only reproduces inside a container and works fine locally and over SSH remote. Correct the locale of the JDT LS process when, and only when, the inherited one cannot represent non-ASCII file names. A deliberately configured non-UTF-8 locale such as zh_CN.GBK is left untouched, and only the LC_CTYPE category is set unless LC_ALL is the variable selecting ASCII, so message, number and time formatting keep following the environment. Fixes microsoft/vscode-java-dependency#1041
0fd6524 to
b599e75
Compare
VerificationVerified end-to-end on Linux against the reported sample: real JDT LS (
Before, reproducing the reported failure line for line: After, with the value this PR injects: At the JVM level, each row run directly against that file name (8 CJK chars = 24 UTF-8 bytes), on Temurin 21.0.12 / glibc 2.35:
Row 3 is why One note for reviewers: this is not a narrow edge case. JDT LS ships only the pure-Java |
|
@datho7561 please help review this PR. |
1 participant
Fixes microsoft/vscode-java-dependency#1041
Problem
Inside containers, importing a workspace that contains a file with a non-ASCII name aborts with:
Every project after the failing one silently disappears. The reporter saw a 7-module Maven project collapse to a single root project, which looks exactly like a multi-module import bug — the encoding cause is invisible from the UI. The sample is
yangzongzhuan/RuoYi, whosedoc/folder contains若依环境使用手册.docx: 8 CJK characters = 24 UTF-8 bytes, matching the 24 replacement characters in the log exactly (one per byte).It reproduces only in containers. The same workspace works locally and over SSH remote, because those sessions have a UTF-8 locale.
Root cause
On POSIX systems the JVM decodes and encodes file names with
sun.jnu.encoding, which it derives from the process locale at startup (OpenJDKjdk-21+35):unix/native/libjava/java_props_md.c#L430-L454—setlocale(LC_ALL, "")thenParseLocale(LC_CTYPE, ..., &sprops.encoding), andsprops.sun_jnu_encoding = sprops.encoding.UnixFileSystem_md.c#L347—readdirbytes are decoded withJNU_NewStringPlatform, which uses that charset (System.c#L121). Under ASCII every byte becomes U+FFFD.UnixPath.java#L124-L130— encoding the name back throwsInvalidPathException("Malformed input or input contains unmappable characters"), the exact message above.Containers commonly start with an unset or
C/POSIXlocale, sosun.jnu.encodingbecomesANSI_X3.4-1968.Crucially,
-Dsun.jnu.encodingand-Dfile.encodingcannot fix this — the property is written from the OS locale during VM startup and the command-line value is ignored. Verified on JDK 21:java -Dsun.jnu.encoding=US-ASCIIstill reports the locale-derived charset. Sojava.jdt.ls.vmargsgives users no way out; the process locale is the only lever, and this extension is the only component in a position to set it.Change
prepareExecutablenow mergesgetUnicodeLocaleEnv()into the JDT LS process environment. It corrects the locale when, and only when, the inherited one cannot represent non-ASCII file names:LC_ALL>LC_CTYPE>LANG, the precedence POSIX defines forLC_CTYPE.C/POSIX/C.ASCIIcases. A deliberately configured non-UTF-8 locale such aszh_CN.GBKis left untouched, since file names on such a system may genuinely be stored in that encoding.LC_ALL=C.UTF-8only whenLC_ALLis the variable selecting ASCII (it dominates every category); otherwise sets justLC_CTYPE=C.UTF-8, so message, number and time formatting keep following the environment.Why Windows returns early
Windows is structurally unaffected, and does not read these variables at all:
windows/native/libjava/java_props_md.c#L691+#L67-L68—sun.jnu.encodingcomes fromGetACP(). The file contains nosetlocale,LC_ALLorLC_CTYPEreference.WinNTFileSystem_md.c#L768passes the UTF-16WIN32_FIND_DATAW.cFileNamestraight toNewString, andWindowsNativeDispatcher.java#L1146-L1167copies the Java String into the native buffer verbatim beforeCreateFileW. (NTFS stores file names in Unicode.)Setting POSIX locale variables there would be a no-op for the JVM while still leaking into every process JDT LS spawns (Maven/Gradle wrappers, Git for Windows, MSYS-based tooling do read them).
macOS is also unaffected —
sun_jnu_encodingis hard-coded to UTF-8 there (java_props_md.c#L451-L452) — and the ASCII check makes this a no-op in practice anyway.Known limitation
C.UTF-8requires glibc >= 2.35 or a distro that ships it (Debian and Ubuntu do; RHEL 7 does not). Where it is missing,setlocalefails and the JVM falls back toC, i.e. current behaviour — no regression, but no fix either. Those users still needen_US.UTF-8generated in the image.Non-VS Code clients that launch JDT LS through
jdtls.pyare unaffected by this change; a matching fix can be made ineclipse.jdt.lsseparately.Tests
test/standard-mode-suite/javaServerStarter.test.tscovers the decision table: Windows early return, unset locale,C,POSIX,C.ASCII,en_US.UTF-8,zh_CN.GBK,LC_ALLoverride,LC_CTYPE-only correction, andLC_CTYPEprecedence overLANG.