Finish port of WindowsDirectory and NativeUnixDirectory, #1342

[paulirwin]

• You've read the Contributor Guide and Code of Conduct.
• You've included unit or integration tests for your change, where applicable.
• You've included inline docs for your change, where applicable.
• There's an open issue for the PR that you are making. If you'd like to propose a change, please open an issue to discuss the change or find an existing issue.

Finish port of WindowsDirectory and NativeUnixDirectory

Fixes #1342

Description

This PR finishes the work that was started on WindowsDirectory and NativeUnixDirectory.

Upstream, this code is a mix of Java and C++. Instead of adding in true native code, I finished the approach that was started of using P/Invoke to call the Win32/POSIX native functions.

This PR also adds tests for these types, which had no tests upstream. These tests all pass on Windows 11 x64, Ubuntu 24.04 x64 in WSL, Ubuntu 24.04 arm64 in Docker, and macOS 26 arm64.

Benchmark Results

These benchmarks are on macOS 26 arm64:

Dir (64 MB)	Write	ReadSequential	ReadRandom
NativeUnix	19.1 ms	47.8 ms	380.6 ms
SimpleFS	149 ms	3.5 ms	1.0 ms
NIOFS	137 ms	5.4 ms	1.1 ms
MMap	139 ms	618 ms	5.6 ms
RAM	148 ms	1.8 ms	0.06 ms

Notes:

• NativeUnix is an order of magnitude faster for writes, but is much slower for reads, especially random reads. This is due to deliberately bypassing the OS-level IO cache with O_DIRECT. It is worth reviewing to determine if this read performance should be improved.
• The MMap slow read performance is likely related to Search performance issue with MMapDirectory under load #1151

I have not yet run benchmarks on WindowsDirectory.

AI: This PR was mostly written using Claude Code, Opus 4.8.

[NightOwl888]

Thanks.

Based on the benchmarks, I am starting to wonder whether we should aim to use native calls (or at least RandomAccess) instead of FileStream for writes on all of our FSDirectory implementations, since upstream they simply used FileChannel (which is a thin layer over native calls). Although, whatever solution we come up with, we should aim for getting the durability patched (#894) above all else.

Shouldn't we add these native calls to our existing native layer so we can utilize them to make a backport of RandomAccess for .NET Framework and other things that we are better off using native code for?

I didn't do a full review of this, but I found a few things we should address to make this implementation more complete.

[paulirwin]

Benchmarks for WindowsDirectory, before making PR feedback modifications above, are in:

64 MB file (mean, ms):

Dir	ReadSequential	ReadRandom	ConcurrentCloneReads
Windows	11.84	7.84	103.4
SimpleFS	11.72	8.52	79.0
NIOFS	20.83	7.55	133.3
MMap	415.59	5.97	937.1
RAM	6.49	0.12	4.7

Notes:

• WindowsDirectory does not override CreateOutput with a custom implementation, so it was not benchmarked
• Windows is faster than the average times of SimpleFS and NIOFS for non-concurrent reads.
• Windows is within noise of SimpleFS for sequential reads, and faster for random reads. It is slightly slower than NIOFS for random reads.
• Concurrent reads track well between SimpleFS and NIOFS.
• Again, concurrent reads in MMap is likely Search performance issue with MMapDirectory under load #1151

[paulirwin]

I meant to reply earlier to your top-level review comment... replies below.

Based on the benchmarks, I am starting to wonder whether we should aim to use native calls (or at least RandomAccess) instead of FileStream for writes on all of our FSDirectory implementations, since upstream they simply used FileChannel (which is a thin layer over native calls). Although, whatever solution we come up with, we should aim for getting the durability patched (#894) above all else.

I like this, but it will not change the public API, so it can be broken out as a separate issue (even for post-beta-19) as a perf optimization opportunity.

Shouldn't we add these native calls to our existing native layer so we can utilize them to make a backport of RandomAccess for .NET Framework and other things that we are better off using native code for?

Same as above, I think this can be broken out separately as a possible post-beta-19 perf optimization opportunity. In this particular case, these calls are very OS-dependent just like they are upstream (hence the names of the directories), so there's no harm in them being alongside the OS-specific directory code for now. That would be an easy refactor later as part of this possible perf improvement as a separate issue/PR.