286pm ext buffers#2725
Conversation
The external buffer pool addresses each L2 buffer as `b_L2seg = seg + n*64` (paragraphs) -- real-mode segment arithmetic. In 286 protected mode `seg` is a GDT selector, so `selector + n*64` produces a garbage selector, and the kernel #GPs the instant it touches any buffer past the first (during root mount). So CONFIG_FS_EXTERNAL_BUFFER could not be enabled in PM: it was stuck at the L1-only 8K cache. Give each L2 buffer its own GDT descriptor based at the chunk's physical address + n*BLOCK_SIZE, reachable at selector:0 (matching the existing "no offset to buffer" design). The copy path (fmemcpyw loading that selector) then works unchanged. GDT budget is fine: 64 buffer descriptors of 502 dynamic slots. Validated on QEMU/KVM: PM boots with CONFIG_FS_EXTERNAL_BUFFER=y, mounts root through the ext buffers (no #GP), meminfo shows the 64K BUF pool as a selector, and write/read/cached-read match real-mode ext buffers with no measurable overhead. First step toward a PM buffer cache above 1MB -- same descriptor mechanism, just base the buffers > 1MB. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Follow-on to the descriptor-based ext buffer support: instead of allocating the L2 pool from the conventional arena (where it competes with processes for the 639K), source it from extended memory above 1MB. On a 286 that region is reached naturally through GDT descriptors, so no unreal mode / LOADALL / INT15. - setup.S / pm286.c: enable the A20 gate on the PM path (fast A20 via port 0x92 in gdt_init; setup.S's enable_a20_gate only fires for hma=kernel). Without A20 a >1MB access wraps into low memory. - pm286.c / seg286.h: himem_alloc() -- a bump allocator over the extended memory reported by SETUP_XMS_KBYTES (INT 15h AH=88h at boot), handing out 1K-granular physical chunks above 1MB. - buffer.c: in PM the ext alloc loop takes its chunk from himem_alloc() and add_buffers() gives each buffer its own descriptor based at that >1MB physical. Widen add_buffers()'s 'seg' parameter from ramdesc_t to addr_t: with CONFIG_FS_XMS off ramdesc_t is 16-bit (seg_t), which truncated the 32-bit himem physical to 0 (descriptors ended up based at 0, corrupting low memory). Validated on QEMU/KVM: boots, mounts root and does file I/O through the >1MB buffers, at the same speed as conventional/real-mode ext buffers. Frees ~64K (the pool size) of the conventional pool, verified same-setup (both telnet+net): conventional ext 212K free vs himem 275K free. Conventional footprint is now just the 8K L1 staging area + buffer heads; the 64K payload lives above 1MB. Not yet surfaced in meminfo (himem is a separate pool, like XMS -- neither an arena segment nor a heap block); a separate "extended pool" report line would close that. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Hello @duzenko, Let's keep this open for a while, while the initial PM port is better integrated, and more time can be spent reflecting on proper kernel design for it and additional enhancements like this for PM. This PR contains all sorts of problems: ifdefs, duplicated allocation routines, non-portable A20 code, etc. I potentially like the idea of using AI for research ideas, but in general think building a kernel using AI isn't a great idea, unless you're just doing it for yourself. The tool excels at reading entire source trees so it can quickly produce a to-the-point solution that can work, but as you can see, almost always just produces #ifdefs shoe-horning its tightly-defined solution into the source tree. Very kludgy and a maintenance nightmare. Proper kernel design will take a lot more than that. For instance, this PR contains completely non-portable inb/outb instructions to enable the A20 gate right in the middle of a buffer routine (very bad design), despite the fact that there's already very specialized and highly tested A20 code already available elsewhere in the system (that analyzation path was likely eliminated early on until the tool later realized it required A20 and didn't perform another top-down analysis, etc). I imagine the day is here where non-programmers can create large enhancements quite quickly, but from the code I'm seeing, it appears to be just more and more code piled onto a system. For already-large 32- and 64-bit systems with large flat address spaces and seemingly infinite space for kernel size and applications, this isn't a big problem, but has the potential to turn into spaghetti code, almost requiring an AI to maintain it. We will see how that works out in the years to come, for other systems. For ELKS, the problem is in reverse, where resources are extremely limited, and lots of thought as well as a good philosophy has to be present in order to have the system work well, but remaining within its (16-bit) limitations. Do a lot with very little, definitely old-skool. IMO, this philosophy should not change just because the system gets protected mode operation and suddenly has tons more space to operate in. For non-mission-critical, or ordinary applications written with AI, I think the problem is much less severe, as the application serves as the entire container for any problems described above. This in many ways is similar to all applications in the wild: the range of design goodness covers the entire span from ugly to beautiful, well-designed to terrible, but applications are quite compartmentalized and relatively easy to replace (at least for command line apps). In summary, this PR gives some potentially interesting ideas for how to extend the kernel buffer system to XMS completely contained within ifdefs. An actual solution will require a rethink of the buffer system such that no ifdefs are required and appropriate wrapper functions are linked, depending on the configured options and architecture. Thank you! |
|
Hello @duzenko, Thank you for posting this.
As you could probably tell from my post above, it's easy to get overwhelmed trying to keep up with AI output, especially in PR form :) While AI's actual code may sometimes not be very fitting for its intended purpose, I'm finding that they can still be thought provoking. In this case, I've realized we should be able to fairly easily be able to use most all of the existing ELKS XMS code in our new PM mode for buffers and other XMS access, like RAM disks. Currently, ELKS XMS uses 80386 'unreal' mode or the 80286 LOADALL instruction to gain CPU access to XMS memory. For a PM kernel a GDT and protected mode is setup and enabled directly after boot, but the XMS initialization could still be configured and executed, and then use the standard XMS execution path for XMS buffer access. The magic then happens in an enhancement to For the PM kernel, a replacement Protected mode 80286 segments are limited to 64K bytes, while PM on 386+ allows for 4G. While using a single selector for all of memory could be appealing, that's potentially less secure, can't work the same way on 80286, and wouldn't use the same path as the existing XMS code, where the XMS_INT15 path already updates a (different) GDT. The above method is probably better than this AI-PR's solution, which allocates as many selectors as there are EXT or XMS buffers. You may be aware that ELKS can be configured to use 2500 XMS buffers, so that'd be a huge drain on the I'll start work on integrating ELKS XMS to PM. Perhaps 0.9.2 should be released beforehand. I enjoy writing kernel enhancements by hand, but appreciate your posting AI "ideas" (even as code or PRs) to start the process of old-fashioned human thought on a subject. Thank you! |
|
Completed in #2729. |
2 participants
This closes one of the protected mode gaps - EXT buffer.
Two commits: one fixes it in the conventional memory and the other moves it to high memory.
Can be split in two PRs or used for research of alternative solution.
No hurry to review.