LUA16 cannot handle tables > 64K

[PeterFidelman]

Problem:

LUA16.EXE is compiled with -wl which specifies the large memory model, meaning pointer offsets will wrap around at 64KB. Trying to grow data structures, such as tables, to larger than 64KB will cause memory corruption.

Reproduction in LUA16.EXE,

arr = {}
for i = 1, 10000 do arr[i] = i end

This hangs LUA16.EXE, but works just fine in LUA4G.EXE and presumably in the other 32-bit builds as well.

If this is intended behavior it should be documented. Ideally, LUA16.EXE would be built for the huge memory model so data structures wouldn't have a 64KB size limit. I've tried doing this myself but ran into segment size issues; I'll post an update here if I get it working. Reporting it here so maybe someone else can get it working before I do.

[Lethja]

Undesirable behaviour indeed. I'll see if I can get huge models working.

[Lethja]

I've tried doing this myself but ran into segment size issues

Like you said this is the biggest blocker to using the huge memory model instead of large.

Error! E2021: size of segment _TEXT exceeds 64k by 12932 bytes
Error! E2020: size of group AUTO exceeds 64k by 12932 bytes

Here's how the linker mapped out the segments when I built with large and huge:

lua large map.txt

lua huge map.txt

In my experience Watcom isn't particularly efficient with certain system calls in DOS. For example: simply commenting out the os_execute() function in loslib.c allowed wlink to strip out Watcoms heavier than necessary implementation of exec() and give us back almost half the exceeding space.

Error! E2021: size of segment _TEXT exceeds 64k by 6896 bytes
Error! E2020: size of group AUTO exceeds 64k by 6896 bytes

[PeterFidelman]

According to https://web.archive.org/web/20100413134426/http://www.openwatcom.org/index.php/Over-large_Code_Segments#_TEXT_and_the_C_Runtime_Library it seems the fundamental issue is that Watcom tries to place most of of the C runtime library in one segment (_TEXT) which causes problems when a single program needs >64K worth of functions from the C runtime.

[PeterFidelman]

I managed to get 5.4.8 building with the huge memory model -mh with the following hacks:

• add compiler option -fpi87 to remove floating point emulation (require fpu)
• in loslib.c replace os_execute() and os_tmpname with return 1 stubs (very similar to what you did)

This yields a 395,050 byte LUA16.EXE.

Unfortunately it still hangs when trying to build a large table, as before:

arr = {}
for i = 1, 10000 do arr[i] = i end

At this point I wonder if a 2-byte variable is being used to request (or track) an allocation size somewhere, which would wrap around for an allocation of >64KB. There are a couple of integer size warnings emitted during build. I've seen similar warnings indicate severe problems on m68k (TI-92+) which were solved by instructing the compiler to use 32 bits for the int type, which in TIGCC was the -mnoshort option. I wonder if Watcom has an option that forces int to be 32 bits even on a 16-bit 8086 target.

[Lethja]

There are a couple of integer size warnings emitted during build.

There are indeed a couple of nasty warnings originating from inside code macros when building LUA16/LUA21:

lua/lfunc.c(177): Warning! W124: Comparison result always 0
lua/lundump.c(291): Warning! W124: Comparison result always 0
lua/lparser.c(1650): Warning! W124: Comparison result always 0
lua/lcode.c(343): Warning! W124: Comparison result always 1
lua/lcode.c(462): Warning! W124: Comparison result always 1
lua/lobject.c(63): Warning! W124: Comparison result always 0

A lot of which dissappear in the LUA4G/LUANT/LUA22 32-bit builds:

lua/lundump.c(291): Warning! W124: Comparison result always 0
lua/lcode.c(343): Warning! W124: Comparison result always 1

It's also possible that lua.pat could contain some overlooked logic causing a bad value in the 16-bit builds.

I wonder if Watcom has an option that forces int to be 32 bits even on a 16-bit 8086 target.

Not that I know of. I'd be overjoyed to be proven wrong.

[PeterFidelman]

I believe the next issue causing >64KB allocations to be problematic in LUA16 is that the size_t type passed to the allocation functions in lmem.c is only 2 bytes, not 4 bytes, even when the huge memory model is used. It'll also be necessary to check whether Watcom's underlying memory allocation function used by Lua (probably realloc) is in terms of a 2-byte type or a larger one, and if larger, whether the allocator is indeed capable of returning a >64KB region if Lua manages to request one this size.

[PeterFidelman]

Hmm...

From https://open-watcom.github.io/open-watcom-v2-wikidocs/clib.html#malloc, malloc is:

void *malloc( size_t size );

And from https://open-watcom.github.io/open-watcom-v2-wikidocs/clr.html:

For the Open Watcom C/16 and C/32 compilers, the macro size_t is unsigned int.

And from https://open-watcom.github.io/open-watcom-v2-wikidocs/cguide.html#16Mbit__Data_Representation:

An item of type "int" occupies 2 bytes of storage. Its value is in the following range. -32768 <= n <= 32767

Note that "int" is signed and hence "int" and "signed int" are equivalent. If an item of type "int" is to be unsigned, it must be defined as "unsigned int". In this case its value is in the following range. 0 <= n <= 65535

If you are generating code that executes in 16-bit mode, "short int" and "int" are equivalent, "unsigned short int" and "unsigned int" are equivalent, and "signed short int" and "signed int" are equivalent.

So it may be difficult or impossible to extricate ourselves from a 64K limit per allocation, as would be necessary to yield a single contiguous block larger than 64K.

[Lethja]

Sounds like we need to write our own 2D array to get around this problem properly on 16-bit systems. In the meantime, maybe gcc-ia16 can be configured to make int 4 bytes wide?

[PeterFidelman]

I just found that in Watcom's 16-bit models, halloc / hfree might be a path forward for huge allocations. https://open-watcom.github.io/open-watcom-v2-wikidocs/clib.html#halloc

I haven't looked at what the situation would be like in gcc-ia16.

[Lethja]

halloc/hfree would be tricky but possible. Lua seems to use realloc rather that malloc and free and there's no hrealloc equivilent in Watcom itself so we would have to make our own drop in replacement. Failing that, we could always resort to INT 21.

[PeterFidelman]

It looks like gcc-ia16 might only support the tiny, small, and medium memory models? I couldn't find a cli argument for activating models with far data (compact, large, huge), which, if true, might make it hard to build Lua with that compiler.

[Lethja]

It looks like gcc-ia16 might only support the tiny, small, and medium memory models? I couldn't find a cli argument for activating models with far data (compact, large, huge), which, if true, might make it hard to build Lua with that compiler.

Looks like a halloc() rewrite of lua_malloc() and friends is the way forward then. If that function doesn't work then hand written assembly should since all the tools we need are in well documented DOS interrupts. Either way, some Lua functions in the source code will need to be rewritten for the 8086 to achieve that array without locking the system. I'll look into what that rewrite entails after the next release.

[PeterFidelman]

Note: In Lua 5.5 with the large memory model, a >64K table becomes an out-of-memory error instead of memory corruption and a crash. Lua 5.4 was missing a bounds-check, which was added in lua/lua@0c9bec0.

In Lua 5.5 the behavior is now,

> arr = {}
> for i = 1, 10000 do arr[i] = i end
stdin:1: table overflow

which is much less disruptive.

[Lethja]

In Lua 5.5 the behavior is now,

> arr = {}
> for i = 1, 10000 do arr[i] = i end
stdin:1: table overflow

which is much less disruptive.

This is good news. I actually ran into a table overflow error myself a few hours ago while testing the new INFLATE.LUA script on a retro machine (it works correctly on a modern machine but currently needs way too much memory to be usable on a Pentium or earlier machine). It's good to see the same halt message works on your test code to.

Even though there's no risk of a system lockup anymore I still want to investigate if halloc() can be used to push the upper limit of Lua tables available on 16-bit machines. I'll leave the issue open until we know one way or another if it's possible to break the 64k limit.