Intermittent failure causing `print -u2` breakage

[JohnoKing]

This is a dedicated issue for an intermittent test failure in subshell.sh (#344 (comment)).
I've managed to make it reproduce more consistently:

# Note: This problem exhibits with print -u2, but not 1>&2
function foo
{
	typeset -li i
	print -u2 foobar
	for	((i=0; i < 8000; i++))
	do	print abcdefghijk
	done
	print -u2 done
}
out=$(eval "foo | cat" 2>&1)
print "${#out}" >out

for ((i=0; i!=25; i++)); do ksh /tmp/repro; cat out; done
96006
96006
96011
96006
96006
96006
96011
96006
96006
96006
96006
96011
96006
96006
96006
96006
96011
96011
96006
96011
96006
96006
96006
96011
96006

This bug occurs in all 93u+m releases AFAICT, as well as ksh93u+, ksh2020, ksh93v- 2012-08-24 and 93v- 2012-10-04. It does not occur in 93v- releases 2012-10-12 through 2016-01-10. At some point this bug was fixed in 93v-, then unfixed in ksh2020. (Edit: That's likely because ksh93v- defaults cat calls to use the AST builtin, not the external cat.) I may trace this problem further, but my schedule is packed today so no guarantees.

[McDutchie]

Very strange. I cannot reproduce this on macOS at all. I'll test other systems.

[JohnoKing]

I found some time to trace this further and for some reason the following diff from 93v- 2012-10-12 changes the behavior (I have no clue why, probably undefined behavior somewhere):

diff --git a/src/lib/libcmd/Mamfile b/src/lib/libcmd/Mamfile
index c7dc2287..7e4f7f50 100644
--- a/src/lib/libcmd/Mamfile
+++ b/src/lib/libcmd/Mamfile
@@ -946,7 +946,7 @@ exec - set $( sed \
 exec - -e '/^b_[a-z_][a-z_0-9]*(/!d' \
 exec - -e 's/^b_//' \
 exec - -e 's/(.*//' \
-exec - |
+exec - cmdinit.c basename.c cat.c chgrp.c chmod.c chown.c cksum.c cmp.c comm.c cp.c cut.c dirname.c date.c expr.c fds.c fmt.c fold.c getconf.c head.c id.c join.c ln.c logname.c md5sum.c mkdir.c mkfifo.c mktemp.c mv.c paste.c pathchk.c pids.c rev.c rm.c rmdir.c stty.c sum.c sync.c tail.c tee.c tty.c uname.c uniq.c vmstate.c wc.c revlib.c wclib.c fts_fix.c lib.c grep.c xargs.c iconv.c |
 exec - sort -u
 exec - )
 exec - for cmd

# 93v- 2012-10-04 (root is this 93v- source code, not the dev branch)
$ patch -p1 < /tmp/innocuous.diff
$ touch src/cmd/ksh93/data/* src/lib/libcmd/*  # Forces old mamake to rebuild thoroughly enough
$ bin/package make CC=tcc
$ while true; do arch/*/bin/ksh /tmp/repro; cat out; done
96011
96011
96011
96011
96011
96011
96011
96011
96011
96011
---rest of output consistent and correct---

# Regular 93v- 2012-10-04
$ git checkout HEAD -- src/lib/libcmd/Mamfile  # Revert patch
$ touch src/cmd/ksh93/data/* src/lib/libcmd/*  # Forces old mamake to rebuild thoroughly enough
$ bin/package make CC=tcc
$ while true; do arch/*/bin/ksh /tmp/repro; cat out; done
96011
96011
96011
96006
96011
96011
96011
96011
96011
96006
96006
---etc---

(The OS I'm reproducing this on is CachyOS with the x86-64-v3 packages. Edit: I was using uutils cat as /bin/cat, which is relevant, see posts below.)

[JohnoKing]

Another anomaly: The bug occurs when running the script without a hashbang using the static build, but not the dynamic one via bin/package use (on the 93u+m dev branch, current commit):

$ arch/*/bin/ksh
$ while true; do /tmp/repro; cat out; done
96011
96006
96011
96006
96011
96011
96006
---snip---

$ bin/package use
«0»6:~/GitRepos/KornShell/ksh[dev] $ while true; do /tmp/repro; cat out; done
96011
96011
96011
96011
96011
96011
96011

[JohnoKing]

I found that the 93v- lead is probably a red herring. The bug is reproducing for me when the cat executed is uutils cat (so eval "foo | uu-cat"). It does not seem to occur in the original reproducer if the cat is BusyBox, GNU coreutils, or AST. This could be a uutils bug, or it could be a ksh bug. zsh and mksh print -u2 work fine with uutils cat when running the reproducer with uu-cat, plus there's another bug showcased below that I've reproduced with all four cat implementations, so I'm leaning towards ksh or SFIO being buggy.

Related problem:

function foo
{
	typeset -li i
	echo foobar >/dev/stderr
	for	((i=0; i < 8000; i++))
	do	echo abcdefghijk
	done
	echo foobar >/dev/stderr
}
out=$(eval "foo | cat" 2>&1)  # This behaves the same with BusyBox, GNU coreutils, uutils and AST cat
echo "${#out}" >out

Results:

$ while true; do ksh /tmp/repro; cat out; done
6
6
6
6
---snip---

$ while true; do zsh /tmp/repro; cat out; done 
96013
96013
96013

$ while true; do bash /tmp/repro; cat out; done 
96013
96013
96013