apk: keep dir mode bits and ownership in the streaming install path - #2458
Conversation
`installAPKFiles` is the install path taken whenever the target filesystem is not an `apk.WriteHeaderer` -- in tree that is `apko build-cpio`, out of tree melange's qemu runner. It had the two defects chainguard-dev#2455 fixes for `pkg/tarfs`: - `MkdirAll` was passed `header.FileInfo().Mode().Perm()`, which masks to 0o777, so setuid/setgid/sticky on a directory header were dropped. - Nothing applied `header.Uid`/`header.Gid`, so everything installed came out 0:0. A setgid binary such as wolfi postfix's `/usr/bin/postdrop` (0o2755, gid 101) kept its setgid bit but ended up setgid to *root* rather than to the group the package asked for. So `Chmod` newly created directories with every non-type bit, and `Chown` both directories and files. The directory metadata is applied only when we created the directory: `InitDB` makes /tmp 1777 before any package installs, and packages ship a tmp header without the sticky bit. Regular files get a `Chmod` too, because `dirFS.OpenFile` strips the special bits for the on-disk write and Linux clears them again on the next write by an unprivileged process. Symlinks and hardlinks are left alone: `FullFS` has no `Lchown`, and `Chown` would follow the link and retarget its target's ownership. Also fix `InitDB`'s base-directory permission check, which compared `stat.Mode().Perm()` against entries whose perms carry `fs.ModeSticky`. The /tmp entry could never match, so any call where /tmp already existed as 1777 failed with "base directory /tmp has incorrect permissions: 777". Compare every non-type bit, and report both got and want. Fixes chainguard-dev#2456 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`DirFS` mirrors every entry of the directory it wraps into its in-memory overrides, and those overrides are what all mode lookups resolve against -- `dirFS.Stat` takes `Mode()` from them. `seedOverride` passed only `mode.Perm()`, so a sticky directory or a setgid binary already on disk was reported without those bits from then on, and written into the layer that way. Carry every non-type bit for directories and regular files. The char-device branch keeps `Perm()`: its mode goes to `Mknod` in the unix encoding, where the special bits are numbered differently, and they are meaningless on a device node anyway. This is also what makes the /tmp half of chainguard-dev#2456 work end to end. `InitDB`'s base-directory check now compares the sticky bit, but for a `DirFS` over a tree that already had /tmp as 1777 the check was still seeing 0777. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review caught that the previous commit is only half a change. Seeding setuid/setgid out of the backing tree while leaving the node at the default uid 0 turns a setuid file owned by an unprivileged user into a setuid-*root* file in the serialized image -- worse than the dropped bit it replaced. `apk.New`'s fallback filesystem is `DirFS(ctx, "/")`, so the tree being walked can be a whole host root. Take the owner from the same `Sys()` the mode came from and `Chown` the seeded node with it. Symlinks stay unowned: `memFS.getNode` resolves the final path component, so a `Chown` there would retarget the link's target. When `Sys()` carries no owner -- a non-unix platform, or a `FileInfo` a memFS synthesized -- drop setuid/setgid rather than seed them against uid 0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
mattmoor
left a comment
There was a problem hiding this comment.
Verified this end to end (worktree at fb6840a): all five new tests pass, baseDirectories really does carry ModeSticky on /tmp so the old Perm() comparison could never match, the pre-existing-directory gate correctly preserves InitDB's 1777 /tmp (mode and owner), the seedOverride owner/mode pairing closes the setuid-root serialization hazard in the fail-closed direction, and the not-in-scope note about pkg/cpio/layer.go is accurate.
One real bug in the regular-file path, inline below, with a reproduction from a Linux guest. It needs a two-line swap plus a test that can actually catch it — the existing on-disk test structurally cannot, and that's worth fixing while we're here. Holding approval for those.
| // setgid binary means setgid to root rather than to the group | ||
| // the package asked for. | ||
| if err := a.fs.Chown(header.Name, header.Uid, header.Gid); err != nil { | ||
| return nil, fmt.Errorf("error setting owner on %s: %w", header.Name, err) |
There was a problem hiding this comment.
Chmod then Chown in this order clears the just-restored setgid/setuid on disk.
On Linux, chown(2) on an executable regular file clears S_ISUID/S_ISGID — including when the caller is root (since 2.2.13 root is treated like any other user for this). Directories are exempt. Reproduced in a Linux guest (cg microvm -p busybox):
touch /root/f; chmod 2755 /root/f → before: 2755
chown 0:101 /root/f → after-chown-by-root: 755
mkdir /root/d; chmod 2755 /root/d
chown 0:101 /root/d → dir-after-chown: 2755
dirFS.Chmod/Chown write through to the real disk (rwosfs.go:627-643), so on a disk-backed dirFS running as root the sequence is: OpenFile (0755 on disk) → Chmod (02755) → Chown(0,101) → kernel clears → 0755 on disk. The failure is precisely invisible:
- the memFS override keeps 02755, so apko image serialization stays correct;
- the on-disk tree — what melange's qemu runner boots, the consumer this PR names as its motivation — silently loses the bit;
- it also hits the most common case, setuid-root binaries (
/bin/su4755, uid 0: Chmod(4755) → Chown(0,0) → 0755); - unprivileged CI cannot see it:
isUnsupportedByFStolerates EPERM (rwosfs.go:620-625), so the disk chown is skipped and the bits survive.TestInstallAPKFilesModesOnDiskpasses unprivileged and would fail as root.
Fix: swap to Chown-before-Chmod here. The TypeDir path above is exempt per the repro, but the same order there costs nothing and removes the trap for the next reader.
Please also add a test that pins the ordering. No unprivileged test can observe the kernel behavior, so pin the contract instead: a small recording FullFS double that fails if Chmod on a path arrives before Chown on that path (order-recording only — no behavior emulation, per the usual test-double caveats), plus a comment on the call site stating the constraint. Without one, the next refactor reintroduces this silently, and CI will stay green while it does.
There was a problem hiding this comment.
Confirmed and fixed in 078c36a — thanks, this was a real one, and your
account of why CI stayed green is exactly right.
I reproduced the kernel behavior independently before touching anything, and
it is even less forgiving than the repro shows: a no-op chown clears the
bits too, unprivileged. chown $(id -u):$(id -g) on my own file took 2755 →
755 and 4755 → 755, while a 2755 directory kept its bit. So the trigger is
not "chown to a different owner", it is "chown at all".
Both branches now do Chown then Chmod, with the constraint written at the call
site.
On the test: there turns out to be a way to observe the kernel behavior
unprivileged, so I did both.
TestInstallAPKFilesModesOnDiskgains an entry whose header uid/gid are the
test process's own. That is the one case where the disk chown succeeds
without privileges, so it reaches the clearing instead of anEPERM
isUnsupportedByFSswallows. It fails against the pre-fix commit and passes
after — no root needed. The existing gid-101 entry keeps passing under the
old order, which is precisely the blind spot you identified.TestInstallAPKFilesMetadataOrderpins the order itself, as you asked, with
a recordingFullFSthat embeds a real memFS and overrides onlyChmodand
Chownto log which reached each path first. Recording only — every call
still runs against the wrapped filesystem, so there is no emulated behavior
to drift. It asserts["chown", "chmod"]for both the setgid file and the
sticky directory, so it covers the directory path where no kernel behavior
is observable at all. It fails against the pre-fix commit for both paths.
Keeping both: the first proves the real thing on the path that matters, the
second is what actually stops a future reorder, including on the directory
branch.
| return nil, fmt.Errorf("error setting mode on directory %s: %w", header.Name, err) | ||
| } | ||
| if err := a.fs.Chown(header.Name, header.Uid, header.Gid); err != nil { | ||
| return nil, fmt.Errorf("error setting owner on directory %s: %w", header.Name, err) |
There was a problem hiding this comment.
Same order here for symmetry, ideally — directories are exempt from the chown bit-clearing (repro in the main comment), so this one is not a live bug, but having the two paths disagree on ordering invites the next reader to "clean up" the one that matters.
There was a problem hiding this comment.
Done — the directory branch is Chown-then-Chmod in 078c36a too, with a
comment saying it is for symmetry and pointing at the file case for the actual
reason. TestInstallAPKFilesMetadataOrder asserts the order on the sticky
directory as well as the setgid file, so the two paths cannot drift apart
quietly.
Review caught a real bug in the ordering. `chown(2)` on a regular file clears setuid/setgid -- for root as well, and even for a chown that changes nothing. Verified locally: a 2755 file goes to 755 and a 4755 file to 755 across a no-op self-chown, while a 2755 directory keeps its bit. `dirFS.Chmod`/`Chown` write through to the real disk, so the old `Chmod` then `Chown` sequence left an installed setgid binary at 0755 on disk. The in-memory overrides still reported 02755, which is what made the loss silent: the serialized image was right while the tree melange's qemu runner boots was wrong. Unprivileged CI could not see it either, because `isUnsupportedByFS` swallows the `EPERM` from a disk chown to a uid the process does not own, so the chown never happened and the bits survived. Swap both branches to Chown-then-Chmod. Directories are exempt from the clearing, but leaving the two paths disagreeing on the order would invite a cleanup that reintroduces this. Two tests, because they fail for different reasons: - `TestInstallAPKFilesModesOnDisk` gains an entry whose header uid/gid are the test process's own. That is the one case where the disk chown succeeds unprivileged, so it reaches the kernel's clearing rather than an EPERM, and it fails against the old order. The existing gid-101 entry does not, which is exactly why CI stayed green. - `TestInstallAPKFilesMetadataOrder` pins the call order itself with a recording `FullFS` that wraps a real memFS and only logs which of `Chmod`/`Chown` reaches each path first. It covers the directory path too, where no kernel behavior can be observed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
mattmoor
left a comment
There was a problem hiding this comment.
Approving at 078c36a. Verified the respin rather than trusting it:
- Both branches are Chown-then-Chmod, with the constraint documented at the file call site and the symmetry rationale on the directory branch.
- All tests pass at this head, and I mutation-tested the fix: swapping the file path back to Chmod-then-Chown fails both new tests —
selfownedreaches the real kernel clearing (your no-op-chown observation makes it work unprivileged, and it also holds on macOS, where chown by a non-root owner clears the bits too), andTestInstallAPKFilesMetadataOrdercatches it structurally on both paths. - The recording double delegates every call to a real memFS and records only — nothing emulated to drift out of sync.
The no-op-chown discovery is the part I'd have missed: it turns "no unprivileged test can observe this" into a same-uid header entry, which is a strictly better guard than the order pin alone.
Fixes #2456 — both findings, in
pkg/apk/apk, plus the one thing inpkg/apk/fsthat kept the second fix from working end to end.1. The streaming install path drops dir mode bits and all ownership
installAPKFilesis the install path taken whenever the target filesystem isnot an
apk.WriteHeaderer— in treeapko build-cpio, out of tree melange'sqemu runner. Two defects, the same ones #2455 fixes for
pkg/tarfs:MkdirAllwas passedheader.FileInfo().Mode().Perm(), which masks to0o777, so setuid/setgid/sticky on a directory header were dropped.
header.Uid/header.Gid, so everything installed came out0:0. A setgid binary such as wolfi postfix's/usr/bin/postdrop(
0o2755, gid 101) kept its setgid bit but ended up setgid to rootinstead of to the group the package asked for.
So
Chmodnewly created directories with every non-type bit, andChownbothdirectories and files. The directory metadata is applied only when we created
the directory —
InitDBmakes/tmp1777 before any package installs, andpackages ship a
tmpheader without the sticky bit. Regular files get aChmodtoo, becausedirFS.OpenFilestrips the special bits for the on-diskwrite and Linux clears them again on the next write by an unprivileged
process.
Symlinks and hardlinks are left alone:
FullFShas noLchown, andChownwould follow the link and retarget its target's ownership.
The order is load-bearing (caught in review by @mattmoor, commit 4):
chown(2)on a regular file clears setuid/setgid — for root too, and even fora chown that changes nothing — so
Chownhas to come before theChmodthat restores those bits. The old order left an installed setgid binary at
0755 on disk while the in-memory overrides still said 02755, so the
serialized image was right and the tree melange's qemu runner boots was
wrong.
2.
InitDB's base-directory permission check could never match for /tmpIt compared
stat.Mode().Perm()against entries whose perms carryfs.ModeSticky, so the/tmpentry never matched and any call where/tmpalready existed as 1777 failed with
base directory /tmp has incorrect permissions: 777. Now it compares every non-type bit and reports both got andwant. I kept the check as strict as it was, just correct.
3.
seedOverridedropped the same bits (commits 2 and 3)DirFSmirrors the tree it wraps into its in-memory overrides, and thoseoverrides are what all mode lookups resolve against.
seedOverrideseededonly
mode.Perm(), so a sticky directory or setgid binary already on diskwas reported — and written into the layer — without those bits. That is also
what kept fix 2 from working for a
DirFSover a tree that already had/tmpas 1777.
Seeding those bits only works if the owner comes with them, which review
caught (thanks @depthfirst-app): a node seeded setuid but left at the default
uid 0 would serialize as setuid root, and
apk.New's fallback filesystem isDirFS(ctx, "/"), so the tree walked can be a whole host root. So the owneris taken from the same
Sys()as the mode and applied withChown; symlinksstay unowned (
memFS.getNoderesolves the final component, so aChownwouldretarget the link's target, and
FullFShas noLchown); and whenSys()carries no owner, setuid/setgid are dropped rather than paired with uid 0.
Testing
Six new tests, each mutation-checked against the pre-fix code:
TestInstallAPKFilesModesAndOwnership— sticky and setgid dirs, a setgidfile, gid 101, and a pre-existing 1777
/tmpwhose mode and owner mustsurvive a package's
tmpheader.TestInstallAPKFilesModesOnDisk— the same overapkfs.DirFS, assertingthe bits reach the real disk. One entry's header uid/gid are the test
process's own, which is the one case where the disk
Chownsucceedsunprivileged and so reaches the kernel's clearing of setuid/setgid instead
of an
EPERMthe filesystem swallows; that entry fails against the wrongcall order, with no root needed.
TestInstallAPKFilesMetadataOrder— pinsChownbeforeChmodwith arecording
FullFSthat wraps a real memFS and only logs which call reacheseach path first, covering the directory path where no kernel behavior is
observable.
TestInitDBBaseDirectoryPerms— sticky/tmpaccepted, non-stickyrejected.
TestSeedOverride_SpecialModeBits— setuid/setgid/sticky on disk surviveseeding, with the uid/gid asserted against the on-disk
*syscall.Stat_t.TestSeedOverride_NoOwnerDropsSetidBits— with no owner available, setuidand setgid are dropped instead of seeded against uid 0.
go test ./pkg/... ./internal/...passes andgolangci-lint run ./pkg/...isclean. No golden digests moved: the paths this touches are not the ones the
internal/clifixtures build through.Not in scope
pkg/cpio/layer.gonever setsRecord.UID/GID, soapko build-cpiooutputstill lands
0:0no matter what the filesystem records. The metadata is nowcorrect up to the cpio conversion; that last step wants its own change.