Skip to content

apk: keep dir mode bits and ownership in the streaming install path - #2458

Merged
smoser merged 4 commits into
chainguard-dev:mainfrom
smoser:fix/issue-2456-stickydir-bits
Sep 4, 2026
Merged

apk: keep dir mode bits and ownership in the streaming install path#2458
smoser merged 4 commits into
chainguard-dev:mainfrom
smoser:fix/issue-2456-stickydir-bits

Conversation

@smoser

@smoser smoser commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Fixes #2456 — both findings, in pkg/apk/apk, plus the one thing in
pkg/apk/fs that kept the second fix from working end to end.

1. The streaming install path drops dir mode bits and all ownership

installAPKFiles is the install path taken whenever the target filesystem is
not an apk.WriteHeaderer — in tree apko build-cpio, out of tree melange's
qemu runner. Two defects, the same ones #2455 fixes for pkg/tarfs:

  • MkdirAll was passed header.FileInfo().Mode().Perm(), which masks to
    0o777, so setuid/setgid/sticky on a directory header were dropped.
  • Nothing applied header.Uid/header.Gid, so everything installed came out
    0:0. A setgid binary such as wolfi postfix's /usr/bin/postdrop
    (0o2755, gid 101) kept its setgid bit but ended up setgid to root
    instead of to the group the package asked for.

So Chmod newly created directories with every non-type bit, and Chown both
directories and files. The directory metadata is applied only when we created
the directory — InitDB makes /tmp 1777 before any package installs, and
packages ship a tmp header without the sticky bit. Regular files get a
Chmod too, because dirFS.OpenFile strips the special bits for the on-disk
write and Linux clears them again on the next write by an unprivileged
process.

Symlinks and hardlinks are left alone: FullFS has no Lchown, and Chown
would follow the link and retarget its target's ownership.

The order is load-bearing (caught in review by @mattmoor, commit 4):
chown(2) on a regular file clears setuid/setgid — for root too, and even for
a chown that changes nothing — so Chown has to come before the Chmod
that restores those bits. The old order left an installed setgid binary at
0755 on disk while the in-memory overrides still said 02755, so the
serialized image was right and the tree melange's qemu runner boots was
wrong.

2. InitDB's base-directory permission check could never match for /tmp

It compared stat.Mode().Perm() against entries whose perms carry
fs.ModeSticky, so the /tmp entry never matched and any call where /tmp
already existed as 1777 failed with base directory /tmp has incorrect permissions: 777. Now it compares every non-type bit and reports both got and
want. I kept the check as strict as it was, just correct.

3. seedOverride dropped the same bits (commits 2 and 3)

DirFS mirrors the tree it wraps into its in-memory overrides, and those
overrides are what all mode lookups resolve against. seedOverride seeded
only mode.Perm(), so a sticky directory or setgid binary already on disk
was reported — and written into the layer — without those bits. That is also
what kept fix 2 from working for a DirFS over a tree that already had /tmp
as 1777.

Seeding those bits only works if the owner comes with them, which review
caught (thanks @depthfirst-app): a node seeded setuid but left at the default
uid 0 would serialize as setuid root, and apk.New's fallback filesystem is
DirFS(ctx, "/"), so the tree walked can be a whole host root. So the owner
is taken from the same Sys() as the mode and applied with Chown; symlinks
stay unowned (memFS.getNode resolves the final component, so a Chown would
retarget the link's target, and FullFS has no Lchown); and when Sys()
carries no owner, setuid/setgid are dropped rather than paired with uid 0.

Testing

Six new tests, each mutation-checked against the pre-fix code:

  • TestInstallAPKFilesModesAndOwnership — sticky and setgid dirs, a setgid
    file, gid 101, and a pre-existing 1777 /tmp whose mode and owner must
    survive a package's tmp header.
  • TestInstallAPKFilesModesOnDisk — the same over apkfs.DirFS, asserting
    the bits reach the real disk. One entry's header uid/gid are the test
    process's own, which is the one case where the disk Chown succeeds
    unprivileged and so reaches the kernel's clearing of setuid/setgid instead
    of an EPERM the filesystem swallows; that entry fails against the wrong
    call order, with no root needed.
  • TestInstallAPKFilesMetadataOrder — pins Chown before Chmod with a
    recording FullFS that wraps a real memFS and only logs which call reaches
    each path first, covering the directory path where no kernel behavior is
    observable.
  • TestInitDBBaseDirectoryPerms — sticky /tmp accepted, non-sticky
    rejected.
  • TestSeedOverride_SpecialModeBits — setuid/setgid/sticky on disk survive
    seeding, with the uid/gid asserted against the on-disk *syscall.Stat_t.
  • TestSeedOverride_NoOwnerDropsSetidBits — with no owner available, setuid
    and setgid are dropped instead of seeded against uid 0.

go test ./pkg/... ./internal/... passes and golangci-lint run ./pkg/... is
clean. No golden digests moved: the paths this touches are not the ones the
internal/cli fixtures build through.

Not in scope

pkg/cpio/layer.go never sets Record.UID/GID, so apko build-cpio output
still lands 0:0 no matter what the filesystem records. The metadata is now
correct up to the cpio conversion; that last step wants its own change.

smoser and others added 2 commits September 3, 2026 11:04
`installAPKFiles` is the install path taken whenever the target
filesystem is not an `apk.WriteHeaderer` -- in tree that is
`apko build-cpio`, out of tree melange's qemu runner. It had the two
defects chainguard-dev#2455 fixes for `pkg/tarfs`:

- `MkdirAll` was passed `header.FileInfo().Mode().Perm()`, which masks
  to 0o777, so setuid/setgid/sticky on a directory header were
  dropped.
- Nothing applied `header.Uid`/`header.Gid`, so everything installed
  came out 0:0. A setgid binary such as wolfi postfix's
  `/usr/bin/postdrop` (0o2755, gid 101) kept its setgid bit but ended
  up setgid to *root* rather than to the group the package asked for.

So `Chmod` newly created directories with every non-type bit, and
`Chown` both directories and files. The directory metadata is applied
only when we created the directory: `InitDB` makes /tmp 1777 before
any package installs, and packages ship a tmp header without the
sticky bit. Regular files get a `Chmod` too, because `dirFS.OpenFile`
strips the special bits for the on-disk write and Linux clears them
again on the next write by an unprivileged process.

Symlinks and hardlinks are left alone: `FullFS` has no `Lchown`, and
`Chown` would follow the link and retarget its target's ownership.

Also fix `InitDB`'s base-directory permission check, which compared
`stat.Mode().Perm()` against entries whose perms carry
`fs.ModeSticky`. The /tmp entry could never match, so any call where
/tmp already existed as 1777 failed with "base directory /tmp has
incorrect permissions: 777". Compare every non-type bit, and report
both got and want.

Fixes chainguard-dev#2456

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`DirFS` mirrors every entry of the directory it wraps into its
in-memory overrides, and those overrides are what all mode lookups
resolve against -- `dirFS.Stat` takes `Mode()` from them.
`seedOverride` passed only `mode.Perm()`, so a sticky directory or a
setgid binary already on disk was reported without those bits from
then on, and written into the layer that way.

Carry every non-type bit for directories and regular files. The
char-device branch keeps `Perm()`: its mode goes to `Mknod` in the
unix encoding, where the special bits are numbered differently, and
they are meaningless on a device node anyway.

This is also what makes the /tmp half of chainguard-dev#2456 work end to end.
`InitDB`'s base-directory check now compares the sticky bit, but for
a `DirFS` over a tree that already had /tmp as 1777 the check was
still seeing 0777.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Comment thread pkg/apk/fs/rwosfs.go Outdated
Review caught that the previous commit is only half a change. Seeding
setuid/setgid out of the backing tree while leaving the node at the
default uid 0 turns a setuid file owned by an unprivileged user into a
setuid-*root* file in the serialized image -- worse than the dropped
bit it replaced. `apk.New`'s fallback filesystem is `DirFS(ctx, "/")`,
so the tree being walked can be a whole host root.

Take the owner from the same `Sys()` the mode came from and `Chown` the
seeded node with it. Symlinks stay unowned: `memFS.getNode` resolves
the final path component, so a `Chown` there would retarget the link's
target. When `Sys()` carries no owner -- a non-unix platform, or a
`FileInfo` a memFS synthesized -- drop setuid/setgid rather than seed
them against uid 0.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@smoser
smoser requested a review from mattmoor September 3, 2026 17:58

@mattmoor mattmoor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified this end to end (worktree at fb6840a): all five new tests pass, baseDirectories really does carry ModeSticky on /tmp so the old Perm() comparison could never match, the pre-existing-directory gate correctly preserves InitDB's 1777 /tmp (mode and owner), the seedOverride owner/mode pairing closes the setuid-root serialization hazard in the fail-closed direction, and the not-in-scope note about pkg/cpio/layer.go is accurate.

One real bug in the regular-file path, inline below, with a reproduction from a Linux guest. It needs a two-line swap plus a test that can actually catch it — the existing on-disk test structurally cannot, and that's worth fixing while we're here. Holding approval for those.

Comment thread pkg/apk/apk/install.go Outdated
// setgid binary means setgid to root rather than to the group
// the package asked for.
if err := a.fs.Chown(header.Name, header.Uid, header.Gid); err != nil {
return nil, fmt.Errorf("error setting owner on %s: %w", header.Name, err)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Chmod then Chown in this order clears the just-restored setgid/setuid on disk.

On Linux, chown(2) on an executable regular file clears S_ISUID/S_ISGID — including when the caller is root (since 2.2.13 root is treated like any other user for this). Directories are exempt. Reproduced in a Linux guest (cg microvm -p busybox):

touch /root/f; chmod 2755 /root/f     → before: 2755
chown 0:101 /root/f                   → after-chown-by-root: 755
mkdir /root/d; chmod 2755 /root/d
chown 0:101 /root/d                   → dir-after-chown: 2755

dirFS.Chmod/Chown write through to the real disk (rwosfs.go:627-643), so on a disk-backed dirFS running as root the sequence is: OpenFile (0755 on disk) → Chmod (02755) → Chown(0,101) → kernel clears → 0755 on disk. The failure is precisely invisible:

  • the memFS override keeps 02755, so apko image serialization stays correct;
  • the on-disk tree — what melange's qemu runner boots, the consumer this PR names as its motivation — silently loses the bit;
  • it also hits the most common case, setuid-root binaries (/bin/su 4755, uid 0: Chmod(4755) → Chown(0,0) → 0755);
  • unprivileged CI cannot see it: isUnsupportedByFS tolerates EPERM (rwosfs.go:620-625), so the disk chown is skipped and the bits survive. TestInstallAPKFilesModesOnDisk passes unprivileged and would fail as root.

Fix: swap to Chown-before-Chmod here. The TypeDir path above is exempt per the repro, but the same order there costs nothing and removes the trap for the next reader.

Please also add a test that pins the ordering. No unprivileged test can observe the kernel behavior, so pin the contract instead: a small recording FullFS double that fails if Chmod on a path arrives before Chown on that path (order-recording only — no behavior emulation, per the usual test-double caveats), plus a comment on the call site stating the constraint. Without one, the next refactor reintroduces this silently, and CI will stay green while it does.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed and fixed in 078c36a — thanks, this was a real one, and your
account of why CI stayed green is exactly right.

I reproduced the kernel behavior independently before touching anything, and
it is even less forgiving than the repro shows: a no-op chown clears the
bits too, unprivileged. chown $(id -u):$(id -g) on my own file took 2755 →
755 and 4755 → 755, while a 2755 directory kept its bit. So the trigger is
not "chown to a different owner", it is "chown at all".

Both branches now do Chown then Chmod, with the constraint written at the call
site.

On the test: there turns out to be a way to observe the kernel behavior
unprivileged, so I did both.

  • TestInstallAPKFilesModesOnDisk gains an entry whose header uid/gid are the
    test process's own. That is the one case where the disk chown succeeds
    without privileges, so it reaches the clearing instead of an EPERM
    isUnsupportedByFS swallows. It fails against the pre-fix commit and passes
    after — no root needed. The existing gid-101 entry keeps passing under the
    old order, which is precisely the blind spot you identified.
  • TestInstallAPKFilesMetadataOrder pins the order itself, as you asked, with
    a recording FullFS that embeds a real memFS and overrides only Chmod and
    Chown to log which reached each path first. Recording only — every call
    still runs against the wrapped filesystem, so there is no emulated behavior
    to drift. It asserts ["chown", "chmod"] for both the setgid file and the
    sticky directory, so it covers the directory path where no kernel behavior
    is observable at all. It fails against the pre-fix commit for both paths.

Keeping both: the first proves the real thing on the path that matters, the
second is what actually stops a future reorder, including on the directory
branch.

Comment thread pkg/apk/apk/install.go
return nil, fmt.Errorf("error setting mode on directory %s: %w", header.Name, err)
}
if err := a.fs.Chown(header.Name, header.Uid, header.Gid); err != nil {
return nil, fmt.Errorf("error setting owner on directory %s: %w", header.Name, err)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same order here for symmetry, ideally — directories are exempt from the chown bit-clearing (repro in the main comment), so this one is not a live bug, but having the two paths disagree on ordering invites the next reader to "clean up" the one that matters.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done — the directory branch is Chown-then-Chmod in 078c36a too, with a
comment saying it is for symmetry and pointing at the file case for the actual
reason. TestInstallAPKFilesMetadataOrder asserts the order on the sticky
directory as well as the setgid file, so the two paths cannot drift apart
quietly.

Review caught a real bug in the ordering. `chown(2)` on a regular
file clears setuid/setgid -- for root as well, and even for a chown
that changes nothing. Verified locally: a 2755 file goes to 755 and a
4755 file to 755 across a no-op self-chown, while a 2755 directory
keeps its bit.

`dirFS.Chmod`/`Chown` write through to the real disk, so the old
`Chmod` then `Chown` sequence left an installed setgid binary at 0755
on disk. The in-memory overrides still reported 02755, which is what
made the loss silent: the serialized image was right while the tree
melange's qemu runner boots was wrong. Unprivileged CI could not see
it either, because `isUnsupportedByFS` swallows the `EPERM` from a
disk chown to a uid the process does not own, so the chown never
happened and the bits survived.

Swap both branches to Chown-then-Chmod. Directories are exempt from
the clearing, but leaving the two paths disagreeing on the order
would invite a cleanup that reintroduces this.

Two tests, because they fail for different reasons:

- `TestInstallAPKFilesModesOnDisk` gains an entry whose header uid/gid
  are the test process's own. That is the one case where the disk
  chown succeeds unprivileged, so it reaches the kernel's clearing
  rather than an EPERM, and it fails against the old order. The
  existing gid-101 entry does not, which is exactly why CI stayed
  green.
- `TestInstallAPKFilesMetadataOrder` pins the call order itself with a
  recording `FullFS` that wraps a real memFS and only logs which of
  `Chmod`/`Chown` reaches each path first. It covers the directory
  path too, where no kernel behavior can be observed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@mattmoor mattmoor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at 078c36a. Verified the respin rather than trusting it:

  • Both branches are Chown-then-Chmod, with the constraint documented at the file call site and the symmetry rationale on the directory branch.
  • All tests pass at this head, and I mutation-tested the fix: swapping the file path back to Chmod-then-Chown fails both new tests — selfowned reaches the real kernel clearing (your no-op-chown observation makes it work unprivileged, and it also holds on macOS, where chown by a non-root owner clears the bits too), and TestInstallAPKFilesMetadataOrder catches it structurally on both paths.
  • The recording double delegates every call to a real memFS and records only — nothing emulated to drift out of sync.

The no-op-chown discovery is the part I'd have missed: it turns "no unprivileged test can observe this" into a same-uid header entry, which is a strictly better guard than the order pin alone.

@smoser
smoser merged commit c7be9d8 into chainguard-dev:main Sep 4, 2026
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

apk: streaming install path drops directory mode bits and all uid/gid

2 participants