[CHORE] k3s snapshots lifecycle

[Flegma]

As someone on Discord asked, we should register a cronjob to clean snapshots in the /var/lib/rancher/k3s/agent/ directory.

Maybe we can do a cronjob that would clean them on monthly basis for start, and if needed at later point, we can add some .env variable that would allow people to configure how many days they want to keep it.

[lukepolo]

k3s crictl rmi --prune. only issue is it will remove game-server images / game-streamer images too since they wont be running probably

[Flegma]

@lukepolo here's how I'm thinking we tackle this.

First, a quick clarification on what's actually filling up: it's not etcd snapshots, it's containerd's overlayfs image snapshots under /var/lib/rancher/k3s/agent/containerd/... - i.e. the layers behind old ghcr.io/5stackgg/* images, the same stuff fix-versions.sh clears by hand today. I checked one of our nodes (k3s v1.34.4+k3s1) and even a nearly-idle one is already sitting on ~1.6 GB of snapshots; on a node that's churned a lot of image versions this just keeps climbing until the disk fills and the kubelet starts evicting.

One thing to be careful about: we should not rm files out of that directory directly - containerd tracks every snapshot in its own metadata DB, so deleting them underneath it corrupts state and breaks image pulls. The safe path is crictl rmi / kubelet GC, which drop the image and its snapshot together.

So rather than a single cronjob, I'd split it into two layers:

1. A systemd timer running k3s crictl rmi --prune (the main cleanup). --prune removes all unused images (confirmed on the node), and a systemd timer keeps it consistent with the pattern we already use on nodes - same shape as the 5stack-tailscale-state-check.timer we write in game-node-server-setup.sh. Start it weekly/monthly. Works on every node since it only needs the local containerd socket.

2. Kubelet image GC as the native safety net. We already set kubelet-arg in /etc/rancher/k3s/config.yaml, so it's a clean place to add GC thresholds and let the kubelet auto-reclaim under disk pressure between timer runs. And for the "keep N days, configurable" idea from the issue, the kubelet's imageMaximumGCAge is exactly that knob (time-based, unused images only) - we're on 1.34 so it's available; since it's a KubeletConfiguration field we'd wire it through a kubelet config drop-in (exact form is a PR detail).

I'd skip a Kubernetes CronJob for this - it runs as a single pod on one node, but this is per-host cleanup (one pod can't reach every node's containerd), so a per-node timer plus the kubelet itself is the right layer.

A few rollout notes:

• Should apply to both the panel host (install.sh) and game nodes (game-node-server-setup.sh) - both run containerd.
• The host-level units are only written during initial setup right now, so we'll want to fold the unit-writing into the update path too, otherwise existing nodes won't pick it up.
• Make the retention/interval env-configurable (e.g. IMAGE_GC_MAX_AGE_DAYS, default 30) the same way the postgres backup retention works.

If you're good with this direction I'll put up a PR. Happy to keep it to just the timer for a first pass if you'd rather start minimal.

[lukepolo]

i think thats fine except. i dont want to clear out game-streamer / game-server latest image (they most likely wont be running while this prune happens, causing us to delete them when they are in fact the newest)

[Flegma]

Status check (2026-07-10): not done yet, but the implementation is ready for review.

The work lives in PR #513 (open since June 19). It is built specifically around the concern raised here: the prune keys off the :latest tag rather than "is it running", so the current game-server / game-streamer images are kept even while idle, and only superseded untagged ghcr.io/5stackgg/* versions are removed. It deliberately avoids crictl rmi --prune for exactly that reason.

Re-verified today against current main:

• The PR is still mergeable clean; none of its touched paths (update.sh, utils/) have drifted since it was opened.
• The selection logic re-checked: pinned images (pause sandbox) and non-5stack images are never touched, and anything holding a :latest tag survives, running or not.

(This is separate from the registry-side tag prune tried and reverted in game-server-node-connector #15/#16; that was GHCR cleanup in CI, this one reclaims node disk.)

Nothing needed updating on the PR. The remaining decisions are the open questions in the PR body: weekly vs monthly default schedule, whether to also clear non-5stack dangling images, and the jq dependency note. Review when you get a chance @lukepolo.