Harden CI container options and cluster training scripts

GPU CI jobs: drop the explicit root user and the disabled seccomp profile
from the container options; the ubuntu image already runs as root for
the apt-get steps and the default seccomp profile is sufficient.

SLURM/Docker path: the base-image fallback no longer makes the shared
output directory world-writable (chmod 1777). Instead sbatch_train.sh
passes the submitting user's UID/GID into the container and
cluster_container_install_and_train.sh drops privileges with setpriv
before running training, so output files are owned by the user. Verified
in a local ubuntu:22.04 container that files written under the mounted
output directory are owned by the host UID.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Ivan Basov <ibasov@nvidia.com>

Author: ivanbasov

.github/workflows/ci-gpu.yml

@@ -47,7 +47,7 @@ jobs:
       contents: write
     container:
       image: ubuntu:24.04
-      options: -u root --security-opt seccomp=unconfined --shm-size 16g
+      options: --shm-size 16g
       env:
         NVIDIA_VISIBLE_DEVICES: ${{ env.NVIDIA_VISIBLE_DEVICES }}
     timeout-minutes: 45
@@ -129,7 +129,7 @@ jobs:
     runs-on: linux-amd64-gpu-rtxpro6000-latest-1
     container:
       image: ubuntu:24.04
-      options: -u root --security-opt seccomp=unconfined --shm-size 16g
+      options: --shm-size 16g
       env:
         NVIDIA_VISIBLE_DEVICES: ${{ env.NVIDIA_VISIBLE_DEVICES }}
     timeout-minutes: 40
@@ -197,7 +197,7 @@ jobs:
     runs-on: linux-amd64-gpu-rtxpro6000-latest-2
     container:
       image: ubuntu:24.04
-      options: -u root --security-opt seccomp=unconfined --shm-size 16g
+      options: --shm-size 16g
       env:
         NVIDIA_VISIBLE_DEVICES: ${{ env.NVIDIA_VISIBLE_DEVICES }}
     timeout-minutes: 35
@@ -303,7 +303,7 @@ jobs:
       contents: write
     container:
       image: ubuntu:24.04
-      options: -u root --security-opt seccomp=unconfined --shm-size 16g
+      options: --shm-size 16g
       env:
         NVIDIA_VISIBLE_DEVICES: ${{ env.NVIDIA_VISIBLE_DEVICES }}
     timeout-minutes: 20

.github/workflows/long-running-tests.yml

@@ -90,7 +90,7 @@ jobs:
     runs-on: linux-amd64-gpu-rtxpro6000-latest-1
     container:
       image: ubuntu:24.04
-      options: -u root --security-opt seccomp=unconfined --shm-size 16g
+      options: --shm-size 16g
       env:
         NVIDIA_VISIBLE_DEVICES: ${{ env.NVIDIA_VISIBLE_DEVICES }}
     timeout-minutes: 30
@@ -140,7 +140,7 @@ jobs:
     runs-on: linux-amd64-gpu-rtxpro6000-latest-1
     container:
       image: ubuntu:24.04
-      options: -u root --security-opt seccomp=unconfined --shm-size 16g
+      options: --shm-size 16g
       env:
         NVIDIA_VISIBLE_DEVICES: ${{ env.NVIDIA_VISIBLE_DEVICES }}
     timeout-minutes: 90
@@ -219,7 +219,7 @@ jobs:
     runs-on: linux-amd64-gpu-rtxpro6000-latest-1
     container:
       image: ubuntu:24.04
-      options: -u root --security-opt seccomp=unconfined --shm-size 16g
+      options: --shm-size 16g
       env:
         NVIDIA_VISIBLE_DEVICES: ${{ env.NVIDIA_VISIBLE_DEVICES }}
     timeout-minutes: 60
@@ -271,7 +271,7 @@ jobs:
     runs-on: linux-amd64-gpu-rtxpro6000-latest-1
     container:
       image: ubuntu:24.04
-      options: -u root --security-opt seccomp=unconfined --shm-size 16g
+      options: --shm-size 16g
       env:
         NVIDIA_VISIBLE_DEVICES: ${{ env.NVIDIA_VISIBLE_DEVICES }}
     timeout-minutes: 90

TRAINING.md

@@ -296,8 +296,10 @@ $SHARED_OUTPUT_DIR/
 3. **Docker base image**: set `DOCKER_BASE_IMAGE` if your cluster uses a different CUDA runtime.
 
 4. **File systems**: `SHARED_OUTPUT_DIR` should be on a shared/persistent filesystem
-   visible from all nodes (NFS, Lustre, etc.). The sbatch script sets `chmod 1777` for
-   NFS compatibility when using Docker.
+   visible from all nodes (NFS, Lustre, etc.). When using Docker, the training
+   process runs as the submitting user's UID/GID (the base-image path installs
+   dependencies as root, then drops privileges), so no special directory
+   permissions are needed.
 
 5. **No Docker?** The scripts fall back to bare-metal install automatically.
    Ensure the node has internet access (for pip) or pre-install deps via `cluster_install_deps.sh`.

code/scripts/cluster_container_install_and_train.sh

@@ -66,9 +66,24 @@ if [ -z "${PREDECODER_PYTHON:-}" ]; then
   [ -x "${INSTALL_DIR}/miniconda3/envs/predecoder/bin/python" ] && export PREDECODER_PYTHON="${INSTALL_DIR}/miniconda3/envs/predecoder/bin/python"
 fi
 log "PREDECODER_PYTHON=${PREDECODER_PYTHON:-<unset>}"
+
+# Drop privileges for training when the container was started as root on behalf
+# of a host user (HOST_UID/HOST_GID set by sbatch_train.sh). Installation above
+# needs root; training does not, and files written to the shared output
+# directory should be owned by the submitting user.
+RUN_AS=()
+if [ "$(id -u)" -eq 0 ] && [ -n "${HOST_UID:-}" ] && [ "${HOST_UID}" != "0" ] && command -v setpriv >/dev/null 2>&1; then
+  HOST_GID="${HOST_GID:-$HOST_UID}"
+  TRAIN_HOME="${TRAIN_HOME:-/tmp/predecoder-home}"
+  mkdir -p "$TRAIN_HOME" && chown "${HOST_UID}:${HOST_GID}" "$TRAIN_HOME"
+  export HOME="$TRAIN_HOME" USER="predecoder" LOGNAME="predecoder"
+  RUN_AS=(setpriv --reuid="$HOST_UID" --regid="$HOST_GID" --clear-groups --inh-caps=-all)
+  log "Training as uid:gid ${HOST_UID}:${HOST_GID} (HOME=$HOME)."
+fi
+
 if command -v stdbuf >/dev/null 2>&1; then
-  stdbuf -oL -eL bash "$REPO_ROOT/code/scripts/cluster_train.sh"
+  "${RUN_AS[@]}" stdbuf -oL -eL bash "$REPO_ROOT/code/scripts/cluster_train.sh"
 else
-  bash "$REPO_ROOT/code/scripts/cluster_train.sh"
+  "${RUN_AS[@]}" bash "$REPO_ROOT/code/scripts/cluster_train.sh"
 fi
 log "Done."

code/scripts/sbatch_train.sh

@@ -152,13 +152,17 @@ if command -v docker >/dev/null 2>&1 && docker image inspect "$DOCKER_IMAGE" >/d
     "${COMMON_ENV[@]}" \
     "$DOCKER_IMAGE" 2>&1
 elif command -v docker >/dev/null 2>&1 && docker run --rm --gpus all "$DOCKER_BASE_IMAGE" nvidia-smi -L >/dev/null 2>&1; then
-  log "Setting sticky bit on $SHARED_OUTPUT_DIR for NFS/Docker UID compatibility."
-  chmod 1777 "$SHARED_OUTPUT_DIR" 2>/dev/null || true
-  log "Using Docker base image $DOCKER_BASE_IMAGE (install + train)."
+  log "Using Docker base image $DOCKER_BASE_IMAGE (install as root, train as ${HOST_UID}:${HOST_GID})."
+  # The container starts as root to apt-get/pip install, then
+  # cluster_container_install_and_train.sh drops to HOST_UID:HOST_GID before
+  # training so files written to the shared output directory are owned by the
+  # submitting user (and NFS root-squash is not an issue).
   docker run --rm --gpus all \
     -v "${REPO_ROOT}:/app:ro" \
     -v "${SHARED_OUTPUT_DIR}:/data" \
     -e INSTALL_DIR=/opt/predecoder_env \
+    -e "HOST_UID=${HOST_UID}" \
+    -e "HOST_GID=${HOST_GID}" \
     "${COMMON_ENV[@]}" \
     "$DOCKER_BASE_IMAGE" \
     bash -c 'export DEBIAN_FRONTEND=noninteractive; apt-get update -qq && apt-get install -y -qq python3.11 python3.11-venv python3.11-dev curl git coreutils build-essential cmake >/dev/null 2>&1; exec stdbuf -oL -eL bash /app/code/scripts/cluster_container_install_and_train.sh 2>&1'