dist: use bubblewrap for overlayFS mounts, run sccache-dist without root

[rrbutani]

Note

I've opened this PR to get feedback on whether the direction here is acceptable. There are still a few items that need to be addressed (see pending below) before this is ready for merge.

The approach here is more or less the same as in #1628: use CLONE_NEWUSER (unprivileged user namespace) to be able to create a mount namespace + mount the overlayFS mounts without permissions

Unlike #1628, this PR leverages bubblewrap's overlay mount functionality (added in v0.11.0) to create the overlayFS mounts at the sccache-dist → bwrap compile command process boundary. This has a couple of upsides:

• sidesteps the safety issues in Custom sandboxing implementation as linux usernamespace calls - port cachepot/#128 #1628 caused by forking the sccache-dist process
• avoids the issue with overlay fs mounts not being unmounted once the compile command finishes that ultimately led to Custom sandboxing implementation as linux usernamespace calls - port cachepot/#128 #1628 being reverted
  • though, I think this issue was actually caused by Custom sandboxing implementation as linux usernamespace calls - port cachepot/#128 #1628 changing the default sandboxing codepath (i.e. current_ns) to call CLONE_NEWNS in the sccache-dist's rouille request handler thread instead of in the thread spawned for a compile (and not anything to do with the unprivileged user namespace logic)

other notes:

• previously deleting the build directory/the per-compile directories did not run into permission issues because sccache-dist was running as root

pending/open questions:

• fallback support for running on Linux <5.11 and/or with bubblewrap versions older than v0.11.0?
• fallback support for running on machines where unprivileged user namespaces are disabled?
• manual checks for whiteout files/files with userxattr.overlay xattrs in the compile inputs tarball?
  • (see these docs for context)
  • previously this issue was sidestepped by unpacking directly into the overlay mountpoint (target dir); now that we're having bubblewrap handle creating the mountpoint this isn't an option
  • unclear to what degree this is an issue (does not seem like users could do anything nefarious with whiteouts or opaque dirs — worst case a toolchain file is masked?)
  • also: tar::Archive defaults to not restoring xattrs so I think the only consideration is whiteouts?

---

closes #326, closes #1660, closes #1688, supersedes #2706

[codecov-commenter]

Codecov Report

❌ Patch coverage is 0% with 30 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.54%. Comparing base (5d52f91) to head (03b3fcf).
⚠️ Report is 23 commits behind head on main.

Files with missing lines	Patch %	Lines
src/bin/sccache-dist/build.rs	0.00%	30 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2728      +/-   ##
==========================================
+ Coverage   74.51%   74.54%   +0.02%     
==========================================
  Files          70       70              
  Lines       39652    39642      -10     
==========================================
+ Hits        29546    29550       +4     
+ Misses      10106    10092      -14     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[rrbutani]

@sylvestre friendly ping; is there anything I can do to make this PR easier to review?

[luhenry]

Getting that PR to be merged would unblock PyTorch's build of sccache on riscv64. See https://github.com/pytorch/pytorch/pull/182278/changes#diff-0a862d15a4e6dff44196719a594b983aece596c7dc38eba416d83e12a95206e6 for details

[sylvestre]

having tests that cover this change would be helpful :)

[luhenry]

@rrbutani gentle ping on that issue? happy to have a look on adding tests a well.

[rrbutani]

Thanks for the ping, apologies for the delay.

I've updated the existing sccache-dist tests to run sccache-dist server as an unprivileged user.

The first commit in this PR has only these changes; running (for example) cargo test --test dist --features dist-tests -- test_dist_basic at that commit will (as we expect) fail with EPERM. The next commit (unchanged from earlier) updates sccache-dist server to work w/o privileges and gets the dist-tests passing again.

having tests that cover this change would be helpful :)

@sylvestre Is ^ sufficient?

[rrbutani]

@sylvestre friendly ping