compiler: fall back to local on distributed-compile failure by jetm · Pull Request #2777 · mozilla/sccache

jetm · 2026-07-21T17:46:44Z

Part of #2775 (splitting the OpenEmbedded/Yocto sccache-dist work into focused
PRs; replaces the closed #2750).

Problem

Any remote failure during a distributed compile - a non-zero result, a dropped
output, or unmatched metadata - failed the whole build instead of just the
distribution, so a single distributable input that misbehaved remotely broke the
build. Separately, sccache's own log output on stderr corrupted autoconf/libtool
probe output (for example a libtool -fPIC probe reading wrapper logs off
stderr).

Approach

Treat a distributed-compile failure (a non-zero remote result, or a declared
output that did not materialize) as a trigger to recompile locally.
Honor optional outputs so a missing optional artifact does not fail the
compile.
Route wrapper log output off the compiler's stderr.
Log the distribute/local decision at info and any remote failure at debug, so
the fallback path is diagnosable.

Evidence

Local fallback is exercised repeatedly per cold Yocto build with zero build
failures; the libtool -fPIC probe reproduction is fixed.

sccache logged the distribute-vs-local decision only at debug ("Compiling locally", "Attempting distributed compilation"), while an infrastructure fallback warned. At info a successful distribution and a local compile were both silent, so the only visible dist signal was the failure path - leaving no way to see the distribute/fallback ratio without the full debug firehose. Diagnosing why a distributed build under-distributes meant guessing. Promote the decision points to info and add a log on successful distribution naming the server and exit code, so SCCACHE_LOG=info gives a per-compile dist trace (attempt then distributed-on-server, compiled-locally, or falling-back-with-reason). sccache only emits logs when SCCACHE_LOG is set, so default runs are unaffected. Signed-off-by: Javier Tia <javier@peridio.com> (cherry picked from commit 950ddfdb3e801ecda2d07ade4f6651ffacc750cb)

A distributed compile the build-server rejects is often a distribution artifact, not a genuine compiler error: an object that .incbin's a binary the inputs packager does not ship - the kernel's vdso, embedded-config, and dtb wrappers - cannot be assembled remotely and returns non-zero, which failed the whole build with no recourse and forced the kernel to be excluded wholesale. Treat a non-zero remote result as a fallback trigger rather than a terminal error: recompile locally, which either succeeds (confirming a dist-only artifact) or reproduces the genuine error. A remote failure can no longer break a build that would compile locally. Only failing dist compiles are affected - successful ones are returned unchanged - so the kernel now distributes (1124/1128 compiles), with its handful of .incbin objects falling back to local. Signed-off-by: Javier Tia <javier@peridio.com> (cherry picked from commit b53367ce3083f7ca52861f84a89bbe530c5e80df)

A distributed compile can return a successful (exit 0) result yet omit a declared output. glibc's ldconfig.o and sprof.o compile fine on the build-server, but it does not return their `.o.dt` dependency file, so zipping the compiler outputs fails fatally ("failed to open file ...o.dt: No such file") with no recourse. One dropped output among 6427 forced glibc to be excluded from distribution wholesale, even though 6425 of its compiles distribute cleanly. Capture the declared output paths before the compilation is moved into the packagers, and after a successful remote compile verify each one exists on disk. A missing output is a distribution artifact, not a compiler error, so bail into the existing local-recompile fallback (the same one that already salvages non-zero remote results), which reproduces the full output set. Only compiles whose dist output set is incomplete are affected; complete ones are returned unchanged. glibc now distributes, with ldconfig.o and sprof.o falling back to local. Signed-off-by: Javier Tia <javier@peridio.com> (cherry picked from commit 3b365f3dfa596c4b50815ac4424ee796503232f6)

When sccache runs as a compiler wrapper (`sccache <compiler> ...`) it shares stderr with the wrapped compiler. A build's feature probes treat any unexpected stderr as a compiler failure: libtool's -fPIC check (_LT_COMPILER_OPTION) sets pic_flag="" whenever the probe compile emits stderr, even on exit 0. With SCCACHE_LOG set, the client's own env_logger records ("Attempting to read config file", "Server sent CompileStarted") landed on that shared stderr, so an autotools package configured with pic_flag="" and produced non-PIC objects that failed to link into a shared library (relocation R_X86_64_32 ... recompile with -fPIC). It looked concurrency-dependent because the client's startup logging varies with server-connection timing, so the probe stderr only sometimes diverged from libtool's expected boilerplate. Initialize logging after parsing the command so the compile-wrapper case can choose its target: route sccache's own records to SCCACHE_ERROR_LOG when set, keep stderr only when it is an interactive terminal (so `SCCACHE_LOG=debug sccache gcc ...` at a prompt still shows logs), and discard them when stderr is captured by a build. The wrapped compiler's forwarded stderr is untouched. Signed-off-by: Javier Tia <javier@peridio.com> (cherry picked from commit 63df688bd154dcc97a8b1a8f809d8cc92a952042)

Softening the rust dep-info rewriter (previous commit) was not enough to make rust distribute: a rust compile still fell back to local, now caught by the second guard. After a distributed compile the client verifies every declared output exists and, if one is missing, discards the result and recompiles locally. That guard ignored the `optional` flag, so a missing optional output forced a local recompile even though the output was declared droppable - stranding every rust compile on the client because the rust dep-info (.d) file, which the build server does not reliably return, was declared non-optional. Make the guard honor `optional`: only non-optional outputs must exist. The C dep file and glibc's dropped .o.dt stay non-optional, so their local-fallback (the reason that guard exists) is unchanged; a `-gsplit-dwarf` .dwo, already declared optional, no longer wrongly forces fallback. Then mark the rust .d optional, since it only feeds cargo rebuild tracking a from-scratch bitbake do_compile never consumes. With both, a distributed rust compile whose .d is absent keeps its object and rlib instead of recompiling locally. Signed-off-by: Javier Tia <floss@jetm.me> (cherry picked from commit 0ac0fe17badabb93fd8b22e348bb3f5995780877)

When a distributed compile exits non-zero the client logs only the exit code and falls back to a local recompile, discarding the remote compiler's stdout and stderr. That output is the only place the actual failure is visible - a missing target std, an input the packager did not ship, a sandbox path that does not resolve - so diagnosing a dist regression meant patching in a temporary dump and rebuilding the binary each time. Emit the remote stdout/stderr at debug on the non-zero path. SCCACHE_LOG=debug now surfaces the real remote error with no rebuild, while a normal SCCACHE_LOG=info run stays quiet (the one-line fallback reason is still logged at warn by the caller). Signed-off-by: Javier Tia <floss@jetm.me> (cherry picked from commit 1a4182c1db95b777f6e0ffcac11120e0f2730b0d)

AJIOB · 2026-07-25T18:10:37Z

Hi, thanks for the PR.

Looks like it also resolves #2700 and #2745

AJIOB · 2026-07-26T04:31:38Z

For discussion: maybe we need a feature flag for that

AJIOB · 2026-07-28T06:19:29Z

Hi @sylvestre ,

Is it OK for you?

codecov-commenter · 2026-07-28T07:19:13Z

Codecov Report

❌ Patch coverage is 85.51724% with 21 lines in your changes missing coverage. Please review.
✅ Project coverage is 70.93%. Comparing base (7cd4f2c) to head (974dbe8).
⚠️ Report is 13 commits behind head on main.

Files with missing lines	Patch %	Lines
src/compiler/compiler.rs	79.80%	21 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2777      +/-   ##
==========================================
+ Coverage   70.90%   70.93%   +0.03%     
==========================================
  Files          71       71              
  Lines       40393    40495     +102     
==========================================
+ Hits        28642    28727      +85     
- Misses      11751    11768      +17

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

codspeed-hq · 2026-07-28T07:22:37Z

Merging this PR will degrade performance by 45.99%

❌ 10 regressed benchmarks
✅ 48 untouched benchmarks
⏩ 6 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	`BASE`	`HEAD`	Efficiency
❌	Memory	`lru_sequential_scan_pattern`	128 B	260 B	-50.77%
❌	Memory	`lru_hotcold_access_pattern`	128 B	260 B	-50.77%
❌	Memory	`hash_multiple_files`	128 B	260 B	-50.77%
❌	Memory	`time_macro_finder_chunked`	128 B	260 B	-50.77%
❌	Memory	`lru_cache_get_hit`	128 B	260 B	-50.77%
❌	Memory	`hash_large_data`	192 B	324 B	-40.74%
❌	Memory	`hash_header_file`	192 B	324 B	-40.74%
❌	Memory	`cache_key_generation`	192 B	324 B	-40.74%
❌	Memory	`time_macro_finder_with_macros`	192 B	324 B	-40.74%
❌	Memory	`time_macro_finder_no_macros`	192 B	324 B	-40.74%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing jetm:compiler-local-fallback (974dbe8) with main (7cd4f2c)}

6 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

jetm and others added 6 commits July 21, 2026 10:33

jetm mentioned this pull request Jul 21, 2026

sccache-dist for OpenEmbedded/Yocto: upstreaming tracker jetm/sccache#1

Open

16 tasks

Navigation Menu

Uh oh!

compiler: fall back to local on distributed-compile failure - #2777

compiler: fall back to local on distributed-compile failure#2777
jetm wants to merge 6 commits into
mozilla:mainfrom
jetm:compiler-local-fallback

jetm commented Jul 21, 2026

Uh oh!

AJIOB commented Jul 25, 2026 •

edited

Loading

Uh oh!

AJIOB commented Jul 26, 2026

Uh oh!

AJIOB commented Jul 28, 2026

Uh oh!

codecov-commenter commented Jul 28, 2026 •

edited

Loading

Uh oh!

codspeed-hq Bot commented Jul 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

jetm commented Jul 21, 2026

Problem

Approach

Evidence

Uh oh!

AJIOB commented Jul 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AJIOB commented Jul 26, 2026

Uh oh!

AJIOB commented Jul 28, 2026

Uh oh!

codecov-commenter commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

codspeed-hq Bot commented Jul 28, 2026

Merging this PR will degrade performance by 45.99%

Performance Changes

Footnotes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

AJIOB commented Jul 25, 2026 •

edited

Loading

codecov-commenter commented Jul 28, 2026 •

edited

Loading