Ask
Please add a supported setting — env var or config — for the maximum number of index builds the daemon runs at once, box-wide. Something like CBM_MAX_CONCURRENT_BUILDS=1, where extra requests queue instead of starting another --index-worker.
Why
On an 8-core / 32 GB host, three cli --index-worker --index-worker-build <same hash> processes built the same repository at once under 0.10.8 and together took ~7 GB RSS plus swap. Memory PSI crossed the pressure guard of another service on the box and that service returned 503 for 34 minutes. The index itself was fine; the collateral damage was the problem.
What I measured (0.10.8 vs 0.11.0)
| Case |
0.10.8 |
0.11.0 |
3 simultaneous index_repository on the same repo |
3 workers on the same build hash (the incident above) |
1 worker; all three clients returned status: indexed — the de-duplication works, thank you |
3 simultaneous index_repository on 3 different repos |
3 concurrent workers |
3 concurrent workers |
So 0.11.0 fixes the duplicate-same-repo case. What is still missing is a cap across distinct repos, which is the case a multi-repo host hits routinely: several agents each ask for their own repository at the same time and N builds start together.
What I could and could not find
physical_job_limit shows up in daemon.start (physical_job_limit=4 on 8 cores) but there seems to be no way to set it from outside.CBM_WORKERS controls threads within a build, not the number of builds.CBM_MEM_BUDGET_MB is honoured (mem.init ... source=CBM_MEM_BUDGET_MB) and helps per-worker, but it does not bound how many workers start.
If physical_job_limit is already the intended knob, exposing it as an env var would be enough.
Workaround in use
Starting the daemon inside a systemd scope with MemoryHigh/MemoryMax — workers inherit the daemon's cgroup, so the daemon and all of its builds share one ceiling. It bounds the blast radius but does not serialize builds, which is why the setting would help.
Ask
Please add a supported setting — env var or config — for the maximum number of index builds the daemon runs at once, box-wide. Something like
CBM_MAX_CONCURRENT_BUILDS=1, where extra requests queue instead of starting another--index-worker.Why
On an 8-core / 32 GB host, three
cli --index-worker --index-worker-build <same hash>processes built the same repository at once under 0.10.8 and together took ~7 GB RSS plus swap. Memory PSI crossed the pressure guard of another service on the box and that service returned 503 for 34 minutes. The index itself was fine; the collateral damage was the problem.What I measured (0.10.8 vs 0.11.0)
index_repositoryon the same repostatus: indexed— the de-duplication works, thank youindex_repositoryon 3 different reposSo 0.11.0 fixes the duplicate-same-repo case. What is still missing is a cap across distinct repos, which is the case a multi-repo host hits routinely: several agents each ask for their own repository at the same time and N builds start together.
What I could and could not find
physical_job_limitshows up indaemon.start(physical_job_limit=4on 8 cores) but there seems to be no way to set it from outside.CBM_WORKERScontrols threads within a build, not the number of builds.CBM_MEM_BUDGET_MBis honoured (mem.init ... source=CBM_MEM_BUDGET_MB) and helps per-worker, but it does not bound how many workers start.If
physical_job_limitis already the intended knob, exposing it as an env var would be enough.Workaround in use
Starting the daemon inside a systemd scope with
MemoryHigh/MemoryMax— workers inherit the daemon's cgroup, so the daemon and all of its builds share one ceiling. It bounds the blast radius but does not serialize builds, which is why the setting would help.