feat: background workers (config + ensure)#2393
Open
nicolas-grekas wants to merge 5 commits intophp:mainfrom
Open
feat: background workers (config + ensure)#2393nicolas-grekas wants to merge 5 commits intophp:mainfrom
nicolas-grekas wants to merge 5 commits intophp:mainfrom
Conversation
nicolas-grekas
added a commit
to nicolas-grekas/frankenphp
that referenced
this pull request
May 4, 2026
Adds the worker-to-HTTP shared-state surface deferred from the config+ensure split (php#2393): - frankenphp_set_vars(array $vars): void publishes a snapshot from a background worker. Persistent (pemalloc) memory, RWMutex-protected, cross-thread safe. Skips work when data is identical (=== check). - frankenphp_get_vars(string $name): array reads the latest snapshot. Pure read; throws if the worker is not running or has not published yet. - ensure_background_worker now blocks until the named worker has called set_vars at least once (the readiness signal). The fire-and-forget semantics from the config-only PR become a stronger contract here with no API change visible to callers. - Two-mode ensure: fail-fast in HTTP-worker bootstrap (before frankenphp_handle_request) so a broken dependency surfaces at boot rather than serving degraded traffic; tolerant inside requests so the restart-with-backoff cycle can recover from transient boot failures. - Boot-failure capture: the worker's last PHP error (message, file, line, exit status) is recorded so ensure() can throw a descriptive RuntimeException on timeout. The persistent storage path uses opcache-immutable arrays (zero-copy share), interned strings (no copy), and rich type support: null, scalars, arrays (nested), enums. Tests cover happy-path roundtrips, type coverage, ensure() blocking on first set_vars, fail-fast vs tolerant modes, boot-failure reporting, and the catch-all + scope interactions with vars.
nicolas-grekas
added a commit
to nicolas-grekas/frankenphp
that referenced
this pull request
May 4, 2026
Adds the worker-to-HTTP shared-state surface deferred from the config+ensure split (php#2393): - frankenphp_set_vars(array $vars): void publishes a snapshot from a background worker. Persistent (pemalloc) memory, RWMutex-protected, cross-thread safe. Skips work when data is identical (=== check). - frankenphp_get_vars(string $name): array reads the latest snapshot. Pure read; throws if the worker is not running or has not published yet. - ensure_background_worker now blocks until the named worker has called set_vars at least once (the readiness signal). The fire-and-forget semantics from the config-only PR become a stronger contract here with no API change visible to callers. - Two-mode ensure: fail-fast in HTTP-worker bootstrap (before frankenphp_handle_request) so a broken dependency surfaces at boot rather than serving degraded traffic; tolerant inside requests so the restart-with-backoff cycle can recover from transient boot failures. - Boot-failure capture: the worker's last PHP error (message, file, line, exit status) is recorded so ensure() can throw a descriptive RuntimeException on timeout. The persistent storage path uses opcache-immutable arrays (zero-copy share), interned strings (no copy), and rich type support: null, scalars, arrays (nested), enums. Tests cover happy-path roundtrips, type coverage, ensure() blocking on first set_vars, fail-fast vs tolerant modes, boot-failure reporting, and the catch-all + scope interactions with vars.
2bd6f64 to
dfc0a26
Compare
Introduce background workers via WithWorkerBackground() (Go API) and the Caddyfile `background` token on workers. Background workers share the PHP runtime with HTTP threads but don't serve HTTP requests. They expose a stop pipe (frankenphp_get_worker_handle()) so PHP scripts can park on stream_select and exit gracefully when FrankenPHP drains. The handler auto-restarts the worker on crash with quadratic backoff capped at 1s. The bg worker name is global in this commit; follow-ups will add ensure(), per-php_server scoping, pools, and shared-state APIs (set_vars/get_vars).
Adds frankenphp_ensure_background_worker(string $name): void on top of the minimal background worker from the previous commit. Fire-and-forget: the function lazy-starts the named worker if it is not already running and returns once a thread has been launched, without waiting for the PHP script to reach any particular state (no readiness signal exists in this build; that arrives with the set_vars/get_vars step that follows). Registry + lookup layer: - backgroundWorkerRegistry tracks the template options (env, watch, maxConsecutiveFailures, requestOptions) from one declaration plus the live worker instances spawned from it. Catch-all registries carry a maxWorkers cap. - backgroundWorkerLookup holds a name->registry map plus a single catch- all slot. resolve() falls back to catch-all when the name is not declared. Catch-all dispatch: - A name-less background-worker declaration matches any ensure() name at runtime. max_threads on a catch-all is the cap on how many distinct lazy-started instance names it can host (default 16). Caddyfile no longer requires "name" on background workers, and accepts max_threads > 1 on the catch-all (still rejected on named bg workers). Named lazy path: - A num=0 named declaration registers the worker struct at init but defers thread attach until ensure() schedules it. ensure() reuses the existing struct via workersByName instead of creating a duplicate. calculateMaxThreads now reserves per-bg-worker thread budget separately from HTTP-worker counts and scales catch-all reservations with the declared max_threads, so lazy starts always have a slot to schedule into. metrics.TotalWorkers is registered for bg workers so StartWorker calls in the bg-worker thread aren't silent no-ops in bg-only deployments. $_SERVER['FRANKENPHP_WORKER_NAME'] is now populated for background workers so catch-all instances can tell which name they were started under (lets sentinel-based tests distinguish job-a from job-b). Tests (background_worker_ensure_test.go) cover: - ensure() on a declared num=0 named worker lazy-starts it - ensure() on a name matched by catch-all spawns from the catch-all template; two distinct names produce two independent instances - ensure() with no catch-all and an undeclared name returns the config error - catch-all max_threads cap rejects the (cap+1)th distinct name
Adds a BackgroundScope opaque type (int under the hood; obtain values via NextBackgroundWorkerScope) so each php_server block gets its own isolation boundary for background workers. Zero is the global/embed scope. - backgroundLookups map[BackgroundScope]*backgroundWorkerLookup replaces the single global backgroundLookup. Each scope has its own named registry + catch-all so two blocks can declare bg workers with the same user-facing name without colliding. - buildBackgroundWorkerLookups iterates declarations into their scope's lookup; each declaration still owns its own registry. registry.declared remembers the *worker for a named declaration so lazy-start (num=0) reuses it without scanning the global workersByName map (which is not scope-aware for bg workers). - getLookup(thread) resolves the active scope from the calling thread: worker handler -> request context -> global (0). Scopes that declared their own workers stay strictly isolated; an empty scope falls through to the global lookup so embed-mode workers stay reachable. - Go options: WithWorkerBackgroundScope tags a declaration; the new WithRequestBackgroundScope tags a request so ensure() from a regular HTTP request resolves to the right block's lookup. - Caddy wiring: FrankenPHPModule.Provision allocates one scope per module instance (idempotent across re-provisions) and threads it into worker declarations and ServeHTTP. - workersByName collision check now skips bg workers; they resolve via their scope's lookup, so the same PHP-visible name can appear in two scopes without tripping the duplicate guard. - C side: go_frankenphp_ensure_background_worker now takes the calling thread index so getLookup can resolve the scope from the active handler / request context. Tests: - TestNextBackgroundWorkerScopeIsDistinct: counter hands out unique non-zero scopes. - TestBackgroundWorkerSameNameDifferentScope: two named bg workers with the same user-facing name in distinct scopes both Init successfully and own distinct registries. - TestBackgroundWorkerCatchAllPerScope: ensure() in scope A consumes scope A's catch-all only; scope B's catch-all stays empty. Verified by inspecting the per-scope lookup and the live workers slice via package-internal access. Deferred to follow-ups: pools (num > 1 per named worker, max_threads > 1 for named workers), multiple declarations sharing one entrypoint file in one scope, FRANKENPHP_WORKER_BACKGROUND server flag, batch ensure.
Lifts the remaining constraints on background workers: - Pools: named bg workers can now declare num > 1 (pool of threads per worker) and max_threads > 1. The Caddyfile-level rejections in unmarshalWorker are dropped. - Per-thread stop-pipe: the write fd moved from worker to handler. Each thread in a pool gets its own stop pipe, so drain() can wake them independently. Pools no longer overwrite one another's fd through the shared worker struct. - Multi-entrypoint: multiple named bg workers in the same scope can share the same entrypoint file. Drops the filename-uniqueness rejection in newWorker (it was already skipped via allowPathMatching, this lifts the last Caddyfile-level path check that prevented two named bg workers pointing at the same fixture). Tests: - TestBackgroundWorkerPool: declares num=3, asserts 3 distinct sentinel files appear (each thread tempnam()'s a unique file). - TestBackgroundWorkerMultiEntrypoint: two named bg workers share one entrypoint file; both Init successfully and produce sentinels.
Two small, related polish steps on the bg-worker surface, landing together: - frankenphp_ensure_background_worker now accepts string|array. The array form lazy-starts every named worker fire-and-forget, with the same semantics as the single-string call repeated N times. Input is validated up-front: empty arrays raise ValueError, non-string elements raise TypeError, empty-string and duplicate names raise ValueError. Validation happens before any worker is started so a bad input never leaves a half-spawned batch behind. - $_SERVER['FRANKENPHP_WORKER_BACKGROUND'] = true in background worker scripts, alongside the existing FRANKENPHP_WORKER_NAME wiring. Gives scripts a single-key branch for "am I a bg worker?" without having to probe other frankenphp_* helpers. Set unconditionally for bg workers (catch-all instances with no declared name still see the flag, just no name). ## Tests - TestEnsureBackgroundWorkerBatch: ensure(['a','b','c']) starts three catch-all-resolved instances; assert three per-name sentinels appear. - TestEnsureBackgroundWorkerBatchEmpty: [] raises ValueError. Driven through a PHP fixture that catches the throwable since the validation lives in the Zend parameter-parsing path. - TestEnsureBackgroundWorkerBatchNonString: ['ok-name', 42] raises TypeError, same fixture pattern. - TestEnsureBackgroundWorkerBatchDuplicate: ['dup','dup'] raises ValueError (duplicate names rejected, not silently deduped). - TestBackgroundWorkerBgFlag: bg worker writes var_export() of $_SERVER['FRANKENPHP_WORKER_BACKGROUND'] to a sentinel; assert the exact value is the bool true.
dfc0a26 to
2632517
Compare
nicolas-grekas
added a commit
to nicolas-grekas/frankenphp
that referenced
this pull request
May 4, 2026
Adds the worker-to-HTTP shared-state surface deferred from the config+ensure split (php#2393): - frankenphp_set_vars(array $vars): void publishes a snapshot from a background worker. Persistent (pemalloc) memory, RWMutex-protected, cross-thread safe. Skips work when data is identical (=== check). - frankenphp_get_vars(string $name): array reads the latest snapshot. Pure read; throws if the worker is not running or has not published yet. - ensure_background_worker now blocks until the named worker has called set_vars at least once (the readiness signal). The fire-and-forget semantics from the config-only PR become a stronger contract here with no API change visible to callers. - Two-mode ensure: fail-fast in HTTP-worker bootstrap (before frankenphp_handle_request) so a broken dependency surfaces at boot rather than serving degraded traffic; tolerant inside requests so the restart-with-backoff cycle can recover from transient boot failures. - Boot-failure capture: the worker's last PHP error (message, file, line, exit status) is recorded so ensure() can throw a descriptive RuntimeException on timeout. The persistent storage path uses opcache-immutable arrays (zero-copy share), interned strings (no copy), and rich type support: null, scalars, arrays (nested), enums. Tests cover happy-path roundtrips, type coverage, ensure() blocking on first set_vars, fail-fast vs tolerant modes, boot-failure reporting, and the catch-all + scope interactions with vars.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feat: background workers (config + ensure)
First half of the split suggested in #2287. Lands a minimum-viable background-worker subsystem: config surface, lifecycle, lazy-start via
ensure(), per-php_serverscoping, named pools, multi-entrypoint, plus a$_SERVERflag for bg-aware scripts. The worker-to-HTTP shared-state APIs (frankenphp_set_vars/frankenphp_get_vars) and the docs are deferred to a follow-up PR — they're independent and easier to review separately.What lands
PHP API
frankenphp_ensure_background_worker(string|array $name): void— declares a dependency on one or more bg workers. Fire-and-forget: lazy-starts the named worker (or pulls from a catch-all) if not already running, returns once a thread has been launched. Validates input upfront (ValueErrorfor empty array / empty string / duplicate names;TypeErrorfor non-string elements) so a bad batch never leaves a half-spawned set behind.frankenphp_get_worker_handle(): resource— readable stream that signals graceful shutdown. PHP scripts park onstream_select; FrankenPHP closes the write end during drain soselectwakes with EOF.In CLI mode these functions aren't exposed.
Caddyfile
backgroundmarks a worker as non-HTTP.namepins an exact worker name; declarations withoutnameare catch-alls for lazy-started instances.numon a named bg worker eagerly starts that many instances;num 0(or omitted) defers start untilensure().max_threadson a catch-all caps how many distinct lazy-started instances it can host.max_consecutive_failuresdefaults to 6 (same as HTTP workers).max_execution_timeis automatically disabled for bg workers.Go API
WithWorkerBackground()marks a worker declaration as background.WithWorkerBackgroundScope(scope)tags a declaration with a scope.WithRequestBackgroundScope(scope)tags a request soensure()from a regular HTTP request resolves to the right block's lookup.NextBackgroundWorkerScope()hands out a freshBackgroundScopevalue (opaqueintunder the hood; zero is the global/embed scope).Per-
php_serverscopingEach
php_serverblock gets its own scope. The same user-facing worker name can live in multiple blocks without collision;ensure()resolves through the calling thread's scope (worker handler → request context → global).Pools and multi-entrypoint
num > 1on a named bg worker spawns N threads sharing the same name. Each thread has its own stop pipe so drain can wake them independently.Server variables
$_SERVER['FRANKENPHP_WORKER_NAME']carries the resolved worker name (catch-all instances see the name they were started under).$_SERVER['FRANKENPHP_WORKER_BACKGROUND'] = truefor bg workers — single-key branch for "am I a bg worker?".Lifecycle
max_consecutive_failuresaborts startup if hit during the boot phase.What's deferred
A follow-up PR adds:
frankenphp_set_vars(array $vars): void— publish persistent vars from a bg worker.frankenphp_get_vars(string $name): array— pure read, with generational cache so repeated calls within a request return the same array instance (===is O(1)).frankenphp_set_vars-driven readiness signal that letsensure()block until a worker has bootstrapped (turning fire-and-forget into a stronger contract without an API change).docs/background-workers.mdreference.That split keeps the surfaces independent: this PR is the lifecycle/wiring; the follow-up is the data plane.
Tests
End-to-end tests use file sentinels (workers
toucha path provided via env) instead of cross-thread observation, since this PR has no shared-state API yet:TestBackgroundWorkerLifecycle/TestBackgroundWorkerCrashRestarts/TestBackgroundWorkerWithoutHTTPTestBackgroundWorkerRestartForceKillsStuckThread(force-kill drill on a bg worker stuck insleep(60))TestEnsureBackgroundWorkerNamedLazy/TestEnsureBackgroundWorkerCatchAll/TestEnsureBackgroundWorkerCatchAllCap/TestEnsureBackgroundWorkerUndeclaredTestNextBackgroundWorkerScopeIsDistinct/TestBackgroundWorkerSameNameDifferentScope/TestBackgroundWorkerCatchAllPerScopeTestBackgroundWorkerPool/TestBackgroundWorkerMultiEntrypointTestEnsureBackgroundWorkerBatch(and the three validation-error variants)TestBackgroundWorkerBgFlagTest plan
go test ./...cleanRestartWorkers(), observe re-spawnphp_serverblocks with same-named bg workers, verify they don't collide