fix local queue processing after enqueue - #2880
Merged
bghira merged 1 commit intoJul 23, 2026
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR ensures locally-queued training jobs can start immediately after being enqueued (when queue normalization makes them startable), by kicking the local GPU allocator right after enqueue for non-approval jobs. It also adds unit coverage to verify the allocator kick happens for normal queued jobs but not for approval-held jobs.
Changes:
- Invoke pending-local-job processing immediately after enqueue for non-approval local jobs.
- Add
_process_pending_local_jobs()helper to trigger the local allocator. - Add unit tests asserting pending-job processing is triggered (or intentionally not triggered) after queueing.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
simpletuner/simpletuner_sdk/server/services/training_service.py |
Calls into a new helper after enqueue to kick local pending-job processing (skipped for approval-held jobs). |
tests/test_training_service_gpu.py |
Adds tests covering the new “kick allocator after enqueue” behavior for queued vs approval-held jobs. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+1555
to
+1573
| def _process_pending_local_jobs() -> None: | ||
| """Kick the local GPU allocator after a local job is queued.""" | ||
| import asyncio | ||
|
|
||
| from .local_gpu_allocator import get_gpu_allocator | ||
|
|
||
| async def _async_process(): | ||
| started = await get_gpu_allocator().process_pending_jobs() | ||
| if started: | ||
| logger.info("Started %d pending local job(s): %s", len(started), started) | ||
|
|
||
| try: | ||
| asyncio.get_running_loop() | ||
| import concurrent.futures | ||
|
|
||
| with concurrent.futures.ThreadPoolExecutor() as pool: | ||
| pool.submit(lambda: asyncio.run(_async_process())) | ||
| except RuntimeError: | ||
| asyncio.run(_async_process()) |
bghira
force-pushed
the
agent/local-queue-process-after-enqueue
branch
from
July 23, 2026 14:26
c86b419 to
258954e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Ports the queue-start portion of whitejt2/SimpleTuner@afc7319 as a focused follow-up to #2877.
After #2877, an incomplete queued GPU preference is no longer persisted as a hard pin. However, the submit path can still return a queued result and wait for a later release/callback to process the pending job. This change kicks the local GPU allocator immediately after non-approval local jobs are queued, so jobs that became startable during queue normalization can start without waiting for unrelated queue activity.
Approval-held jobs intentionally do not process pending jobs.
Validation
.venv/bin/python -m unittest tests.test_training_service_gpu tests.test_local_gpu_allocator -vNotes
The original fork commit also changed checkpoint replacement behavior, validation adapter deletion, and one-based GPU normalization. Those pieces were not included here: checkpoint destination deletion needs a separate data-loss review, validation cleanup is partially superseded by current
main, and the current WebUI submits detecteddevice.indexvalues rather than one-based IDs.Stack
agent/local-gpu-incomplete-queue-preferences)