fix local gpu queue incomplete preferences

[bghira]

Summary

Ports the queued local GPU preference fix from whitejt2/SimpleTuner@53bc999.

Queued local jobs store GPU preferences in allocated_gpus until the allocator starts them. When a multi-process job only has an incomplete preference, such as [0] for two processes, the pending-job processor can treat it as a hard constraint and leave the job queued even when enough other GPUs are available.

This change drops incomplete queued GPU preferences before persisting or reprocessing them, while preserving complete explicit preferences and any_gpu behavior.

Validation

• .venv/bin/python -m unittest tests.test_local_gpu_allocator tests.test_training_service_gpu -v

[Copilot]

Pull request overview

Ports a fix for local GPU queue handling so that queued local jobs don’t get stuck when they have an incomplete GPU preference list (e.g., [0] for a 2-process job). This ensures the allocator can still start jobs when enough GPUs are available, while preserving explicit complete preferences and any_gpu behavior.

Changes:

• Add _queued_preferred_gpus() to drop incomplete queued GPU preferences before persisting allocated_gpus.
• Update LocalGPUAllocator.process_pending_jobs() to fall back from incomplete stored preferences at queue-processing time.
• Add/extend unit tests to cover the persistence and pending-job fallback behavior.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
simpletuner/simpletuner_sdk/server/services/training_service.py	Adds queued GPU preference sanitization and uses it when enqueuing local jobs.
simpletuner/simpletuner_sdk/server/services/local_gpu_allocator.py	Ensures queued jobs with incomplete preferences don’t deadlock the allocator.
tests/test_training_service_gpu.py	Adds tests for queued GPU preference persistence rules.
tests/test_local_gpu_allocator.py	Adds a regression test for pending-job fallback from incomplete GPU preferences and improves the mock store’s allocated GPU tracking.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.