feat(server): separate model placement and load balancing

[Graffioh]

This PR lets one dflash_server process serve several models through one HTTP address. You choose where each model loads, which model gets requests first, and whether requests may fall back to another model when the preferred model is busy.

For example, Qwen can stay on the R9700 and DeepSeek4 on Strix Halo. Changing the primary GPU reverses request priority while keeping both model placements unchanged.

Request behavior

Situation	What happens
Load balancing is disabled (the default)	Only the selected primary model loads and serves. Other configured model blocks are unused.
Load balancing is enabled and the primary has capacity	The primary serves the request.
The primary has no free slot or enough available KV-cache capacity	The server tries the other loaded models in their configured order.
Every eligible model is busy	The request waits in memory, subject to --routing-queue-limit. A full waiting queue returns HTTP 503.
A request cannot fit a model's context or total KV capacity	That model is skipped. When no model can fit, the request returns HTTP 400.
A request has already started generating	It stays on that model until completion, cancellation, or failure.

The generation request's model field does not select or pin a model. Omitted names, auto, configured names, and other client aliases all follow the operator's priority. The response identifies the model that actually answered. This applies to Chat Completions, Messages, and Responses.

Fallback happens before response headers or prefill. Each attempt renders and tokenizes the original request with the selected model's own template, tokenizer, and defaults. Validation errors and backend failures other than admission-capacity rejection terminate the request without fallback.

Launch options

Each --model <path> starts a model block. Model-specific options following it apply to that model until the next --model.

Option	Scope	Meaning / default
--model <path>	Per model	Starts a block and selects its GGUF file. The first path may also be positional for existing single-model commands.
--load-balancing-primary-gpu <backend:gpu>	Whole launch	Selects the primary by matching exactly one block's target device. If omitted, the first block is primary. Does not change model placement or enable balancing.
--load-balancing	Whole launch	Enables primary-first capacity fallback and loads all configured models. Off by default; requires at least two model blocks.
--routing-queue-limit <N>	Listener; first block	Maximum waiting routing requests. Default 32; 0 returns 503 immediately when every eligible model is busy.

The old --primary-gpu, --load-balancing-next-model, and --next-model spellings are rejected.

Example: Qwen on R9700, DeepSeek4 on Strix Halo

This example assumes R9700 is hip:0 and Strix Halo is hip:1. Replace the paths and device IDs with those on the host.

dflash_server --load-balancing --load-balancing-primary-gpu hip:0 \
  --model /models/qwen38-27b.gguf \
  --model-name qwen --target-device hip:0 \
  --draft /models/qwen38-dflash2.gguf --draft-device hip:0 \
  --max-ctx 4096 --max-concurrency 4 --kv-pool-tokens 16384 \
  --cache-type-k q8_0 --cache-type-v q8_0 --fa-window 0 \
  --default-max-tokens 2048 --hard-limit-reply-budget 1024 \
  --host 127.0.0.1 --port 8080 --routing-queue-limit 32 \
  --model /models/deepseek-v4-flash.gguf \
  --model-name ds4 --target-device hip:1 \
  --max-ctx 8192 --max-concurrency 1 \
  --default-max-tokens 2048 --hard-limit-reply-budget 1024 \
  --prefix-cache-slots 0 --prefill-cache-slots 0 --disk-prefix-cache off \
  --ds4-fused-decode --ds4-expert-top-k 6 --ds4-prefill exact

Here Qwen can hold up to four active reservations, and DS4 one. KV pressure can trigger fallback before all four Qwen slots are occupied. Change only the primary flag to --load-balancing-primary-gpu hip:1 to try DS4 first. Remove --load-balancing to load and serve only that selected primary.

Ownership and observability

The listener reuses the existing model loaders and workers. Each model owns its tokenizer, defaults, backend, and scheduler. A listener reservation lasts through engine retirement and output draining, so a disconnect cannot expose capacity while GPU work is still running. The scheduler owns actual KV admission. GPU draft top-K and CUDA sampling scratch are owned by each worker thread and released on their allocation device when the worker exits. Native single-request workers finish active requests during SIGTERM, including terminal response events; genuine client disconnects still cancel generation, and routing waiters receive 503.

/v1/models lists the loaded models. /props and /status/json expose their placement, priority order, capacity, in-flight reservations, and waiting counts. In multi-model serving, token counting requires an explicit model name and uses that model's tokenizer without fallback.

Limits

Waiting is in memory, without a server queue deadline or FIFO guarantee. Capacity release wakes waiters; a 250 ms retry also observes engine capacity changes, disconnects, and shutdown. Slow readers can retain reservations while output drains.

This change selects a model before generation. It does not migrate active streams, suspend requests to SSD, or recover universally from GPU OOM. Admission protects the prompt and rolling decode headroom; later cache exhaustion keeps the existing error/retirement behavior. Cross-model continuation would require text replay because KV/recurrent states are incompatible.

Environment settings remain process-wide. With balancing enabled, startup rejects unsupported shared-policy combinations, including KVFlash/Spark CLI controls, compression, remote drafts, and target splitting. With balancing disabled, the selected primary retains single-model CLI features; unused blocks do not apply environment policy. No throughput or latency improvement is claimed.

Validation

• 500 server unit tests passed, including native streaming/non-streaming shutdown completion, routed native/hybrid draining, disconnect cancellation, and waiter cleanup. All nine CLI model-name/policy cases passed.
• All 13 native CUDA sampler/top-K fixture tests passed on DGX GB10 (sm_121), including both concurrent-worker tests; the full NVIDIA GPU CI job passed. CI now explicitly runs these fixtures and the three HIP top-K cases on AMD.
• HIP top-K parity and concurrent-worker/device-switch tests passed. The deterministic interleaving that previously returned another worker's candidate IDs now returns each caller's correct IDs.
• Qwen3.8-27B + DFlash2 on R9700 and native DS4 on Strix each finish all 400 requested tokens with [DONE] after SIGTERM. Their complete outputs match the clean pre-PR reference exactly; the prior head stopped at 104 and 101 generated tokens respectively.
• Final real-model qualification passed 76 assertions: native Qwen + DS4 and batched Qwen + DS4 routing, fallback across all three APIs, queue overflow, waiter abandonment, disconnect cleanup, capacity reuse, and shutdown 503 responses. Native active requests drain normally; batched slots retain their existing cancellation/finalization behavior.
• Three concurrent two-Qwen DDTree/DFlash2 request pairs completed cleanly and matched the earlier CPU top-K control. A disabled-balancing launch with an unused KVFlash-configured DS4 block successfully served through the selected paged Qwen model. All test processes exited.

The server was rebuilt with the clean PR dependencies for both gfx1201 and gfx1151. Native NVIDIA execution passed on DGX GB10 in CI; all 10 local sampler checks also passed using a disposable HIP adaptation. Tests use the repository-converted GGUF DFlash2 draft because the existing safetensors loader does not load its selector.

Usage and detailed boundaries: MODEL_LOAD_BALANCING.md. The loader is adapted from commit 63151ff8, preserved behind luce_box#103 and luce_box#105.

[cubic-dev-ai]

All reported issues were addressed across 22 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 4 files (changes from recent commits).

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 10 files (changes from recent commits).

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>