Skip to content

Add real Ideogram 4 unconditional transformer support - #3076

Merged
bghira merged 3 commits into
mainfrom
feature/ideogram-unconditional-transformer
Aug 14, 2026
Merged

Add real Ideogram 4 unconditional transformer support#3076
bghira merged 3 commits into
mainfrom
feature/ideogram-unconditional-transformer

Conversation

@bghira

@bghira bghira commented Aug 14, 2026

Copy link
Copy Markdown
Owner

What

Ideogram 4 performs CFG through a second, image-only unconditional transformer with zeroed text conditioning. Until now SimpleTuner approximated the unconditional branch by running negative prompts through the conditional transformer, which produces structurally different guidance — particularly visible as noisy targets when AnyFlow distillation fuses guidance at higher scales.

  • --ideogram_load_unconditional_transformer loads the frozen unconditional transformer through the same indexed state-dict machinery as the conditional path, respecting --ideogram_fp8_base_upcast, and wires it into both pipeline constructions so validation runs the real asymmetric-CFG branch.
  • --ideogram_uncond_ramtorch builds the unconditional transformer on CPU and streams its layers to the accelerator per forward pass via the shared _apply_ramtorch_layers helper (new force flag bypasses the global --ramtorch gate), for fitting both transformers on smaller cards. VRAM cost of the resident option: ~10 GiB fp8, ~18 GiB upcast.
  • AnyFlow's _unconditional_batch marks batches with is_unconditional_pass, and Ideogram4.model_predict dispatches marked batches to the unconditional transformer with zeroed llm conditioning, so fused-guidance distillation trains against the model's true unconditional output.
  • Quickstart Validation section documents both options (en, es, ja, zh, hi, pt-BR).

Testing

.venv/bin/python -m unittest tests.test_ideogram_unconditional tests.test_ideogram_fp8_linear tests.helpers.distillation.test_anyflow_distiller -v -f — 75 tests pass (1 environment-dependent skip). Coverage includes flag-off/flag-on loading, fp8 upcast handling, the ramtorch offload path, model_predict dispatch geometry, and the unconditional batch marker.

bghira added 3 commits August 13, 2026 19:59
--ideogram_load_unconditional_transformer loads the frozen image-only
unconditional transformer via the same indexed state-dict machinery as
the conditional path, respecting ideogram_fp8_base_upcast, and wires it
into both pipeline constructions so validation runs the real
asymmetric-CFG branch.

--ideogram_uncond_ramtorch builds the unconditional transformer on CPU
and reuses the shared _apply_ramtorch_layers helper (new force flag
bypasses the global --ramtorch gate) to stream its layers to the
accelerator per forward pass.

AnyFlow's _unconditional_batch now marks batches with
is_unconditional_pass, and Ideogram4.model_predict dispatches marked
batches to the unconditional transformer with zeroed llm conditioning
and image-only geometry, matching the pipeline's uncond branch. Without
the flag, the proxy path through the conditional transformer is
unchanged.
Covers flag-off/flag-on loading (single vs dual _build_transformer
calls, frozen eval-mode result, fp8 upcast handling), the ramtorch
offload path, the model_predict dispatch geometry, and AnyFlow's
is_unconditional_pass batch marker. The fp8 upcast test helper now
pins the new config flags so Mock truthiness cannot trigger the
unconditional load.
Extends the Ideogram 4 quickstart Validation section (and its es, ja,
zh, hi, pt-BR translations) with ideogram_load_unconditional_transformer
and ideogram_uncond_ramtorch, the ~10 GiB fp8 / ~18 GiB upcast VRAM
cost, and the AnyFlow fused-guidance behaviour when the real
unconditional branch is loaded.
@bghira
bghira force-pushed the feature/ideogram-unconditional-transformer branch from 889f505 to c24adf9 Compare August 14, 2026 02:00
@bghira
bghira merged commit d527373 into main Aug 14, 2026
3 checks passed
@bghira
bghira deleted the feature/ideogram-unconditional-transformer branch August 14, 2026 02:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant