Enable cache support (FirstBlockCache) for Cosmos3 Omni#14154
Open
johnnynunez wants to merge 1 commit into
Open
Enable cache support (FirstBlockCache) for Cosmos3 Omni#14154johnnynunez wants to merge 1 commit into
johnnynunez wants to merge 1 commit into
Conversation
- Add CacheMixin to Cosmos3OmniTransformer so pipe.transformer.enable_cache()
works (FirstBlockCacheConfig etc.).
- Register Cosmos3VLTextMoTDecoderLayer in TransformerBlockRegistry. The MoT
dual-stream layer takes/returns (und_seq, gen_seq); the gen stream carries
the denoised tokens and is mapped to hidden_states for cache decisions,
since the und/text stream is quasi-static across denoising steps and would
produce degenerate always-hit cache decisions.
- Add encoder_hidden_states_argument_name to TransformerBlockMetadata and use
the metadata argument names in the FBC hooks (previously hardcoded
'hidden_states'/'encoder_hidden_states'; the existing
hidden_states_argument_name field was ignored by first_block_cache).
- Wrap Cosmos3OmniPipeline cond/uncond transformer calls in
cache_context('cond'/'uncond') so stateful caches keep separate CFG state,
matching the Wan pipeline pattern.
Contributor
|
Hi @johnnynunez, thanks for the PR! It does not appear to link an issue it fixes. If this PR addresses an existing issue, please add a closing keyword (e.g. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Enables the native diffusers cache framework (
CacheMixin→FirstBlockCacheConfig, etc.) for Cosmos3 Omni (nvidia/Cosmos3-Nano/nvidia/Cosmos3-Super,Cosmos3OmniPipeline).Before this PR,
pipe.transformer.enable_cache(...)fails becauseCosmos3OmniTransformerdoesn't inheritCacheMixin, and even with the mixin the FBC hooks can't read the block I/O: Cosmos3's MoT dual-stream decoder layers useforward(und_seq, gen_seq, rotary_emb) -> (und_seq, gen_seq)instead of thehidden_states/encoder_hidden_statescontract.Changes
Cosmos3OmniTransformer: addCacheMixin.TransformerBlockRegistry: registerCosmos3VLTextMoTDecoderLayerwithhidden_states_argument_name="gen_seq"/encoder_hidden_states_argument_name="und_seq"and return indices(1, 0). Mapping the gen stream tohidden_statesis load-bearing: the cache decision must track the denoised tokens — theund(text/understanding) stream is quasi-static across denoising steps and would produce degenerate always-hit decisions.TransformerBlockMetadata: addencoder_hidden_states_argument_name(analogous to the existinghidden_states_argument_name), and make the FBC hooks use both metadata fields instead of hardcoded"hidden_states"/"encoder_hidden_states"strings. Note the existinghidden_states_argument_namefield (used by Kandinsky5'svisual_embed) was silently ignored byfirst_block_cache.py— this also fixes that for any model with a custom argument name.Cosmos3OmniPipeline: wrap the cond/uncond transformer calls incache_context("cond"/"uncond"), following the Wan pipeline pattern, so stateful caches keep separate CFG state (the pipeline runs CFG as two separate transformer calls).Measured (real hardware)
nvidia/Cosmos3-NanoT2V, RTX PRO 6000 Blackwell (96 GB, sm_120), torch 2.12.1+cu132, 189 frames / 35 steps / UniPCflow_shift=10, fixed seed, warmup excluded, median of 3:threshold=0.10threshold=0.10+torch.compilethreshold=0.10Output latents finite (no NaN, mean|v| ≈ 0.67–0.77, in line with dense runs). As expected for step-caching, the sample is different-but-equivalent vs the dense trajectory at the same seed (this is inherent to FBC, not specific to this integration).
Who can review?
@a-r-r-o-w @DN6 (cache hooks / Cosmos)