Qwen3-Omni-30B-A3B-Instruct

[qingzwang]

Note: The below template includes items meant for model contributions only. For other contributions such as bug fixes, features, etc., only fill out the relevant portions of the form.

Description

Add Qwen3-Omni-30B-A3B-Instruct to the contrib models folder. This is a full multimodal (audio-in + audio-out) implementation running all
five neural network modules on Neuron with TP=8: Thinker MoE (48 layers), Vision encoder, Audio encoder, Talker MoE (20 layers), and
Unified Code Predictor (15-step unrolled). Includes progressive TTFB optimizations (pipelined thinker↔talker, streaming code2wav) that
bring audio-output TTFB from 2.7s down to 1.76s on real conversations.

Model Information

Model Name: Qwen3-Omni-30B-A3B-Instruct

Model Architecture: Mixture-of-Experts multimodal transformer (thinker 48L MoE + talker 20L MoE + vision encoder + audio encoder +
code predictor)

Purpose: Multimodal inference — ASR (speech→text) and TTS (text→speech / audio-in→audio-out)

Checklist

Required Components

• Accuracy Test (test/integration/test_model.py)

  • Integration test validates thinker text model compilation and inference on Neuron
  • Uses logit validation via get_generate_outputs
  • Test compiles and runs on Neuron with TP=32

• README.md with the following sections:

  • Usage Example: Multiple usage examples (ASR, TTS, streaming)
  • Compatibility Matrix: Tested on trn2.48xlarge with Neuron SDK 2.24
  • Example Checkpoints: Qwen/Qwen3-Omni-30B-A3B-Instruct on HuggingFace Hub
  • Testing Instructions: Commands to run ASR and audio-output pipelines

• Source Code (src/)

  • Complete modeling code for all five modules
  • Properly structured in contrib folder hierarchy

Optional Components

• Unit Tests (CPU-based)
  • test/unit/test_config_and_state_dict.py: config loading and HF→Neuron state dict conversion tests
  • Runs on CPU without Neuron devices

Folder Structure

/contrib/models/Qwen3-Omni-30B-A3B-Instruct/
README.md
BENCHMARK_OMNI2_TTFB.md
/src
modeling_qwen3_omni.py
modeling_qwen3_omni_text.py
modeling_qwen3_omni_audio.py
modeling_qwen3_omni_talker.py
modeling_qwen3_omni_code_predictor.py
modeling_qwen3_omni_vision.py
modeling_qwen3_omni_moe.py
_upstream_compat.py
_model_path.py
/test
/unit
test_config_and_state_dict.py
/integration
test_model.py

Testing

How did you test this change?

Tested on trn2.48xlarge (8 Neuron cores, TP=8). Ran:

• ASR on LibriSpeech test-clean (100 samples): 18.5% WER, RTF 0.12x
• Full audio-output pipeline: 3.8s end-to-end for 7.9s audio (RTF 0.48x)
• Streaming TTFB benchmark on 100 real conversations: mean 1759ms, p95 1822ms
• Unit tests on CPU for config/state-dict validation

Test Results:

Pipeline	Metric	Value
ASR (100 samples)	WER	18.5%
ASR (100 samples)	RTF	0.12x
TTS (short prompt)	Total latency	3.8s for 7.9s audio
TTS (short prompt)	RTF	0.48x
Streaming TTFB (100 convs)	Mean	1759 ms
Streaming TTFB (100 convs)	p95	1822 ms

Compatibility

Tested with:

• Neuron SDK Version(s): 2.24
• Instance Type(s): trn2.48xlarge
• PyTorch Version: 2.9.1
• Python Version: 3.10

Additional Information

• All five modules (thinker, vision, audio encoder, talker, code predictor) run on Neuron
• Code2Wav (codec→waveform) stays on CPU (~1s) as the overhead of Neuron dispatch would negate the win for this small model
• First compilation takes ~45 min total; subsequent runs reuse cached NEFFs
• Requires ~60 GB CPU RAM for HF model (projections still on CPU)
• Pipelined thinker↔talker optimization overlaps text generation with talker startup

Related Issues

N/A

vLLM Integration

• This model/feature is intended for use with vLLM
• Documentation includes vLLM registration instructions

---

By submitting this PR, I confirm that:

• I have read and followed the contributing guidelines
• This is a community contribution and may have limited testing compared to officially-supported models
• The code follows best practices and is well-documented
• All required components listed above are included