Skip to content

fix(minimax): repair HF parity reference - map legacy rotary_dim to partial_rotary_factor - #48486

Open
dajiaohuang wants to merge 3 commits into
huggingface:mainfrom
dajiaohuang:fix/48241-minimax-rope-bug
Open

fix(minimax): repair HF parity reference - map legacy rotary_dim to partial_rotary_factor#48486
dajiaohuang wants to merge 3 commits into
huggingface:mainfrom
dajiaohuang:fix/48241-minimax-rope-bug

Conversation

@dajiaohuang

@dajiaohuang dajiaohuang commented Sep 2, 2026

Copy link
Copy Markdown

CPU CI GPU run-slow

What does this PR do?

Fixes #48241.

Released MiniMaxAI/MiniMax-M2* checkpoints are trained with partial RoPE (rotary_dim: 64 of head_dim: 128) and express it through a legacy rotary_dim config field; their original remote-code config derived partial_rotary_factor = rotary_dim / head_dim in __init__. The in-tree MiniMaxM2Config keeps rotary_dim as an opaque extra attribute, so rope_parameters never receives a partial factor and MiniMaxM2RotaryEmbedding silently rotates the full head dimension with a full-width frequency ladder — wrong frequency spacing and wrong rotate-half pairing in every attention layer (measured: mean KL 13.1 / cosine 0.02 vs a faithful reference on MiniMax-M2.7 over 2,048 tokens; details in the issue).

This maps the legacy field in a convert_rope_params_to_dict override on MiniMaxM2Config (in modular_minimax_m2.py), following the GPTNeoXConfig rotary_pct precedent:

  • MiniMaxM2Config(rotary_dim=64)rope_parameters["partial_rotary_factor"] == 0.5, inv_freq length 32;
  • an explicit partial_rotary_factor in rope_parameters still wins;
  • configs without rotary_dim are unaffected.

Before submitting

Who can review?

@ArthurZucker @Cyrilvallez

@dajiaohuang

Copy link
Copy Markdown
Author

Fixed the missed modular conversion output in commit 26c0e74. The PR's new convert_rope_params_to_dict method is now present in generated configuration_minimax_m2.py, resolving the repository consistency diff and the KeyError in test_legacy_rotary_dim_maps_to_partial_rotary_factor. Validation: Python compileall passed and git diff --check passed.

dajiaohuang and others added 2 commits September 4, 2026 14:42
…elope

Released MiniMaxAI/MiniMax-M2* checkpoints are trained with partial RoPE
(rotary_dim 64 of head_dim 128) and express it through a legacy rotary_dim
config field; their original remote-code config derived partial_rotary_factor
= rotary_dim / head_dim in __init__. The in-tree MiniMaxM2Config keeps
rotary_dim as an opaque extra attribute, so rope_parameters never receives
a partial factor and MiniMaxM2RotaryEmbedding silently rotates the full head
dimension with a full-width frequency ladder.

Map the legacy field in convert_rope_params_to_dict, following the
GPTNeoXConfig rotary_pct precedent; an explicit partial_rotary_factor still
takes priority and configs without rotary_dim are unaffected.

Fixes huggingface#48241
@dajiaohuang
dajiaohuang force-pushed the fix/48241-minimax-rope-bug branch from 26c0e74 to e434db8 Compare September 4, 2026 06:46
@dajiaohuang

Copy link
Copy Markdown
Author

Follow-up after the latest consistency run: the failure also compared against upstream main at c119ec3, which had advanced beyond the fork's stale main and introduced inherited conversion output in minimax_m3_vl/step3p7. I rebased the existing branch onto current upstream main and regenerated the MiniMax M2 config output, including exact converter formatting, in commit e434db8. The branch now contains only the modular source, matching generated configuration, and focused test changes. The earlier CI failure was reproduced from job 100931027864.

@dajiaohuang

Copy link
Copy Markdown
Author

The latest consistency failure (run 33845763052 / job 100937312164) was from generated descendants of MiniMaxM2Config: minimax_m3_vl and step3p7 were missing the inherited converter override, while the MiniMaxM2 generated file itself was correct. Regenerated both configuration files on top of upstream main and pushed as a20b527. The focused modular conversion check now passes with UTF-8 enabled.

@github-actions

github-actions Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

[For maintainers] Suggested jobs to run (before merge)

run-slow: minimax_m2, minimax_m3_vl, step3p7

@github-actions

github-actions Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

CI recap

Dashboard: View test results in Grafana
Latest run: 33845763052:2
Result: success | Jobs: 8 | Tests: 367 | Failures: 1 | Duration: 3m 42s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MiniMaxM2 silently applies full-head RoPE: config drops the checkpoints' legacy rotary_dim field

1 participant