Skip to content

Add ep_size to ParallelismConfig so FSDP2 can compose with expert parallelism - #4202

Open
YeonwooSung wants to merge 1 commit into
huggingface:mainfrom
YeonwooSung:feat/parallelism-config-ep-size
Open

Add ep_size to ParallelismConfig so FSDP2 can compose with expert parallelism#4202
YeonwooSung wants to merge 1 commit into
huggingface:mainfrom
YeonwooSung:feat/parallelism-config-ep-size

Conversation

@YeonwooSung

Copy link
Copy Markdown

What does this PR do?

ParallelismConfig had dp_replicate, dp_shard, tp, cp, and sp, but no expert-parallel axis. A 2-way FSDP 脳 4-way EP layout on 8 GPUs could not be expressed, and every downstream stack had to invent its own mesh.

This adds ep_size (env PARALLELISM_CONFIG_EP_SIZE, CLI --parallelism_config_ep_size, accelerate config when FSDP2 parallelism is enabled). Default is 1.

  • total_size and non_data_parallel_size include ep_size
  • "ep" is in non_dp_dim_names when ep_size > 1 (same batch on every EP rank, like TP/CP)
  • "ep" is not in fsdp_dim_names / dp_shard_cp_dim_names, so FSDP2 does not double-shard expert weights
  • Mesh order: dp_replicate, dp_shard, cp, sp, ep, tp (only enabled dims appear)
  • prepare_data_loader treats EP like TP/CP when mapping ranks onto data-parallel groups

Expert placement stays with the model. Accelerate only builds accelerator.torch_device_mesh["ep"].

cfg = ParallelismConfig(dp_shard_size=2, ep_size=4)
accelerator = Accelerator(
    fsdp_plugin=FullyShardedDataParallelPlugin(fsdp_version=2),
    parallelism_config=cfg,
)
# mesh["ep"] is the expert group; mesh["dp_shard"] is the FSDP group

Fixes #4200. Config/mesh half of the FSDP2 脳 EP work around #4178 / huggingface/transformers#48204 / huggingface/trl#6869.

Not in this PR: token-choice dispatch, FSDP2 wrap-heuristic changes, USP (CP+SP).

Tests

  • Mesh shapes for EP-only, FSDP+EP, FSDP+EP+TP, HSDP+EP
  • EP is not flattened into FSDP dims
  • Env PARALLELISM_CONFIG_EP_SIZE
  • ep_size=0 raises
  • Default ep_size=1 does not change existing CP/TP meshes
  • dp_replicate>1 + ep>1 is allowed
  • DataLoader rank mapping keeps one batch per EP group

Who can review?

@SunMarc @qgallouedec

Expert parallelism is a first-class mesh dimension. FSDP2 still shards
non-expert parameters on dp_shard(+cp); the model places expert weights
on the ep submesh. Accelerate does not implement token dispatch.

Fixes huggingface#4200
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add ep_size to ParallelismConfig so FSDP2 can compose with expert parallelism

1 participant