diff --git a/README.md b/README.md index 6029a190..b0032cf4 100644 --- a/README.md +++ b/README.md @@ -26,11 +26,13 @@ Below are blueprints of supported models along with their documentation. | [**SGLang disaggregated P/D inference**](scripts/sglang_disagg/README.MD) | Distributed Inference P/D disaggregation with SGLang (MoRI IO, Mooncake) | Llama-3.1-8B, Qwen3-32B, Llama-3.3-70B-FP8, Llama-3.1-405B-FP8, Mixtral-8x7B, DeepSeek-V3, DeepSeek-R1 | | [**SGLang disaggregated P/D inference with WideEP/LargeEP**](scripts/sglang_disagg/README.MD) | Distributed Inference P/D disaggregation with SGLang with WideEP/LargeEP | DeepSeek-V3, DeepSeek-R1 | | [**KVCache Transfer Bench**](scripts/kvcache_transfer_bench/README.md) | Inter-node Transfer Benchmark | no specific models | +| [**Primus pretrain**](#primus-pretrain) | LLM pretraining through the [Primus](https://github.com/AMD-AGI/Primus) launcher (Megatron, TorchTitan, MaxText, and other backends) | Config-driven; see `scripts/Primus/examples/` | ## Table of Contents - [Prerequisites](#prerequisites) - [Quick Start](#quick-start) +- [Primus pretrain](#primus-pretrain) - [Usage Guide](#usage-guide) - [Running Models](#running-models) - [Tag Functionality](#tag-functionality) @@ -52,11 +54,15 @@ Below are blueprints of supported models along with their documentation. ## Quick Start -1. **Clone the repository**: +1. **Clone the repository** (include the Primus submodule if you use [Primus pretrain](#primus-pretrain)): ```bash - git clone + git clone --recurse-submodules cd MAD ``` + If you already cloned without submodules, initialize Primus with: + ```bash + git submodule update --init scripts/Primus + ``` 2. **Install dependencies**: ```bash @@ -68,7 +74,20 @@ Below are blueprints of supported models along with their documentation. madengine run --tags pyt_huggingface_bert ``` +## Primus pretrain + +MAD integrates [AMD-AGI/Primus](https://github.com/AMD-AGI/Primus) as a Git submodule at **`scripts/Primus`**. The **`primus_pretrain`** entry in `models.json` uses **`docker/primus.ubuntu.amd.Dockerfile`** and **`scripts/primus_pretrain/`** (`run.sh` wraps Primus `examples/run_pretrain.sh`, copies logs under the madengine run directory, and writes **`primus_perf_output.csv`** for throughput / TFLOPs / MFU when logs include those metrics). + +- **Run with madengine** (tags include `primus`, `training`, `pretrain`): + ```bash + madengine run --tags primus_pretrain + ``` +- **Choose a config**: pass Primus YAML via script args, e.g. `--config_path examples/torchtitan/configs/MI300X/your_config.yaml` (path is relative to the Primus repo root). For SLURM or Kubernetes, you can set **`PRIMUS_CONFIG_PATH`** to the same path instead. +- **Hugging Face–backed configs**: set **`HF_TOKEN`**, or **`MAD_SECRET_HFTOKEN`** (madengine v2 can inject the latter via `additional_context.docker_env_vars`). +- **Docker build (madengine path)**: build from the **repository root** so `COPY scripts/Primus/` in `docker/primus.ubuntu.amd.Dockerfile` resolves; `madengine build` uses repo context for Dockerfiles whose path contains `primus`. +- **Optional discovery (madengine only)**: `scripts/primus_pretrain/get_models_json.py` can expose individual Primus example YAMLs as separate models when used with madengine’s discover-models flow. The legacy `tools/run_models.py` path is not supported for `primus_pretrain`, because it builds with `./docker` as context and will not satisfy this Dockerfile’s repo-root `COPY scripts/Primus/` requirement. +For more detail, see comments in `docker/primus.ubuntu.amd.Dockerfile` and `scripts/primus_pretrain/run.sh`. ## Usage Guide diff --git a/scripts/Primus b/scripts/Primus index 30cf4514..e50a78b0 160000 --- a/scripts/Primus +++ b/scripts/Primus @@ -1 +1 @@ -Subproject commit 30cf451497f24aebb5e031e8d4850d5f449433b4 +Subproject commit e50a78b09599dac4d3f5f404d8551b54b5b5d83b