Skip to content

NVIDIA Dynamo

Dynamo is a new modular inference framework designed for serving large language models (LLMs) in multi-node distributed environments. It enables seamless scaling of inference workloads across GPU nodes and the dynamic allocation of GPU workers to address traffic bottlenecks at various stages of the model pipeline.

This GitHub organization hosts repositories for Dynamo's core components and integrations, including:

Core Framework

  • Distributed inference runtime with Rust-based orchestration
  • Python bindings for workflow customization
  • Multi-GPU/multi-node serving capabilities

LLM Optimized Components

  • Disaggregated Serving Engine: Decoupling of prefill and decode to optimize for throughput at latency SLOs
  • Intelligent Routing System: Prefix-based and load-aware request distribution
  • KV Cache Management: Distributed KV Cache management

NVIDIA Optimized Transfer Library (NIXL)

  • Abstracts memory of heterogeneous devices, i.e., CPU, GPU, storage, and enables most efficient and low-latency communication among them
  • Integrates with distributed inference servers such as Dynamo. This library will target distributed inference communication patterns to effectively transfer the KV cache in disaggregated LLM serving platforms.

Getting Started

To learn more about NVIDIA Dynamo Inference Serving Platform, please refer to the Dynamo developer page and read our Quickstart Guide for container setup and basic workflows.

Documentation

User documentation on Dynamo features, APIs, and architecture is located in the Dynamo documents folder on GitHub.

FAQ

Consult the Dynamo FAQ Guide for frequently asked questions and answers.

Contribution & Support

  • Follow Contribution Guidelines
  • Report issues via GitHub Discussions
  • Enterprise support available through NVIDIA AI Enterprise

License

Apache 2.0 licensed with third-party attributions documented in each repository.

Note

APIs and components may evolve based on community feedback

Pinned Loading

  1. dynamo dynamo Public

    A Datacenter Scale Distributed Inference Serving Framework

    Rust 8.5k 1.9k

  2. nixl nixl Public

    NVIDIA Inference Xfer Library (NIXL)

    C++ 1.4k 536

  3. aiconfigurator aiconfigurator Public

    Offline optimization of your disaggregated Dynamo graph

    Python 469 175

  4. aiperf aiperf Public

    AIPerf is a comprehensive benchmarking tool that measures the performance of generative AI models served by your preferred inference solution.

    Python 900 274

  5. grove grove Public

    Kubernetes enhancements for Network Topology Aware Gang Scheduling & Autoscaling

    Go 298 122

  6. modelexpress modelexpress Public

    Model Express is a Rust-based component meant to be placed next to existing model inference systems to speed up their startup times and improve overall performance.

    Python 205 120

Repositories

Showing 10 of 19 repositories
  • dynamo Public

    A Datacenter Scale Distributed Inference Serving Framework

    ai-dynamo/dynamo's past year of commit activity
    Rust 8,544 1,887 929 (1 issue needs help) 1,213 Updated Nov 22, 2026
  • snapshot Public

    Snapshot gets GPU pods ready in seconds instead of minutes — by restoring a fully initialized GPU worker instead of starting one from scratch. Kubernetes-native checkpoint & restore that runs alongside your existing stack.

    ai-dynamo/snapshot's past year of commit activity
    Go 177 Apache-2.0 60 142 (4 issues need help) 106 Updated Nov 22, 2026
  • ai-dynamo/frontend-crates's past year of commit activity
    Rust 31 Apache-2.0 94 33 65 Updated Nov 22, 2026
  • grove Public

    Kubernetes enhancements for Network Topology Aware Gang Scheduling & Autoscaling

    ai-dynamo/grove's past year of commit activity
    Go 298 Apache-2.0 122 82 (9 issues need help) 80 Updated Nov 22, 2026
  • aisim Public

    AISim predicts LLM serving behavior and searches for strong deployment configurations offline, without bringing up a GPU serving cluster

    ai-dynamo/aisim's past year of commit activity
    Python 139 67 37 95 Updated Nov 21, 2026
  • modelexpress Public

    Model Express is a Rust-based component meant to be placed next to existing model inference systems to speed up their startup times and improve overall performance.

    ai-dynamo/modelexpress's past year of commit activity
    Python 205 Apache-2.0 120 59 73 Updated Nov 21, 2026
  • kvcr Public
    ai-dynamo/kvcr's past year of commit activity
    Python 76 Apache-2.0 34 5 24 Updated Nov 22, 2026
  • nixl Public

    NVIDIA Inference Xfer Library (NIXL)

    ai-dynamo/nixl's past year of commit activity
    C++ 1,373 536 79 198 Updated Nov 21, 2026
  • velo Public
    ai-dynamo/velo's past year of commit activity
    Rust 13 Apache-2.0 6 1 8 Updated Nov 21, 2026
  • aitune Public

    NVIDIA AITune is an inference toolkit designed for tuning and deploying Deep Learning models with a focus on NVIDIA GPUs.

    ai-dynamo/aitune's past year of commit activity
    Python 290 Apache-2.0 38 2 23 Updated Nov 21, 2026