[examples] Add WideEP DP group fault tolerance example#54
Open
jeffreywang-anyscale wants to merge 16 commits intomainfrom
Open
[examples] Add WideEP DP group fault tolerance example#54jeffreywang-anyscale wants to merge 16 commits intomainfrom
jeffreywang-anyscale wants to merge 16 commits intomainfrom
Conversation
Signed-off-by: Jeffrey Wang <jeffreywang@anyscale.com>
g6.12xlarge (4x L4 GPUs) has 192 GiB RAM, not 32 GiB. The incorrect value caused free pod shape validation to fail on deployment. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Robert Nishihara <rkn@anyscale.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Robert Nishihara <rkn@anyscale.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Robert Nishihara <rkn@anyscale.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Robert Nishihara <rkn@anyscale.com>
- Rewrite README to match the pattern of other examples (Install CLI, Clone, Deploy, Query, Understanding, Shutdown) - Split into two clear demos: autoscaling service and fault tolerance job - Add job.yaml so fault_tolerance_demo.py can be run as an Anyscale job from a laptop without needing direct cluster access Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Robert Nishihara <rkn@anyscale.com>
Use the deployed service + console terminal to demonstrate fault tolerance, which is simpler and more visual than submitting a job. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Robert Nishihara <rkn@anyscale.com>
Signed-off-by: jeffreywang <jeffreywang@anyscale.com>
| # Deploy with: anyscale service deploy -f autoscaling/service.yaml | ||
|
|
||
| name: wide-ep-autoscaling | ||
| image_uri: anyscale/ray-llm:nightly-py311-cu128 |
Author
There was a problem hiding this comment.
we can replace this with 2.55 image once it's released next week
Signed-off-by: Jeffrey Wang <jeffreywang@anyscale.com>
Signed-off-by: Jeffrey Wang <jeffreywang@anyscale.com>
Author
|
Deepseek DP group fault tolerance ( 
|
| env_vars: | ||
| VLLM_DISABLE_COMPILE_CACHE: "1" | ||
|
|
||
| - name: simulate-fault |
Author
There was a problem hiding this comment.
unify to the same service and put both endpoints under the same application
Author
There was a problem hiding this comment.
flatten the directories
| # | ||
| # Deploy with: anyscale service deploy -f fault_tolerance/service.yaml | ||
|
|
||
| name: wide-ep-fault-tolerance |
Author
There was a problem hiding this comment.
test different clouds
Signed-off-by: Jeffrey Wang <jeffreywang@anyscale.com>
Signed-off-by: Jeffrey Wang <jeffreywang@anyscale.com>
Signed-off-by: Jeffrey Wang <jeffreywang@anyscale.com>
Signed-off-by: Jeffrey Wang <jeffreywang@anyscale.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add examples for demonstrating WideEP fault tolerance and autoscaling capabilities with Anyscale services.
Tests
Verified by prompting CC to "Read README.md and run demo 1" and "Run demo 2" and was able to observe expected replica changes from the Ray Serve dashboard.