generated from amazon-archives/__template_MIT-0
-
Notifications
You must be signed in to change notification settings - Fork 1k
adding lambda-durable-functions-nodejs-calling-ecs #2959
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
dsiz
wants to merge
6
commits into
aws-samples:main
Choose a base branch
from
dsiz:dsiz-feature-lambda-durable-functions-nodejs-calling-ecs
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
cb41d6f
adding lambda-durable-functions-nodejs-calling-ecs
dsiz 8bd1679
Remove polling pattern, keep only callback approach for ECS integration
dsiz 7cb8038
Applied PR #2959 feedback and removed polling pattern and other concerns
dsiz dc9dd7b
Applied all PR #2959 feedback
dsiz 3f35723
Change ECS to Amazon ECS in title
dsiz 7d87964
Replace console.log with context.logger per PR feedback
dsiz File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,26 @@ | ||
| # Node modules | ||
| node_modules/ | ||
| package-lock.json | ||
|
|
||
| # SAM build artifacts | ||
| .aws-sam/ | ||
| samconfig.toml | ||
|
|
||
| # IDE | ||
| .vscode/ | ||
| .idea/ | ||
| *.swp | ||
| *.swo | ||
| *~ | ||
|
|
||
| # OS | ||
| .DS_Store | ||
| Thumbs.db | ||
|
|
||
| # Test outputs | ||
| response.json | ||
| *.log | ||
|
|
||
| # Environment | ||
| .env | ||
| .env.local |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,220 @@ | ||
| # AWS Lambda durable functions with Amazon ECS Integration | ||
|
|
||
| This pattern demonstrates how to use AWS Lambda durable functions to orchestrate long-running Amazon ECS Fargate tasks. The Lambda function can wait up to 24 hours for ECS task completion without incurring compute charges during the wait period. | ||
|
|
||
| **Important:** Please check the [AWS documentation](https://docs.aws.amazon.com/lambda/latest/dg/durable-functions.html) for regions currently supported by AWS Lambda durable functions. | ||
|
|
||
| Learn more about this pattern at Serverless Land Patterns: https://serverlessland.com/patterns/lambda-durable-functions-nodejs-calling-ecs | ||
|
|
||
| ## Architecture | ||
|
|
||
|  | ||
|
|
||
| The pattern uses Lambda durable functions with the callback pattern to orchestrate ECS Fargate tasks cost-effectively. | ||
|
|
||
| ### Workflow Steps | ||
|
|
||
| 1. **Lambda function invoked** with task parameters (message, processing time) | ||
| 2. **Durable function creates callback ID** using `context.waitForCallback()` | ||
| 3. **ECS Fargate task started** with callback ID passed as environment variable | ||
| 4. **Lambda function pauses** (no compute charges during wait) | ||
| 5. **ECS task processes workload** and logs progress to CloudWatch | ||
| 6. **ECS task completes** (in production, would call `SendDurableExecutionCallbackSuccess`) | ||
| 7. **Lambda function resumes** and returns result | ||
|
|
||
| ## Key Features | ||
|
|
||
| - ✅ **24-Hour Wait Time** - Can wait up to 24 hours for ECS task completion | ||
| - ✅ **No Compute Charges During Wait** - Function suspended during wait period | ||
| - ✅ **Callback Pattern** - ECS tasks call Lambda APIs directly to resume execution | ||
| - ✅ **CloudWatch Logs** - Full visibility into both Lambda and ECS execution | ||
| - ✅ **Generic Container** - Uses public Python image, easily replaceable | ||
| - ✅ **Fargate Serverless** - No EC2 instances to manage | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| * [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) installed and configured | ||
| * [AWS SAM CLI](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install.html) installed | ||
| * Node.js runtime (see [supported runtimes for durable functions](https://docs.aws.amazon.com/lambda/latest/dg/durable-supported-runtimes.html)) | ||
|
|
||
| ## Deployment | ||
|
|
||
| 1. Navigate to the pattern directory: | ||
| ```bash | ||
| cd lambda-durable-functions-nodejs-calling-ecs | ||
| ``` | ||
|
|
||
| 2. Build the SAM application: | ||
| ```bash | ||
| sam build | ||
| ``` | ||
|
|
||
| 3. Deploy the application: | ||
| ```bash | ||
| sam deploy --guided | ||
| ``` | ||
|
|
||
| During the guided deployment: | ||
| - Accept default values or customize as needed | ||
| - Allow SAM CLI to create IAM roles when prompted | ||
| - Note the function name from the outputs | ||
|
|
||
| 4. Note the `CallbackFunctionName` from the CloudFormation outputs | ||
|
|
||
| ## Testing | ||
|
|
||
| ### Test the Callback Pattern | ||
|
|
||
| Invoke the Lambda function with a test payload: | ||
|
|
||
| ```bash | ||
| aws lambda invoke \ | ||
| --function-name <CallbackFunctionName>:prod \ | ||
| --invocation-type Event \ | ||
| --payload '{"message":"Test ECS task","processingTime":8}' \ | ||
| --cli-binary-format raw-in-base64-out \ | ||
| response.json | ||
| ``` | ||
|
|
||
| **Note:** A qualified ARN (version or alias) is required for durable functions. See [invoking durable functions](https://docs.aws.amazon.com/lambda/latest/dg/durable-invoking.html#durable-invoking-qualified-arns). | ||
|
|
||
| ### Monitor Execution | ||
|
|
||
| Check Lambda logs: | ||
| ```bash | ||
| aws logs tail /aws/lambda/<CallbackFunctionName> --since 2m --follow | ||
| ``` | ||
|
|
||
| Check ECS task logs: | ||
| ```bash | ||
| aws logs tail /ecs/lambda-ecs-durable-demo --since 2m --follow | ||
| ``` | ||
|
|
||
| ### Expected Output | ||
|
|
||
| **Lambda Logs:** | ||
| ``` | ||
| Starting Lambda durable function - Callback Pattern | ||
| Callback ID created: <callback-id> | ||
| Starting ECS task with callback ID... | ||
| ECS task started: arn:aws:ecs:... | ||
| ``` | ||
|
|
||
| **ECS Logs:** | ||
| ``` | ||
| === ECS Task Started === | ||
| Callback ID: <callback-id> | ||
| Message: Test ECS task | ||
| Processing Time: 8 seconds | ||
| Simulating work... | ||
| === Task Completed Successfully === | ||
| Result: {"status":"completed","message":"Processed: Test ECS task"} | ||
| Note: In production, call Lambda SendDurableExecutionCallbackSuccess API here | ||
| ``` | ||
|
|
||
| ## How It Works | ||
|
|
||
| ### Lambda durable function (Node.js) | ||
|
|
||
| The Lambda function uses the `@aws/durable-execution-sdk-js` package: | ||
|
|
||
| ```javascript | ||
| const { withDurableExecution } = require('@aws/durable-execution-sdk-js'); | ||
|
|
||
| exports.handler = withDurableExecution(async (event, context) => { | ||
| // Create callback and start ECS task | ||
| const result = await context.waitForCallback( | ||
| 'ecs-task-callback', | ||
| async (callbackId) => { | ||
| // Start ECS task with callback ID | ||
| const response = await ecs.send(new RunTaskCommand({ | ||
| // ... pass callbackId as environment variable | ||
| })); | ||
| }, | ||
| { timeout: { hours: 1 } } | ||
| ); | ||
|
|
||
| return result; | ||
| }); | ||
| ``` | ||
|
|
||
| ### ECS Task (Python) | ||
|
|
||
| The ECS container receives the callback ID and processes the workload. In production, it would call the Lambda API: | ||
|
|
||
| ```bash | ||
| aws lambda send-durable-execution-callback-success \ | ||
| --callback-id $CALLBACK_ID \ | ||
| --result '{"status":"completed","data":"..."}' | ||
| ``` | ||
|
|
||
| ### Key Configuration | ||
|
|
||
| **Lambda Function:** | ||
| - Runtime: `nodejs22.x` (see [supported runtimes](https://docs.aws.amazon.com/lambda/latest/dg/durable-supported-runtimes.html)) | ||
| - `AutoPublishAlias: prod` | ||
| - `DurableConfig` with execution timeout and retention period | ||
|
|
||
| **ECS Task:** | ||
| - Launch type: `FARGATE` | ||
| - Public subnet with `assignPublicIp: ENABLED` | ||
| - Container image: `public.ecr.aws/docker/library/python:3.12-alpine` | ||
| - CloudWatch Logs enabled | ||
|
|
||
| ## Customization | ||
|
|
||
| ### Replace the ECS Container | ||
|
|
||
| The pattern uses a generic Python container for demonstration. To use your own container: | ||
|
|
||
| 1. Update the `Image` in the `ECSTaskDefinition` resource | ||
| 2. Ensure your container: | ||
| - Reads the `CALLBACK_ID` environment variable | ||
| - Calls `aws lambda send-durable-execution-callback-success` on completion | ||
| - Calls `aws lambda send-durable-execution-callback-failure` on error | ||
|
|
||
| ### Adjust Timeouts | ||
|
|
||
| Modify the durable function timeout in `template.yaml`: | ||
|
|
||
| ```yaml | ||
| DurableConfig: | ||
| ExecutionTimeout: 86400 # 24 hours in seconds | ||
| RetentionPeriodInDays: 7 | ||
| ``` | ||
|
|
||
| And the callback timeout in the handler: | ||
|
|
||
| ```javascript | ||
| context.waitForCallback('ecs-task-callback', async (callbackId) => { | ||
| // ... | ||
| }, { | ||
| timeout: { hours: 24 }, // Maximum wait time | ||
| heartbeatTimeout: { minutes: 5 } // Optional heartbeat | ||
| }) | ||
| ``` | ||
|
|
||
| ## Cleanup | ||
|
|
||
| Delete the stack: | ||
|
|
||
| ```bash | ||
| sam delete | ||
| ``` | ||
|
|
||
| ## Cost Considerations | ||
|
|
||
| - **Lambda:** Charged only for active execution time, not during wait periods | ||
| - **ECS Fargate:** Charged per vCPU and memory per second while tasks run | ||
| - **CloudWatch Logs:** Charged for log ingestion and storage | ||
| - **VPC:** NAT Gateway charges if using private subnets (this pattern uses public subnets) | ||
|
|
||
| ## Additional Resources | ||
|
|
||
| - [AWS Lambda durable functions Documentation](https://docs.aws.amazon.com/lambda/latest/dg/durable-functions.html) | ||
| - [Amazon ECS on AWS Fargate](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/AWS_Fargate.html) | ||
| - [AWS SAM Documentation](https://docs.aws.amazon.com/serverless-application-model/) | ||
|
|
||
| --- | ||
|
|
||
| © 2026 Amazon Web Services, Inc. or its affiliates. All Rights Reserved. | ||
68 changes: 68 additions & 0 deletions
68
lambda-durable-functions-nodejs-calling-ecs/example-pattern.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,68 @@ | ||
| { | ||
| "title": "AWS Lambda durable functions in NodeJS calling Amazon ECS", | ||
| "description": "Orchestrate long-running ECS Fargate tasks using Lambda durable functions with callback pattern", | ||
| "language": "Node.js", | ||
| "level": "200", | ||
| "framework": "AWS SAM", | ||
| "introBox": { | ||
| "headline": "How it works", | ||
| "text": [ | ||
| "This pattern demonstrates how to use AWS Lambda durable functions to orchestrate Amazon ECS Fargate tasks that can run for up to 24 hours.", | ||
| "The Lambda function creates a callback ID, starts an ECS task with that ID, and then automatically suspends execution without incurring compute charges during the wait period.", | ||
| "When the ECS task completes, it calls the Lambda SendDurableExecutionCallbackSuccess API to resume the durable function and return results.", | ||
| "This pattern is ideal for long-running batch jobs, data processing, ML training, or any workload that exceeds Lambda's 15-minute timeout." | ||
| ] | ||
| }, | ||
| "gitHub": { | ||
| "template": { | ||
| "repoURL": "https://github.com/aws-samples/serverless-patterns/tree/main/lambda-durable-functions-nodejs-calling-ecs", | ||
| "templateURL": "serverless-patterns/lambda-durable-functions-nodejs-calling-ecs", | ||
| "projectFolder": "lambda-durable-functions-nodejs-calling-ecs", | ||
| "templateFile": "template.yaml" | ||
| } | ||
| }, | ||
| "resources": { | ||
| "bullets": [ | ||
| { | ||
| "text": "AWS Lambda durable functions Documentation", | ||
| "link": "https://docs.aws.amazon.com/lambda/latest/dg/durable-functions.html" | ||
| }, | ||
| { | ||
| "text": "Amazon ECS on AWS Fargate", | ||
| "link": "https://docs.aws.amazon.com/AmazonECS/latest/developerguide/AWS_Fargate.html" | ||
| }, | ||
| { | ||
| "text": "Lambda Durable Execution SDK for JavaScript", | ||
| "link": "https://www.npmjs.com/package/@aws/durable-execution-sdk-js" | ||
| }, | ||
| { | ||
| "text": "SendDurableExecutionCallbackSuccess API", | ||
| "link": "https://docs.aws.amazon.com/lambda/latest/api/API_SendDurableExecutionCallbackSuccess.html" | ||
| } | ||
| ] | ||
| }, | ||
| "deploy": { | ||
| "text": [ | ||
| "sam build", | ||
| "sam deploy --guided" | ||
| ] | ||
| }, | ||
| "testing": { | ||
| "text": [ | ||
| "See the GitHub repo for detailed testing instructions." | ||
| ] | ||
| }, | ||
| "cleanup": { | ||
| "text": [ | ||
| "<code>sam delete</code>" | ||
| ] | ||
| }, | ||
| "authors": [ | ||
| { | ||
| "name": "Surya Sai D", | ||
| "image": "", | ||
| "bio": "Surya works as a Technical Account Manager at AWS. He is an expert in Serverless frameworks and Event Driven Architectures. Surya is also passionate on technical writing and has contributed to AWS blogs and other Open Source Content.", | ||
| "linkedin": "surya-sai-d-64920416a" | ||
| } | ||
| ] | ||
| } |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would remove this as you otherwise need to expand the points below. For example Lambda durable functions still charges for durable operations and persisted state.