Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions lambda-durable-functions-nodejs-calling-ecs/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# Node modules
node_modules/
package-lock.json

# SAM build artifacts
.aws-sam/
samconfig.toml

# IDE
.vscode/
.idea/
*.swp
*.swo
*~

# OS
.DS_Store
Thumbs.db

# Test outputs
response.json
*.log

# Environment
.env
.env.local
220 changes: 220 additions & 0 deletions lambda-durable-functions-nodejs-calling-ecs/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,220 @@
# AWS Lambda durable functions with Amazon ECS Integration

This pattern demonstrates how to use AWS Lambda durable functions to orchestrate long-running Amazon ECS Fargate tasks. The Lambda function can wait up to 24 hours for ECS task completion without incurring compute charges during the wait period.

**Important:** Please check the [AWS documentation](https://docs.aws.amazon.com/lambda/latest/dg/durable-functions.html) for regions currently supported by AWS Lambda durable functions.

Learn more about this pattern at Serverless Land Patterns: https://serverlessland.com/patterns/lambda-durable-functions-nodejs-calling-ecs

## Architecture

![Architecture Diagram](architecture.png)

The pattern uses Lambda durable functions with the callback pattern to orchestrate ECS Fargate tasks cost-effectively.

### Workflow Steps

1. **Lambda function invoked** with task parameters (message, processing time)
2. **Durable function creates callback ID** using `context.waitForCallback()`
3. **ECS Fargate task started** with callback ID passed as environment variable
4. **Lambda function pauses** (no compute charges during wait)
5. **ECS task processes workload** and logs progress to CloudWatch
6. **ECS task completes** (in production, would call `SendDurableExecutionCallbackSuccess`)
7. **Lambda function resumes** and returns result

## Key Features

- ✅ **24-Hour Wait Time** - Can wait up to 24 hours for ECS task completion
- ✅ **No Compute Charges During Wait** - Function suspended during wait period
- ✅ **Callback Pattern** - ECS tasks call Lambda APIs directly to resume execution
- ✅ **CloudWatch Logs** - Full visibility into both Lambda and ECS execution
- ✅ **Generic Container** - Uses public Python image, easily replaceable
- ✅ **Fargate Serverless** - No EC2 instances to manage

## Prerequisites

* [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) installed and configured
* [AWS SAM CLI](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install.html) installed
* Node.js runtime (see [supported runtimes for durable functions](https://docs.aws.amazon.com/lambda/latest/dg/durable-supported-runtimes.html))

## Deployment

1. Navigate to the pattern directory:
```bash
cd lambda-durable-functions-nodejs-calling-ecs
```

2. Build the SAM application:
```bash
sam build
```

3. Deploy the application:
```bash
sam deploy --guided
```

During the guided deployment:
- Accept default values or customize as needed
- Allow SAM CLI to create IAM roles when prompted
- Note the function name from the outputs

4. Note the `CallbackFunctionName` from the CloudFormation outputs

## Testing

### Test the Callback Pattern

Invoke the Lambda function with a test payload:

```bash
aws lambda invoke \
--function-name <CallbackFunctionName>:prod \
--invocation-type Event \
--payload '{"message":"Test ECS task","processingTime":8}' \
--cli-binary-format raw-in-base64-out \
response.json
```

**Note:** A qualified ARN (version or alias) is required for durable functions. See [invoking durable functions](https://docs.aws.amazon.com/lambda/latest/dg/durable-invoking.html#durable-invoking-qualified-arns).

### Monitor Execution

Check Lambda logs:
```bash
aws logs tail /aws/lambda/<CallbackFunctionName> --since 2m --follow
```

Check ECS task logs:
```bash
aws logs tail /ecs/lambda-ecs-durable-demo --since 2m --follow
```

### Expected Output

**Lambda Logs:**
```
Starting Lambda durable function - Callback Pattern
Callback ID created: <callback-id>
Starting ECS task with callback ID...
ECS task started: arn:aws:ecs:...
```

**ECS Logs:**
```
=== ECS Task Started ===
Callback ID: <callback-id>
Message: Test ECS task
Processing Time: 8 seconds
Simulating work...
=== Task Completed Successfully ===
Result: {"status":"completed","message":"Processed: Test ECS task"}
Note: In production, call Lambda SendDurableExecutionCallbackSuccess API here
```

## How It Works

### Lambda durable function (Node.js)

The Lambda function uses the `@aws/durable-execution-sdk-js` package:

```javascript
const { withDurableExecution } = require('@aws/durable-execution-sdk-js');

exports.handler = withDurableExecution(async (event, context) => {
// Create callback and start ECS task
const result = await context.waitForCallback(
'ecs-task-callback',
async (callbackId) => {
// Start ECS task with callback ID
const response = await ecs.send(new RunTaskCommand({
// ... pass callbackId as environment variable
}));
},
{ timeout: { hours: 1 } }
);

return result;
});
```

### ECS Task (Python)

The ECS container receives the callback ID and processes the workload. In production, it would call the Lambda API:

```bash
aws lambda send-durable-execution-callback-success \
--callback-id $CALLBACK_ID \
--result '{"status":"completed","data":"..."}'
```

### Key Configuration

**Lambda Function:**
- Runtime: `nodejs22.x` (see [supported runtimes](https://docs.aws.amazon.com/lambda/latest/dg/durable-supported-runtimes.html))
- `AutoPublishAlias: prod`
- `DurableConfig` with execution timeout and retention period

**ECS Task:**
- Launch type: `FARGATE`
- Public subnet with `assignPublicIp: ENABLED`
- Container image: `public.ecr.aws/docker/library/python:3.12-alpine`
- CloudWatch Logs enabled

## Customization

### Replace the ECS Container

The pattern uses a generic Python container for demonstration. To use your own container:

1. Update the `Image` in the `ECSTaskDefinition` resource
2. Ensure your container:
- Reads the `CALLBACK_ID` environment variable
- Calls `aws lambda send-durable-execution-callback-success` on completion
- Calls `aws lambda send-durable-execution-callback-failure` on error

### Adjust Timeouts

Modify the durable function timeout in `template.yaml`:

```yaml
DurableConfig:
ExecutionTimeout: 86400 # 24 hours in seconds
RetentionPeriodInDays: 7
```

And the callback timeout in the handler:

```javascript
context.waitForCallback('ecs-task-callback', async (callbackId) => {
// ...
}, {
timeout: { hours: 24 }, // Maximum wait time
heartbeatTimeout: { minutes: 5 } // Optional heartbeat
})
```

## Cleanup

Delete the stack:

```bash
sam delete
```

## Cost Considerations
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would remove this as you otherwise need to expand the points below. For example Lambda durable functions still charges for durable operations and persisted state.


- **Lambda:** Charged only for active execution time, not during wait periods
- **ECS Fargate:** Charged per vCPU and memory per second while tasks run
- **CloudWatch Logs:** Charged for log ingestion and storage
- **VPC:** NAT Gateway charges if using private subnets (this pattern uses public subnets)

## Additional Resources

- [AWS Lambda durable functions Documentation](https://docs.aws.amazon.com/lambda/latest/dg/durable-functions.html)
- [Amazon ECS on AWS Fargate](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/AWS_Fargate.html)
- [AWS SAM Documentation](https://docs.aws.amazon.com/serverless-application-model/)

---

© 2026 Amazon Web Services, Inc. or its affiliates. All Rights Reserved.
68 changes: 68 additions & 0 deletions lambda-durable-functions-nodejs-calling-ecs/example-pattern.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
{
"title": "AWS Lambda durable functions in NodeJS calling Amazon ECS",
"description": "Orchestrate long-running ECS Fargate tasks using Lambda durable functions with callback pattern",
"language": "Node.js",
"level": "200",
"framework": "AWS SAM",
"introBox": {
"headline": "How it works",
"text": [
"This pattern demonstrates how to use AWS Lambda durable functions to orchestrate Amazon ECS Fargate tasks that can run for up to 24 hours.",
"The Lambda function creates a callback ID, starts an ECS task with that ID, and then automatically suspends execution without incurring compute charges during the wait period.",
"When the ECS task completes, it calls the Lambda SendDurableExecutionCallbackSuccess API to resume the durable function and return results.",
"This pattern is ideal for long-running batch jobs, data processing, ML training, or any workload that exceeds Lambda's 15-minute timeout."
]
},
"gitHub": {
"template": {
"repoURL": "https://github.com/aws-samples/serverless-patterns/tree/main/lambda-durable-functions-nodejs-calling-ecs",
"templateURL": "serverless-patterns/lambda-durable-functions-nodejs-calling-ecs",
"projectFolder": "lambda-durable-functions-nodejs-calling-ecs",
"templateFile": "template.yaml"
}
},
"resources": {
"bullets": [
{
"text": "AWS Lambda durable functions Documentation",
"link": "https://docs.aws.amazon.com/lambda/latest/dg/durable-functions.html"
},
{
"text": "Amazon ECS on AWS Fargate",
"link": "https://docs.aws.amazon.com/AmazonECS/latest/developerguide/AWS_Fargate.html"
},
{
"text": "Lambda Durable Execution SDK for JavaScript",
"link": "https://www.npmjs.com/package/@aws/durable-execution-sdk-js"
},
{
"text": "SendDurableExecutionCallbackSuccess API",
"link": "https://docs.aws.amazon.com/lambda/latest/api/API_SendDurableExecutionCallbackSuccess.html"
}
]
},
"deploy": {
"text": [
"sam build",
"sam deploy --guided"
]
},
"testing": {
"text": [
"See the GitHub repo for detailed testing instructions."
]
},
"cleanup": {
"text": [
"<code>sam delete</code>"
]
},
"authors": [
{
"name": "Surya Sai D",
"image": "",
"bio": "Surya works as a Technical Account Manager at AWS. He is an expert in Serverless frameworks and Event Driven Architectures. Surya is also passionate on technical writing and has contributed to AWS blogs and other Open Source Content.",
"linkedin": "surya-sai-d-64920416a"
}
]
}
Loading