New serverless pattern - eventbridge-pipes-sqs-to-dynamodb

eventbridge-pipes-sqs-to-dynamodb/README.md

@@ -0,0 +1,92 @@
+# Amazon SQS to Amazon DynamoDB using Amazon EventBridge Pipes with Amazon API Gateway and CDK/Python
+
+This pattern will send messages from an SQS queue to a DynamoDB table via API Gateway using EventBridge Pipes.
+
+![Architecture](./architecture.png)
+
+Learn more about this pattern at Serverless Land Patterns: [https://serverlessland.com/patterns/eventbridge-pipes-sqs-to-dynamodb](https://serverlessland.com/patterns/eventbridge-pipes-sqs-to-dynamodb)
+
+Important: this application uses various AWS services and there are costs associated with these services after the Free Tier usage - please see the [AWS Pricing page](https://aws.amazon.com/pricing/) for details. You are responsible for any AWS costs incurred. No warranty is implied in this example.
+
+## Requirements
+
+- [Create an AWS account](https://portal.aws.amazon.com/gp/aws/developer/registration/index.html) if you do not already have one and log in. The IAM user that you use must have sufficient permissions to make necessary AWS service calls and manage AWS resources.
+- [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) installed and configured
+- [Git Installed](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)
+- [AWS CDK](https://docs.aws.amazon.com/cdk/latest/guide/cli.html) installed and configured
+
+## Deployment Instructions
+
+1. Create a new directory, navigate to that directory in a terminal and clone the GitHub repository:
+   ```bash
+   git clone https://github.com/aws-samples/serverless-patterns
+   ```
+2. Change directory to the pattern directory:
+   ```bash
+   cd serverless-patterns/eventbridge-pipes-sqs-to-dynamodb
+   ```
+3. To manually create a virtualenv on MacOS and Linux:
+    ```bash
+    $ python3 -m venv .venv
+    ```
+4. After the init process completes and the virtualenv is created, you can use the following
+step to activate your virtualenv.
+    ```bash
+    $ source .venv/bin/activate
+    ```
+5. If you are a Windows platform, you would activate the virtualenv like this:
+    ```bash
+    % .venv\Scripts\activate.bat
+    ```
+6. Once the virtualenv is activated, you can install the required dependencies.
+    ```bash
+    $ pip install -r requirements.txt
+    ```
+7. To deploy the application:
+    ```bash
+    $ cdk deploy
+    ```
+
+## How it works
+
+This template will create an SQS queue, EventBridge Pipe, API Gateway and a DynamoDB table.
+
+Messages sent to the SQS queue are polled by EventBridge Pipe. EventBridge Pipe processes the messages and sends them to API Gateway endpoint. API Gateway transforms the message and writes the data to DynamoDB table using direct integration.
+
+## Testing
+
+Once this stack is deployed in your AWS account, copy the SQS queue name value from the output.
+
+Alternatively, retrieve the queue URL using AWS CLI:
+```sh
+aws sqs list-queues --query 'QueueUrls[?contains(@, `EntryPointToEventbridgePipe`)]'
+```
+
+Then, send a message to the SQS queue as follows:
+```sh
+aws sqs send-message \
+    --queue-url "<REPLACE_WITH_OUTPUT_FROM_PREVIOUS_COMMAND>" \
+    --message-body '{"Message": "{\"content\":\"Test message\",\"params\":{\"name\":\"Mario\",\"surname\":\"Rossi\"}}"}'
+```
+
+Navigate to [AWS Console DynamoDB Tables](https://console.aws.amazon.com/dynamodbv2/home#tables) and check for a table with name `Audit-Table`.
+When you check the DynamoDB table, you can see the entry with all the attributes parsed by API Gateway.
+
+Alternatively, you can scan the table using AWS CLI:
+```sh
+aws dynamodb scan --table-name Audit-Table
+```
+
+## Cleanup
+
+1. Delete the stack
+
+```bash
+cdk destroy
+```
+
+---
+
+Copyright 2026 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+
+SPDX-License-Identifier: MIT-0

eventbridge-pipes-sqs-to-dynamodb/app.py

@@ -0,0 +1,9 @@
+#!/usr/bin/env python3
+import os
+import aws_cdk as cdk
+from eventbridge_pipes_sqs_to_dynamodb.eventbridge_pipes_sqs_to_dynamodb import EventbridgePipesSqsToDynamodb
+
+app = cdk.App()
+stack = EventbridgePipesSqsToDynamodb(app, "EventbridgePipesSqsToDynamodb")
+
+app.synth()

eventbridge-pipes-sqs-to-dynamodb/cdk.json

@@ -0,0 +1,3 @@
+{
+  "app": "python3 app.py"
+}

eventbridge-pipes-sqs-to-dynamodb/eventbridge_pipes_sqs_to_dynamodb/eventbridge_pipes_sqs_to_dynamodb.py

@@ -0,0 +1,218 @@
+from aws_cdk import (
+    Stack,
+    aws_apigateway as apigateway,
+    aws_dynamodb as dynamodb,
+    aws_logs as logs,
+    aws_pipes as pipes,
+    aws_sqs as sqs,
+    aws_iam as iam,
+    RemovalPolicy,
+    CfnOutput
+)
+from constructs import Construct
+import aws_cdk as cdk
+
+class EventbridgePipesSqsToDynamodb(Stack):
+
+    def __init__(self, scope: Construct, construct_id: str, env: cdk.Environment = None, **kwargs) -> None:
+        super().__init__(scope, construct_id, env=env, **kwargs)
+
+        # SQS queue for decoupling and buffering events
+        source_queue = sqs.Queue(
+            self, "EntryPointToEventbridgePipe",
+            visibility_timeout=cdk.Duration.seconds(60),
+            retention_period=cdk.Duration.days(4),
+            enforce_ssl=True
+        )
+
+        # DynamoDB table
+        self.table = dynamodb.Table(
+            self, "EventTableNew",
+            table_name="Audit-Table",
+            partition_key=dynamodb.Attribute(
+                name="id",
+                type=dynamodb.AttributeType.STRING
+            ),
+            billing_mode=dynamodb.BillingMode.PAY_PER_REQUEST,
+            removal_policy=RemovalPolicy.DESTROY,
+            point_in_time_recovery_specification=dynamodb.PointInTimeRecoverySpecification(
+                point_in_time_recovery_enabled=True
+            )
+        )
+
+        # IAM role for API Gateway to access DynamoDB
+        api_gateway_role = iam.Role(
+            self, "ApiGatewayDynamoDBRole",
+            assumed_by=iam.ServicePrincipal("apigateway.amazonaws.com"),
+            inline_policies={
+                "DynamoDBAccess": iam.PolicyDocument(
+                    statements=[
+                        iam.PolicyStatement(
+                            actions=[
+                                "dynamodb:PutItem",
+                                "dynamodb:GetItem",
+                                "dynamodb:UpdateItem",
+                                "dynamodb:DeleteItem",
+                                "dynamodb:Query",
+                                "dynamodb:Scan"
+                            ],
+                            resources=[self.table.table_arn]
+                        )
+                    ]
+                )
+            }
+        )
+
+        # CloudWatch Log Group for API Gateway
+        api_log_group = logs.LogGroup(
+            self, "ApiGatewayLogGroup",
+            removal_policy=RemovalPolicy.DESTROY
+        )
+
+        stage_name = "test"
+
+        # API Gateway with Resource policy to be invoked only by Eventbridge Pipes
+        self.api = apigateway.RestApi(
+            self, "RegionalApi",
+            rest_api_name="EventBridge-DynamoDB-API",
+            endpoint_configuration=apigateway.EndpointConfiguration(
+                types=[apigateway.EndpointType.REGIONAL]
+            ),
+            policy=iam.PolicyDocument(
+                statements=[
+                    iam.PolicyStatement(
+                        effect=iam.Effect.ALLOW,
+                        principals=[iam.ServicePrincipal("pipes.amazonaws.com")],
+                        actions=["execute-api:Invoke"],
+                        resources=[f"execute-api:/{stage_name}/POST/events"]
+                    )
+                ]
+            ),
+            deploy_options=apigateway.StageOptions(
+                stage_name=stage_name,
+                access_log_destination=apigateway.LogGroupLogDestination(api_log_group),
+                access_log_format=apigateway.AccessLogFormat.clf(),
+                logging_level=apigateway.MethodLoggingLevel.INFO,
+                data_trace_enabled=True
+            )
+        )
+
+        # API Gateway integration with DynamoDB
+        dynamodb_integration = apigateway.AwsIntegration(
+            service="dynamodb",
+            action="PutItem",
+            options=apigateway.IntegrationOptions(
+                credentials_role=api_gateway_role,
+                request_templates={
+                    "application/json": f'''#set($inputRoot = $input.path('$'))
+#set($body = $util.parseJson($inputRoot.body))
+#set($message = $util.parseJson($body.Message))
+{{
+    "TableName": "{self.table.table_name}",
+    "Item": {{
+        "id": {{
+            "S": "$context.requestId"
+        }},
+        "createdAt": {{
+            "S": "$context.requestTime"
+        }},
+        "name": {{
+            "S": "$message.params.name"
+        }},
+        "surname": {{
+            "S": "$message.params.surname"
+        }},
+        "content": {{
+            "S": "$util.escapeJavaScript($message.content)"
+        }}
+    }}
+}}'''
+                },
+                integration_responses=[
+                    apigateway.IntegrationResponse(
+                        status_code="200",
+                        response_templates={
+                            "application/json": '{"status": "success", "id": "$context.requestId"}'
+                        }
+                    )
+                ]
+            )
+        )    
+
+        # API Gateway resource and method with IAM authentication
+        events_resource = self.api.root.add_resource("events")
+        events_resource.add_method(
+            "POST",
+            dynamodb_integration,
+            authorization_type=apigateway.AuthorizationType.IAM,
+            method_responses=[
+                apigateway.MethodResponse(
+                    status_code="200",
+                    response_models={
+                        "application/json": apigateway.Model.EMPTY_MODEL
+                    }
+                )
+            ]
+        )
+
+        # IAM role for EventBridge Pipe
+        pipe_role = iam.Role(
+            self, "EventBridgePipeRole",
+            assumed_by=iam.ServicePrincipal("pipes.amazonaws.com").with_conditions({
+                "StringEquals": {
+                    "aws:SourceAccount": cdk.Stack.of(self).account,
+                    "aws:SourceArn": f"arn:aws:pipes:{cdk.Stack.of(self).region}:{cdk.Stack.of(self).account}:pipe/EventBridgePipe"
+                }
+            }),
+            inline_policies={
+                "SqsPipeSourceAccess": iam.PolicyDocument(
+                    statements=[
+                        iam.PolicyStatement(
+                            actions=[
+                                "sqs:ReceiveMessage",
+                                "sqs:DeleteMessage",
+                                "sqs:GetQueueAttributes"
+                            ],
+                            resources=[source_queue.queue_arn]
+                        )
+                    ]
+                ),
+                "ApiGatewayPipeTargetAccess": iam.PolicyDocument(
+                    statements=[
+                        iam.PolicyStatement(
+                            actions=[
+                                "execute-api:Invoke",
+                                "execute-api:ManageConnections"
+                            ],
+                            resources=[f"arn:aws:execute-api:{cdk.Stack.of(self).region}:{cdk.Stack.of(self).account}:{self.api.rest_api_id}/{stage_name}/*"]
+                        )
+                    ]
+                )
+            }
+        )
+
+        # EventBridge Pipe: SQS → API Gateway
+        self.pipe = pipes.CfnPipe(
+            self, "SqsToApiGatewayPipe",
+            role_arn=pipe_role.role_arn,
+            name="EventBridgePipe",
+            desired_state="RUNNING",
+            source=source_queue.queue_arn,
+            source_parameters=pipes.CfnPipe.PipeSourceParametersProperty(
+                sqs_queue_parameters=pipes.CfnPipe.PipeSourceSqsQueueParametersProperty(
+                    batch_size=1
+                )
+            ),
+            target=f"arn:aws:execute-api:{cdk.Stack.of(self).region}:{cdk.Stack.of(self).account}:{self.api.rest_api_id}/{stage_name}/POST/events",
+            target_parameters=pipes.CfnPipe.PipeTargetParametersProperty(
+                http_parameters=pipes.CfnPipe.PipeTargetHttpParametersProperty(
+                    header_parameters={
+                        "Content-Type": "application/json"
+                    }
+                )
+            )
+        )
+
+        # Output
+        CfnOutput(self, "QueueName", value=source_queue.queue_name)
+

eventbridge-pipes-sqs-to-dynamodb/example-pattern.json

@@ -0,0 +1,65 @@
+{
+    "title": "Amazon SQS to Amazon DynamoDB via Amazon EventBridge Pipes using CDK/Python",
+    "description": "Pattern that sends messages from SQS queue to DynamoDB table via API Gateway using EventBridge Pipes. Implemented with CDK using Python.",
+    "language": "Python",
+    "level": "300",
+    "framework": "AWS CDK",
+    "introBox": {
+      "headline": "How it works",
+      "text": [
+        "Messages sent to the SQS queue are polled by EventBridge Pipe.",
+        "EventBridge Pipe processes the messages and sends them to API Gateway endpoint.",
+        "API Gateway transforms the message and writes the data to DynamoDB table using direct integration."
+      ]
+    },
+    "gitHub": {
+      "template": {
+        "repoURL": "https://github.com/aws-samples/serverless-patterns/tree/main/eventbridge-pipes-sqs-to-dynamodb",
+        "templateURL": "serverless-patterns/eventbridge-pipes-sqs-to-dynamodb",
+        "projectFolder": "eventbridge-pipes-sqs-to-dynamodb",
+        "templateFile": "eventbridge-pipes-sqs-to-dynamodb/eventbridge_pipes_sqs_to_dynamodb/eventbridge_pipes_sqs_to_dynamodb.py"
+      }
+    },
+    "resources": {
+      "bullets": [
+        {
+          "text": "Amazon SQS",
+          "link": "https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/welcome.html"
+        },
+        {
+          "text": "EventBridge Pipes Documentation",
+          "link": "https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes-event-source.html"
+        },
+        {
+          "text": "Cloudformation API for Pipes",
+          "link": "https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-pipes-pipe.html"
+        },
+        {
+          "text": "Pipes Documentation for CDK v2 Python",
+          "link": "https://docs.aws.amazon.com/cdk/api/v2/python/aws_cdk.aws_pipes/CfnPipe.html"
+        },
+        {
+          "text": "API Gateway DynamoDB Integration",
+          "link": "https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-overview-developer-experience.html"
+        }
+      ]
+    },
+    "deploy": {
+      "text": ["cdk deploy"]
+    },
+    "testing": {
+      "text": ["See the README in the GitHub repo for detailed testing instructions."]
+    },
+    "cleanup": {
+      "text": ["Delete the stack: <code>cdk destroy</code>."]
+    },
+    "authors": [
+      {
+        "name": "Michele Scarimbolo",
+        "image": "https://avatars.githubusercontent.com/u/229997073",
+        "bio": "Technical Account Manager at AWS. Serverless specialist and enthusiast.",
+        "linkedin": "michele-scarimbolo",
+        "twitter": ""
+      }
+    ]
+  }

eventbridge-pipes-sqs-to-dynamodb/requirements.txt

@@ -0,0 +1,2 @@
+aws-cdk-lib==2.181.1
+constructs>=10.0.0,<11.0.0