fix(ingestion): replace SDK integration with Lambda for S3 listing

[iankhou]

Summary

• Replaces the Step Functions CallAwsService SDK integration for ListObjectsV2 with a dedicated Lambda function
• The Lambda returns only the Key field for each S3 object, eliminating the 256KB payload limit issue entirely
• Removes the MaxKeys retry chain (Try1000→Try900→...) that was needed to work around States.DataLimitExceeded
• Simplifies the state machine graph significantly

Problem

The SDK integration returns the full S3 ListObjectsV2 response including per-object metadata (ETag, LastModified, Size, StorageClass) for up to 1000 objects. This response can approach or exceed the 256KB Step Functions payload limit.

There's an uncatchable edge case where the response is just under 256KB (so the state reports success), but Step Functions aborts the execution at the state transition boundary when internal metadata pushes the total over the limit. This ExecutionFailed event is not tied to a specific state, making addCatch(States.DataLimitExceeded) ineffective.

Solution

A Lambda function calls ListObjectsV2 and returns only what the workflow needs:

• Contents[].Key — the object keys
• NextContinuationToken — for pagination (only when present)

This reduces the payload from ~200-300KB to ~50-100KB for 1000 objects, well within the limit. The Lambda controls its own output size, making the payload limit concern irrelevant.

Test plan

• Run npx projen build to regenerate the construct class and bundle (projen auto-discovers *.lambda.ts files)
• Verify snapshot tests pass after regeneration
• Deploy and observe the daily reprocessing run succeeds with MaxKeys=1000
• Monitor Lambda invocation metrics and errors

Related

• Temporary mitigation: fix(ingestion): reduce MaxKeys to avoid uncatchable payload limit failure #1885 (reduces MaxKeys to 750)

[iankhou]

Testing this in development to verify that the lambda works as expected.