Add Application Signals logs pipeline translator and allow metric routing to otlp endpoint by jj22ee · Pull Request #2111 · aws/amazon-cloudwatch-agent

jj22ee · 2026-05-08T00:09:57Z

Description of the issue

Application Signals currently collects metrics and traces. This PR updates and refactors the applicationsignals translator to also generate OTel logs and metrics pipelines that route OTLP data to CloudWatch via the CW OTLP endpoints, with automatic per-service log group creation and selective metrics routing.

Customers configure logs.logs_collected.application_signals with optional log_group_name (supporting placeholders like {service.name}) and log_stream_name. The provisioner extension creates log groups/streams on first encounter.

Description of changes

Update the existing translator/translate/otel/pipeline/applicationsignals/ translator with routing connectors for both logs and metrics signals. Each signal uses a 3-pipeline topology (receive → route → export) connected via a routing connector.

Metrics (3 pipelines via routing connector):

metrics/application_signals_metrics_route: OTLP receiver → routing connector
metrics/application_signals_metrics_logs_destination (default): routing → existing AppSignals processors → EMF exporter
metrics/application_signals_metrics_otlp_destination: routing → batch → otlphttp to CW OTLP monitoring endpoint
- Routes by: attributes["Telemetry.Source"] == "ServiceEvents" (datapoint context)

Logs (3 pipelines via routing connector):

logs/application_signals_logs_route: OTLP receiver → transform → attributestocontext → transform → routing connector
- Routes by: attributes["event.name"] == "aws.service_events.aggregate_profile" (log context)
- If there are no placeholders, transform and attributestocontext are not used
- temporary resource attributes (prefixed with temporary_key.) are used to help build the aws.log.group.name/aws.log.stream.name attributes in the first transform, and will be discarded in the second transform.
logs/application_signals_logs_batch (default): routing → batch → otlphttp to CW OTLP logs endpoint
logs/application_signals_logs_nobatch: routing → otlphttp (no batch, for large payloads)
- Auth chain: otlphttp → headers_setter → awscloudwatchlogsprovisioner → sigv4auth

Traces (unchanged):

Single pipeline with existing processors and X-Ray exporter

Notable behaviors:

Logs auto-opt-in: logs pipeline activates when metrics_collected.application_signals is configured. Explicitly disable via: logs.logs_collected.application_signals.disabled: true
Multiple placeholders supported in both group and stream (e.g. /{deployment.environment}/prefix/{service.name}) (Default log group: /aws/service-events/{service.name})
Customer-set resource attributes for aws.log.group.name/aws.log.stream.name are ignored

New/Updated shared translator packages:

extension/awscloudwatchlogsprovisioner — new
extension/headerssetter — new
processor/attributestocontext — new
connector/routing — new
exporter/otlphttp — new (generic, signal-aware EndpointConfig)
extension/sigv4auth — Updated with WithService option
processor/batchprocessor — Updated with WithMetadataKeys option
processor/transformprocessor — Updated with NewTranslatorWithLogStatements

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

Unit tests
Integration tests in amazon-cloudwatch-agent-test: <TODO: Make PR>

Requirements

Before commiting your code, please do the following steps.

Run make fmt and make fmt-sh
Run make lint

Integration Tests

To run integration tests against this PR, add the ready for testing label.

jj22ee · 2026-05-08T01:41:54Z

+	// TODO: Update default log group prefix before PR is merged.
+	defaultLogGroupPrefix = "/aws/telemetry/"


Still pending and not final.

Will now be /aws/service-events/{service.name}.

jj22ee · 2026-05-08T01:50:18Z

+	defaultLogStreamName  = "default"
+
+	metadataKeyLogGroup  = "aws.cloudwatch.log_group.destination"
+	metadataKeyLogStream = "aws.cloudwatch.log_stream.destination"


These key values are not following any otel spec, they're just arbitrary attributes created by Transform Processor which are used by AttributesToContext Processor, so we don't expect customers to know about these keys or even set them in their exported logs.

However, if customer does know and set these these attributes in their logs, we'll respect it and won't override them.

…gs pipeline

sky333999 · 2026-05-08T20:53:01Z

 	}
+	// Auto-enable logs.logs_collected.application_signals when metrics is configured
+	// Must happen before creating confmap since confmap may copy the map
+	applicationsignalslogs.AutoEnableIfNeeded(m)


You dont need to mutate the json for this.. just rely on the existing AppSignals enabled checks in the pipeline translator.

sky333999 · 2026-05-08T20:53:55Z

 	translators.Merge(containerInsightsTranslators)
 	translators.Set(applicationsignals.NewTranslator(pipeline.SignalTraces))
 	translators.Set(applicationsignals.NewTranslator(pipeline.SignalMetrics))
+	translators.Set(applicationsignalslogs.NewTranslator())


Does this really need to be a new translator? We've been trying to keep one translator per "solution" that can internally create multiple pipelines.

Not needed, moved into the existing applicationsignals translator.

sky333999 · 2026-05-08T20:59:48Z

 	logKey           = common.ConfigKey(common.LogsKey, common.LogsCollectedKey)
 	metricKey        = common.ConfigKey(common.MetricsKey, common.MetricsCollectedKey)
-	skipInputSet     = collections.NewSet[string](files.SectionKey, windowsevents.SectionKey)
+	skipInputSet     = collections.NewSet[string](files.SectionKey, windowsevents.SectionKey, common.AppSignals, common.AppSignalsFallback)


What in your changes forced you to add this?
i.e. what complained when you didnt have this?

Before my change, application_signals only existed under logs.metrics_collected (for metrics pipeline) and traces.traces_collected (for traces pipeline). The adapter never saw it because it only scans logs_collected.

Now that we added logs.logs_collected.application_signals, the adapter scans it and tries to create a telegraf_application_signals receiver which doesn't exist and is why I try to skip it here.

sky333999 · 2026-05-08T21:02:05Z

+
+// NewTranslatorWithLogStatements creates a transform processor translator that
+// executes the given OTTL statements in the "resource" context for logs.
+func NewTranslatorWithLogStatements(name string, statements []string) common.ComponentTranslator {


nit: Can we follow the same pattern you did with WithMetadataKeys in the batch processor translator instead of creating this?

sky333999 · 2026-05-08T21:18:44Z

+}
+
+func (t *translator) ID() component.ID {
+	return component.MustNewID("attributestocontext")


nit: Can we just do return component.NewIDWithName(t.factory.Type(), t.name)?

sky333999 · 2026-05-08T21:22:30Z

+}
+
+func (t *translator) ID() component.ID {
+	return component.NewIDWithName(component.MustNewType("headers_setter"), t.name)


nit: Same here too and for the other components - can we just do return component.NewIDWithName(t.factory.Type(), t.name).

sky333999 · 2026-05-08T21:23:07Z

+
+var _ common.ComponentTranslator = (*translator)(nil)
+
+func NewTranslatorWithName(name string, additionalAuth component.ID, headers []HeaderMapping) common.ComponentTranslator {


nit: The With might be better suited unless we are saying these are all mandatory.

…etrics, selectively export metrics to otlp endpoint

…er bugs

jefchien · 2026-05-20T21:15:03Z

-func NewTranslatorWithName(name string) common.ComponentTranslator {
-	return &translator{name, sigv4authextension.NewFactory()}
+func NewTranslatorWithService(service string) common.ComponentTranslator {
+	return &translator{name: service, service: service, factory: sigv4authextension.NewFactory()}


nit: Don't need a separate field for name since it's always the same as service.

Done in 6b3465c

jefchien · 2026-05-20T21:37:49Z

+	AppSignalsLogs            = ConfigKey(LogsKey, LogsCollectedKey, AppSignals)
 	AppSignalsTracesFallback  = ConfigKey(TracesKey, TracesCollectedKey, AppSignalsFallback)
 	AppSignalsMetricsFallback = ConfigKey(LogsKey, MetricsCollectedKey, AppSignalsFallback)
+	AppSignalsLogsFallback    = ConfigKey(LogsKey, LogsCollectedKey, AppSignalsFallback)


nit: Seems unused

Done in 61ff1d1

jefchien · 2026-05-20T21:41:04Z

+		translators.Set(NewTranslator(signal, SetVariant(metricsVariantLogDest)))
+		translators.Set(NewTranslator(signal, SetVariant(metricsVariantOtlpDest)))
+	case pipeline.SignalLogs:
+		if isLogsDisabled(conf) {


nit: Should have a nil check for the conf.

Done in 62723cf

jefchien · 2026-05-20T22:03:46Z

+                "log_group_name": {
+                  "type": "string",
+                  "minLength": 1
+                },
+                "log_stream_name": {
+                  "type": "string",
+                  "minLength": 1


nit: maxLength could be set to 512 to match https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_PutLogEvents.html although I'm sure the placeholders will allow it to go past that.

Done in 9763ae6.

jefchien · 2026-05-20T22:11:06Z

+            - condition: attributes["Telemetry.Source"] == "ServiceEvents"
+              context: datapoint


Is there a reason this is on the datapoint attributes and not the resource attributes?

In our SDK, it will set the attribute in on counter.add() attributes, so it's a datapoint attribute. SDK can send datapoints with different kinds of Telemetry.Source so we can't have it on resource.attributes.

jefchien · 2026-05-20T22:21:07Z

+// EndpointConfig specifies the base endpoint and signal-specific endpoint for
+// the otlphttp exporter.
+type EndpointConfig struct {
+	BaseEndpoint    string // e.g. "https://logs.us-east-1.amazonaws.com"
+	LogsEndpoint    string // e.g. "https://logs.us-east-1.amazonaws.com/v1/logs"
+	MetricsEndpoint string
+	TracesEndpoint  string
+}


Yes, but I'm fine with what you have as well.

jefchien · 2026-05-20T22:23:57Z

+              error_mode: propagate
+              statements:
+                - set(resource.attributes["aws.cloudwatch.log_group.destination"], Concat(["/aws/service-events/", resource.attributes["service.name"]], "")) where resource.attributes["aws.cloudwatch.log_group.destination"] == nil
+                - replace_pattern(resource.attributes["aws.cloudwatch.log_group.destination"], "<nil>", "undefined")


Should we remove this aws.cloudwatch.log_group.destination resource attribute after attributestocontext sets it in the context? If we don't it's going to be included in every single log we send. Could add a transformprocessor for to clean it up before routing or in the export pipeline.

As discussed, added a transform processor to cleanup: 5cec422

jefchien · 2026-05-20T22:26:56Z

+	metadataKeyLogGroup  = "aws.cloudwatch.log_group.destination"
+	metadataKeyLogStream = "aws.cloudwatch.log_stream.destination"


nit: Consider making the metadata key consistent with the header key.

Do you mean to use the naming aws.log.group.name / aws.log.stream.name instead of aws.cloudwatch.log_...?

Done in cb34b0a

Meant x-aws-log-group/x-aws-log-stream like with https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_HTTP_Endpoints_OTLP.html so it's clear what they're for, but I think it's fine either way.

…butes, handle missing attrs better

…plify its translation logic, update go.mod

jj22ee requested a review from a team as a code owner May 8, 2026 00:09

jj22ee added 4 commits May 7, 2026 17:12

setup cwagent with jj22ee fork components

4522aba

update emf exporter config

99a3578

translator changes

9210013

refactor

9027c02

jj22ee force-pushed the appsignals-logs-translator-pr branch from 83b5680 to 9027c02 Compare May 8, 2026 00:12

jj22ee changed the title ~~Add Translation logic for logs.logs_collected.application_signals~~ Add Application Signals logs pipeline translator with dynamic log group routing May 8, 2026

jj22ee changed the title ~~Add Application Signals logs pipeline translator with dynamic log group routing~~ Add Application Signals logs pipeline translator May 8, 2026

fix failing workfows, minor refactor

1201ff9

jj22ee commented May 8, 2026

View reviewed changes

regenerate sample config yamls for new EMF fields and auto-enabled lo…

d454f8f

…gs pipeline

jj22ee force-pushed the appsignals-logs-translator-pr branch from 51f3c5d to d454f8f Compare May 8, 2026 02:18

jj22ee added 3 commits May 7, 2026 21:47

fix sigv4 issues in unit tests

29a746b

undo unrelated changes in sampleConfig yamls

cd0b010

sampleConfig cleanup

d3d8a4f

sky333999 reviewed May 8, 2026

View reviewed changes

jj22ee and others added 2 commits May 11, 2026 18:24

address comments

f880613

Merge branch 'main' into appsignals-logs-translator-pr

46cd360

jj22ee force-pushed the appsignals-logs-translator-pr branch 2 times, most recently from 960cebe to 46cd360 Compare May 16, 2026 02:31

jj22ee and others added 7 commits May 15, 2026 19:31

Merge branch 'main' into appsignals-logs-translator-pr

8f2783e

update default log group name

30f1c58

refactor application_signals translators - use routing for logs and m…

2a07de6

…etrics, selectively export metrics to otlp endpoint

add disable setting for application-signals logs

ad7599f

update sampleConfig

9d85712

fix lint

5eba9f4

fix metrics routing condition

d5a4647

jj22ee force-pushed the appsignals-logs-translator-pr branch from 5e9f489 to d5a4647 Compare May 16, 2026 06:48

mxiamxia reviewed May 18, 2026

View reviewed changes

Comment thread translator/tocwconfig/sampleConfig/appsignals_and_eks_config.yaml Outdated

mxiamxia reviewed May 18, 2026

View reviewed changes

Comment thread translator/tocwconfig/sampleConfig/appsignals_and_eks_config.yaml

use replace_pattern on destination attribute to avoid <nil> placehold…

96e771b

…er bugs

jefchien reviewed May 20, 2026

View reviewed changes

jj22ee added 11 commits May 20, 2026 17:14

remove redundant name field from sigv4auth translator

6b3465c

remove unused AppSignalsLogsFallback

61ff1d1

add nil check for conf in NewTranslators

62723cf

add maxLength 512 to log_group_name and log_stream_name

9763ae6

unexport newTranslator and setVariant

1b2f6a2

get sigv4 ID from translator instance instead of constructing manually

187c989

remove where-nil guard: always overwrite destination attribute

11a2538

get IDs from translator instances instead of constructing manually

64ce8ce

add cleanup transform to delete plumbing resource attributes

5cec422

rename metadata keys to aws.log.group.name / aws.log.stream.name

cb34b0a

use temp vars for placeholder resolution, avoid mutating source attri…

d3c5ded

…butes, handle missing attrs better

okankoAMZ reviewed May 21, 2026

View reviewed changes

Comment thread translator/translate/otel/pipeline/applicationsignals/translators.go

jefchien added the ready for testing Indicates this PR is ready for integration tests to run label May 21, 2026

okankoAMZ previously approved these changes May 21, 2026

View reviewed changes

jefchien previously approved these changes May 21, 2026

View reviewed changes

Comment thread translator/config/schema.json

add disabled field to application_signals logs schema

ffdf574

jj22ee dismissed stale reviews from jefchien and okankoAMZ via ffdf574 May 21, 2026 17:04

use exported ActionKeyValue from attributestocontextprocessor and sim…

a02ae36

…plify its translation logic, update go.mod

jj22ee force-pushed the appsignals-logs-translator-pr branch from 562e8a3 to a02ae36 Compare May 21, 2026 17:50

jefchien reviewed May 21, 2026

View reviewed changes

Comment thread go.mod Outdated

jefchien removed the ready for testing Indicates this PR is ready for integration tests to run label May 21, 2026

jj22ee and others added 2 commits May 21, 2026 11:08

add headerssetter/translator_test.go and organize go.mod

938e5c2

Merge branch 'main' into appsignals-logs-translator-pr

d356f63

jefchien approved these changes May 21, 2026

View reviewed changes

sky333999 approved these changes May 21, 2026

View reviewed changes

jefchien merged commit 37c1306 into aws:main May 22, 2026
20 of 22 checks passed

jj22ee mentioned this pull request May 22, 2026

Implement ApplicationSignals Logs integ tests for dynamic placeholders for otlp headers, and add Metrics tests for routing & OTLP endpoint aws/amazon-cloudwatch-agent-test#690

Open

		// TODO: Update default log group prefix before PR is merged.
		defaultLogGroupPrefix = "/aws/telemetry/"


		var _ common.ComponentTranslator = (*translator)(nil)

		func NewTranslatorWithName(name string, additionalAuth component.ID, headers []HeaderMapping) common.ComponentTranslator {

		- condition: attributes["Telemetry.Source"] == "ServiceEvents"
		context: datapoint

		metadataKeyLogGroup = "aws.cloudwatch.log_group.destination"
		metadataKeyLogStream = "aws.cloudwatch.log_stream.destination"

Conversation

jj22ee commented May 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of the issue

Description of changes

License

Tests

Requirements

Integration Tests

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

jj22ee commented May 8, 2026 •

edited

Loading