feat(telemetry): add custom headers for OTLP exports (metrics, tracing, access logs) #7674

codefromthecrypt · 2025-12-05T06:44:19Z

What type of PR is this?
feat(telemetry): add new feature

What this PR does / why we need it:
Adds support for custom headers on all OTLP/gRPC exports (metrics, tracing, and access logs), enabling authentication with collectors like Elastic Cloud, Datadog, or cloud providers that require API keys or bearer tokens.

This uses Envoy's GrpcService.initial_metadata to send headers as gRPC metadata.

Release Notes: Yes

Notes:
This PR includes an example showing authenticated OTLP exports for all three signals. The example uses otel-tui, but could use anything else.

Changes

Metrics: Added Headers field to ProxyOpenTelemetrySink in telemetry.metrics.sinks[].openTelemetry
Tracing: Added Headers field to TracingProvider in telemetry.tracing.provider
Access Logs: Added Headers field to OpenTelemetryAccessLog in telemetry.accessLog.settings[].sinks[].openTelemetry

All three use the same pattern: a list of HTTPHeader objects with name and value fields.

FAQ

Why not use SecretObjectReference for sensitive headers?

For EnvoyProxy config that supports file-based standalone mode, Kubernetes secrets are not available. Plain text headers are needed regardless.

For example, Honeycomb requires non-secret headers like x-honeycomb-dataset. Secret support can be added in a follow-up for Kubernetes mode.

internal/infrastructure/host/proxy_infra.go

codecov · 2025-12-05T06:56:22Z

Codecov Report

❌ Patch coverage is 92.00000% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.81%. Comparing base (dd2861f) to head (c3d6f0a).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
internal/gatewayapi/listener.go	66.66%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7674      +/-   ##
==========================================
+ Coverage   72.77%   72.81%   +0.03%     
==========================================
  Files         235      236       +1     
  Lines       35165    35190      +25     
==========================================
+ Hits        25592    25623      +31     
+ Misses       7756     7750       -6     
  Partials     1817     1817

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

api/v1alpha1/envoygateway_types.go

api/v1alpha1/envoyproxy_accesslogging_types.go

codefromthecrypt · 2025-12-05T12:13:31Z

ok made a revision also to PR desc to hopefully answer questions in comments. I will revisit this again monday if there is more feedback.

codefromthecrypt · 2025-12-07T00:30:01Z

pulling into draft while I remove the port dodging things for things we can't control yet.

codefromthecrypt · 2025-12-07T02:11:55Z

also I just realized we never added this for tracing either, so will add that to the PR

codefromthecrypt · 2025-12-07T05:13:35Z

examples/otel-headers/resources/gateway.yaml

+spec:
+  telemetry:
+    metrics:
+      # TODO: Some backends like otel-tui and Elastic require delta temporality


These problems are too much to do in this PR, and I can over time try to clean these up also

codefromthecrypt · 2025-12-07T05:47:56Z

test/standalone/standalone_test.go

+	log := collector.TakeLog()
+	require.NotNil(t, log)
+	require.Contains(t, log.Body.GetStringValue(), `HTTP/1.1" 200`)
+	require.Equal(t, "Bearer test-api-key", testotel.GetAttributeString(log.Attributes, "grpc.metadata.authorization"))


@mathetake @anuraaga you might like this sneaky trick to verify the headers/metadata received by otel. propagate them into fake span attrs

codefromthecrypt · 2025-12-07T06:22:17Z

ok I updated this as I noticed the same thing was missing everywhere, in logs metrics and tracing. fixed so that the tests are coherent and we don't need to go back and clean this up multiple times. There's been a history of fragmentation and not following up later, so doing otel holistically for one thing is a good thing.

api/v1alpha1/envoyproxy_accesslogging_types.go

internal/xds/translator/tracing.go

internal/infrastructure/host/proxy_infra.go

codefromthecrypt · 2025-12-24T03:11:35Z

this is pending merge of #7720 which has been stalled several days without any known reason. If anyone wants this stuff to land, please encourage maintainers to click merge and I'll rebase this after

codefromthecrypt · 2025-12-24T03:21:06Z

if this lasts until 2026 I will close it as unplanned/unwanted. I won't be attending to perpetual rebasing. I want to make things better, but this has been a low-light. Not only were things left half-finished, but finishing them has been beyond arduous

codefromthecrypt · 2025-12-29T01:28:29Z

re ran soup to nuts and the example works fine

zirain · 2025-12-30T01:45:07Z

/retest

codefromthecrypt · 2025-12-31T01:58:13Z

does anyone have access to merge this considering there are 2 approvals?

api/v1alpha1/envoyproxy_tracing_types.go

arkodg

LGTM thanks

Adds a Headers field to the OpenTelemetry configuration for access logging, tracing, and metrics. This allows configuring gRPC initial metadata (e.g., Authorization headers) for authenticated OTLP export to collectors like Elastic, Datadog, or cloud providers. Also adds MinItems=1 validation for OTLP headers. Signed-off-by: Adrian Cole <[email protected]>

…consistency Signed-off-by: Adrian Cole <[email protected]>

codefromthecrypt requested a review from a team as a code owner December 5, 2025 06:44

codefromthecrypt force-pushed the otel-grpc-headers branch from 63fd03f to 04c7b52 Compare December 5, 2025 06:47

github-advanced-security bot found potential problems Dec 5, 2025

View reviewed changes

internal/infrastructure/host/proxy_infra.go Fixed Show fixed Hide fixed

codefromthecrypt force-pushed the otel-grpc-headers branch 2 times, most recently from c58e24a to 5f69a4f Compare December 5, 2025 07:42

zirain reviewed Dec 5, 2025

View reviewed changes

api/v1alpha1/envoygateway_types.go Outdated Show resolved Hide resolved

zirain reviewed Dec 5, 2025

View reviewed changes

api/v1alpha1/envoyproxy_accesslogging_types.go Show resolved Hide resolved

codefromthecrypt force-pushed the otel-grpc-headers branch 2 times, most recently from 2483048 to 36f5bee Compare December 5, 2025 12:13

codefromthecrypt force-pushed the otel-grpc-headers branch from 36f5bee to 1a16872 Compare December 7, 2025 00:24

codefromthecrypt marked this pull request as draft December 7, 2025 00:29

codefromthecrypt force-pushed the otel-grpc-headers branch from 1a16872 to 47998d6 Compare December 7, 2025 00:34

codefromthecrypt changed the title ~~feat(accesslog): add custom headers for OTLP access log exports~~ feat(telemetry): add custom headers for OTLP exports (metrics, tracing, access logs) Dec 7, 2025

codefromthecrypt force-pushed the otel-grpc-headers branch from a5b5a19 to c7e3ecf Compare December 7, 2025 05:00

codefromthecrypt commented Dec 7, 2025

View reviewed changes

codefromthecrypt force-pushed the otel-grpc-headers branch 3 times, most recently from 607296e to 5bff38d Compare December 7, 2025 05:42

codefromthecrypt commented Dec 7, 2025

View reviewed changes

codefromthecrypt force-pushed the otel-grpc-headers branch 2 times, most recently from 412773b to ea11614 Compare December 7, 2025 06:20

codefromthecrypt marked this pull request as ready for review December 7, 2025 06:20

arkodg reviewed Dec 8, 2025

View reviewed changes

api/v1alpha1/envoyproxy_accesslogging_types.go Show resolved Hide resolved

arkodg reviewed Dec 8, 2025

View reviewed changes

internal/xds/translator/tracing.go Show resolved Hide resolved

arkodg reviewed Dec 8, 2025

View reviewed changes

internal/infrastructure/host/proxy_infra.go Show resolved Hide resolved

codefromthecrypt mentioned this pull request Dec 15, 2025

feat(telemetry): make access log format type optional #7720

Merged

codefromthecrypt force-pushed the otel-grpc-headers branch from 1448361 to 207b78d Compare December 17, 2025 03:20

codefromthecrypt force-pushed the otel-grpc-headers branch from 207b78d to f99c543 Compare December 29, 2025 01:27

codefromthecrypt marked this pull request as ready for review December 29, 2025 01:28

zirain previously approved these changes Dec 29, 2025

View reviewed changes

jukie dismissed zirain’s stale review via c9c13b9 December 29, 2025 18:30

jukie previously approved these changes Dec 29, 2025

View reviewed changes

jukie requested review from a team, arkodg and zirain December 29, 2025 18:31

zirain previously approved these changes Dec 30, 2025

View reviewed changes

arkodg reviewed Jan 1, 2026

View reviewed changes

api/v1alpha1/envoyproxy_tracing_types.go Outdated Show resolved Hide resolved

codefromthecrypt dismissed stale reviews from zirain and jukie via b46e22a January 2, 2026 03:04

codefromthecrypt force-pushed the otel-grpc-headers branch from c9c13b9 to b46e22a Compare January 2, 2026 03:04

arkodg approved these changes Jan 2, 2026

View reviewed changes

arkodg requested review from jukie and zirain January 2, 2026 04:13

arkodg added this to the v1.7.0-rc.1 Release milestone Jan 2, 2026

codefromthecrypt added 2 commits January 3, 2026 07:33

feat(telemetry): nest tracing headers in openTelemetry field for API …

c3d6f0a

…consistency Signed-off-by: Adrian Cole <[email protected]>

codefromthecrypt force-pushed the otel-grpc-headers branch from b46e22a to c3d6f0a Compare January 2, 2026 23:33

zirain approved these changes Jan 3, 2026

View reviewed changes

zirain merged commit e2a495d into envoyproxy:main Jan 3, 2026
35 checks passed

feat(telemetry): add custom headers for OTLP exports (metrics, tracing, access logs) #7674

feat(telemetry): add custom headers for OTLP exports (metrics, tracing, access logs) #7674

Conversation

codefromthecrypt commented Dec 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

FAQ

Why not use SecretObjectReference for sensitive headers?

Uh oh!

Uh oh!

codecov bot commented Dec 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

codefromthecrypt commented Dec 5, 2025

Uh oh!

codefromthecrypt commented Dec 7, 2025

Uh oh!

codefromthecrypt commented Dec 7, 2025

Uh oh!

codefromthecrypt Dec 7, 2025

Choose a reason for hiding this comment

Uh oh!

codefromthecrypt Dec 7, 2025

Choose a reason for hiding this comment

Uh oh!

codefromthecrypt commented Dec 7, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codefromthecrypt commented Dec 24, 2025

Uh oh!

codefromthecrypt commented Dec 24, 2025

Uh oh!

codefromthecrypt commented Dec 29, 2025

Uh oh!

zirain commented Dec 30, 2025

Uh oh!

codefromthecrypt commented Dec 31, 2025

Uh oh!

Uh oh!

arkodg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codefromthecrypt commented Dec 5, 2025 •

edited

Loading

codecov bot commented Dec 5, 2025 •

edited

Loading