[BUG]: DD_TAGS from Cluster Agent not injected

Tracer Version(s)

3.38.0

Operating system and platform

Linux x64

Instrumentation Mode

SSI

TFM

Net8.0?

Bug Report

It seems when deploying Datadog with the Helm chart and adding the datadog.tags directive, the resulting DD_TAGS environment variable available on the cluster agent is not injected into instrumented workloads.

Helm (only relevant parts):

datadog:
  tags:
    - 'env:dev'
    - 'region:eu-north-1'
    - 'provider:aws'
    - 'collector:datadog'

  apm:
    socketEnabled: false
    portEnabled: true
    instrumentation:
      enabled: true
      targets:
        - name: 'dotnet'
          namespaceSelector:
            matchNames:
              - 'dotnet-dev'
          ddTraceVersions:
            dotnet: 3.38.0 # latest as of this writing

In the Cluster Agent pod, I can see (again, only relevant bits):

        - name: DD_TAGS
          value: '["env:dev","region:eu-north-1","provider:aws","collector:datadog"]'

      image: eu.gcr.io/datadoghq/cluster-agent:7.75.0

And finally, in an instrumented workload:

Details

apiVersion: v1
kind: Pod
metadata:
  annotations:
    internal.apm.datadoghq.com/applied-target: >-
      {"name":"dotnet","namespaceSelector":{"matchNames":["dotnet-dev"]},"ddTraceVersions":{"dotnet":"3.38.0"}}
  labels:
    app: email
    env: dev
    pod-template-hash: 5799b8dd57
    provider: AWS
    region: eu-north-1
    runtime: dotnet
    version: sha-df71385
  name: email-5799b8dd57-czl9q
  namespace: dotnet-dev
spec:
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - podAffinityTerm:
            labelSelector:
              matchLabels:
                app: email
            topologyKey: topology.kubernetes.io/zone
          weight: 100
  containers:
    - env:
        - name: DD_INSTRUMENTATION_APPLIED_TARGET
          value: >-
            {"name":"dotnet","namespaceSelector":{"matchNames":["dotnet-dev"]},"ddTraceVersions":{"dotnet":"3.38.0"}}
        - name: DD_RUNTIME_METRICS_ENABLED
          value: 'true'
        - name: DD_TRACE_HEALTH_METRICS_ENABLED
          value: 'true'
        - name: DD_LOGS_INJECTION
          value: 'true'
        - name: DD_TRACE_ENABLED
          value: 'true'
        - name: DD_SERVICE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.labels['app']
        - name: DD_SERVICE_K8S_ENV_SOURCE
          value: labels_as_tags
        - name: DD_ENV
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.labels['env']
        - name: DD_VERSION
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.labels['version']
        - name: DD_INSTRUMENTATION_INSTALL_TYPE
          value: k8s_single_step
        - name: DD_INSTRUMENTATION_INSTALL_ID
          value: a289eaa2-683a-453b-815b-6293950d9639
        - name: DD_INSTRUMENTATION_INSTALL_TIME
          value: '1754571794'
        - name: DD_INTERNAL_POD_UID
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.uid
        - name: DD_EXTERNAL_ENV
          value: it-false,cn-email,pu-$(DD_INTERNAL_POD_UID)
        - name: DD_ENTITY_ID
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.uid
        - name: DD_AGENT_HOST
          value: datadog.datadog.svc.cluster.local
        - name: ASPNETCORE_URLS
          value: http://0.0.0.0:8080
        - name: LD_PRELOAD
          value: >-
            /opt/datadog-packages/datadog-apm-inject/stable/inject/launcher.preload.so
        - name: DD_INJECT_SENDER_TYPE
          value: k8s
        - name: DD_INJECT_START_TIME
          value: '1772214719'
        - name: DD_INSTRUMENTATION_LANGUAGES_DETECTED
          value: dotnet
        - name: DD_INSTRUMENTATION_LANGUAGE_DETECTION_INJECTION_ENABLED
          value: 'true'
      image: ghcr.io/company/email:sha-df71385
      imagePullPolicy: IfNotPresent
      livenessProbe:
        failureThreshold: 5
        httpGet:
          path: /healthz
          port: 8080
          scheme: HTTP
        initialDelaySeconds: 10
        periodSeconds: 10
        successThreshold: 1
        timeoutSeconds: 5
      name: email
      ports:
        - containerPort: 8080
          protocol: TCP
      resources:
        limits:
          memory: 256Mi
        requests:
          cpu: 50m
          memory: 256Mi
      startupProbe:
        failureThreshold: 24
        httpGet:
          path: /healthz
          port: 8080
          scheme: HTTP
        initialDelaySeconds: 30
        periodSeconds: 5
        successThreshold: 1
        timeoutSeconds: 5
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
        - mountPath: /opt/datadog-packages/datadog-apm-inject
          name: datadog-auto-instrumentation
          subPath: opt/datadog-packages/datadog-apm-inject
        - mountPath: /etc/ld.so.preload
          name: datadog-auto-instrumentation-etc
          readOnly: true
          subPath: ld.so.preload
        - mountPath: /app/appsettings.json
          name: appsettings-volume
          subPath: appsettings.json
        - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
          name: kube-api-access-w4jrd
          readOnly: true
        - mountPath: /opt/datadog/apm/library
          name: datadog-auto-instrumentation
          subPath: opt/datadog/apm/library
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  imagePullSecrets:
    - name: ghcr-login
  initContainers:
    - args:
        - >-
          cp -r /opt/datadog-packages/datadog-apm-inject/* /datadog-inject &&
          echo
          /opt/datadog-packages/datadog-apm-inject/stable/inject/launcher.preload.so
          > /datadog-etc/ld.so.preload && echo $(date +%s) >>
          /datadog-inject/c-init-time.datadog-init-apm-inject
      command:
        - /bin/sh
        - '-c'
        - '--'
      env:
        - name: DD_INTERNAL_POD_UID
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.uid
        - name: DD_EXTERNAL_ENV
          value: it-true,cn-datadog-init-apm-inject,pu-$(DD_INTERNAL_POD_UID)
        - name: DD_ENTITY_ID
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.uid
        - name: DD_AGENT_HOST
          value: datadog.datadog.svc.cluster.local
        - name: DD_INSTRUMENTATION_APPLIED_TARGET
          value: >-
            {"name":"dotnet","namespaceSelector":{"matchNames":["dotnet-dev"]},"ddTraceVersions":{"dotnet":"3.38.0"}}
        - name: DD_RUNTIME_METRICS_ENABLED
          value: 'true'
        - name: DD_TRACE_HEALTH_METRICS_ENABLED
          value: 'true'
        - name: DD_LOGS_INJECTION
          value: 'true'
        - name: DD_TRACE_ENABLED
          value: 'true'
        - name: DD_SERVICE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.labels['app']
        - name: DD_SERVICE_K8S_ENV_SOURCE
          value: labels_as_tags
        - name: DD_ENV
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.labels['env']
        - name: DD_VERSION
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.labels['version']
      image: eu.gcr.io/datadoghq/apm-inject:0
      imagePullPolicy: IfNotPresent
      name: datadog-init-apm-inject
      resources:
        limits:
          cpu: 50m
          memory: 256Mi
        requests:
          cpu: 50m
          memory: 256Mi
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
        - mountPath: /datadog-inject
          name: datadog-auto-instrumentation
          subPath: opt/datadog-packages/datadog-apm-inject
        - mountPath: /datadog-etc
          name: datadog-auto-instrumentation-etc
        - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
          name: kube-api-access-w4jrd
          readOnly: true
    - args:
        - >-
          sh copy-lib.sh /datadog-lib && echo $(date +%s) >>
          /opt/datadog-packages/datadog-apm-inject/c-init-time.datadog-lib-dotnet-init
      command:
        - /bin/sh
        - '-c'
        - '--'
      env:
        - name: DD_INTERNAL_POD_UID
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.uid
        - name: DD_EXTERNAL_ENV
          value: it-true,cn-datadog-lib-dotnet-init,pu-$(DD_INTERNAL_POD_UID)
        - name: DD_ENTITY_ID
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.uid
        - name: DD_AGENT_HOST
          value: datadog.datadog.svc.cluster.local
        - name: DD_INSTRUMENTATION_APPLIED_TARGET
          value: >-
            {"name":"dotnet","namespaceSelector":{"matchNames":["dotnet-dev"]},"ddTraceVersions":{"dotnet":"3.38.0"}}
        - name: DD_RUNTIME_METRICS_ENABLED
          value: 'true'
        - name: DD_TRACE_HEALTH_METRICS_ENABLED
          value: 'true'
        - name: DD_LOGS_INJECTION
          value: 'true'
        - name: DD_TRACE_ENABLED
          value: 'true'
        - name: DD_SERVICE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.labels['app']
        - name: DD_SERVICE_K8S_ENV_SOURCE
          value: labels_as_tags
        - name: DD_ENV
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.labels['env']
        - name: DD_VERSION
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.labels['version']
      image: eu.gcr.io/datadoghq/dd-lib-dotnet-init:3.38.0
      imagePullPolicy: IfNotPresent
      name: datadog-lib-dotnet-init
      resources:
        limits:
          cpu: 50m
          memory: 256Mi
        requests:
          cpu: 50m
          memory: 256Mi
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
        - mountPath: /datadog-lib
          name: datadog-auto-instrumentation
          subPath: opt/datadog/apm/library/dotnet
        - mountPath: /opt/datadog-packages/datadog-apm-inject
          name: datadog-auto-instrumentation
          subPath: opt/datadog-packages/datadog-apm-inject
        - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
          name: kube-api-access-w4jrd
          readOnly: true
  nodeName: ip-10-110-128-81.eu-north-1.compute.internal
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
    - effect: NoExecute
      key: node.kubernetes.io/not-ready
      operator: Exists
      tolerationSeconds: 300
    - effect: NoExecute
      key: node.kubernetes.io/unreachable
      operator: Exists
      tolerationSeconds: 300
  topologySpreadConstraints:
    - labelSelector:
        matchLabels:
          app: email
      maxSkew: 1
      topologyKey: topology.kubernetes.io/zone
      whenUnsatisfiable: ScheduleAnyway
  volumes:
    - name: appsettings-volume
      secret:
        defaultMode: 420
        items:
          - key: appsettings.json
            path: appsettings.json
        secretName: email-appsettings
    - name: kube-api-access-w4jrd
      projected:
        defaultMode: 420
        sources:
          - serviceAccountToken:
              expirationSeconds: 3607
              path: token
          - configMap:
              items:
                - key: ca.crt
                  path: ca.crt
              name: kube-root-ca.crt
          - downwardAPI:
              items:
                - fieldRef:
                    apiVersion: v1
                    fieldPath: metadata.namespace
                  path: namespace
    - emptyDir: {}
      name: datadog-auto-instrumentation
    - emptyDir: {}
      name: datadog-auto-instrumentation-etc
status:
  conditions:
    - lastProbeTime: null
      lastTransitionTime: '2026-02-27T17:52:01Z'
      status: 'True'
      type: PodReadyToStartContainers
    - lastProbeTime: null
      lastTransitionTime: '2026-02-27T17:52:03Z'
      status: 'True'
      type: Initialized
    - lastProbeTime: null
      lastTransitionTime: '2026-02-27T17:52:35Z'
      status: 'True'
      type: Ready
    - lastProbeTime: null
      lastTransitionTime: '2026-02-27T17:52:35Z'
      status: 'True'
      type: ContainersReady
    - lastProbeTime: null
      lastTransitionTime: '2026-02-27T17:51:59Z'
      status: 'True'
      type: PodScheduled
  containerStatuses:
    - containerID: >-
        containerd://ef935ab592db7e433f7c1f6e1a6a3cfab584627eb4d231717811d83f6cdae693
      image: ghcr.io/company/email:sha-df71385
      imageID: >-
        ghcr.io/company/email@sha256:d7cb533f5683dd3226cd3fc854adee4a6dc3705d8e72b560ec727fb108f69639
      lastState: {}
      name: email
      ready: true
      restartCount: 0
      started: true
      state:
        running:
          startedAt: '2026-02-27T17:52:03Z'
      volumeMounts:
        - mountPath: /opt/datadog-packages/datadog-apm-inject
          name: datadog-auto-instrumentation
        - mountPath: /etc/ld.so.preload
          name: datadog-auto-instrumentation-etc
          readOnly: true
          recursiveReadOnly: Disabled
        - mountPath: /app/appsettings.json
          name: appsettings-volume
        - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
          name: kube-api-access-w4jrd
          readOnly: true
          recursiveReadOnly: Disabled
        - mountPath: /opt/datadog/apm/library
          name: datadog-auto-instrumentation
  hostIP: 10.110.128.81
  hostIPs:
    - ip: 10.110.128.81
  initContainerStatuses:
    - containerID: >-
        containerd://c75834bba1c10edf621f54fa3b4ca7e8d3091fcadf1ebc7e1d8b39b96aff5485
      image: eu.gcr.io/datadoghq/apm-inject:0
      imageID: >-
        eu.gcr.io/datadoghq/apm-inject@sha256:b5677a7a55b085612dced5f1930843abb48227fbc93d37c873ee7e6d80b86fa0
      lastState: {}
      name: datadog-init-apm-inject
      ready: true
      restartCount: 0
      started: false
      state:
        terminated:
          containerID: >-
            containerd://c75834bba1c10edf621f54fa3b4ca7e8d3091fcadf1ebc7e1d8b39b96aff5485
          exitCode: 0
          finishedAt: '2026-02-27T17:52:00Z'
          reason: Completed
          startedAt: '2026-02-27T17:52:00Z'
      volumeMounts:
        - mountPath: /datadog-inject
          name: datadog-auto-instrumentation
        - mountPath: /datadog-etc
          name: datadog-auto-instrumentation-etc
        - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
          name: kube-api-access-w4jrd
          readOnly: true
          recursiveReadOnly: Disabled
    - containerID: >-
        containerd://2c160edffcab681ef963fb12064eebfb456a18ed0efd5718dd53f00f27afcca4
      image: eu.gcr.io/datadoghq/dd-lib-dotnet-init:3.38.0
      imageID: >-
        eu.gcr.io/datadoghq/dd-lib-dotnet-init@sha256:cb381708166b42419f14ee4e79da55515e09021a5ebde5f295838b87c5df84ad
      lastState: {}
      name: datadog-lib-dotnet-init
      ready: true
      restartCount: 0
      started: false
      state:
        terminated:
          containerID: >-
            containerd://2c160edffcab681ef963fb12064eebfb456a18ed0efd5718dd53f00f27afcca4
          exitCode: 0
          finishedAt: '2026-02-27T17:52:02Z'
          reason: Completed
          startedAt: '2026-02-27T17:52:01Z'
      volumeMounts:
        - mountPath: /datadog-lib
          name: datadog-auto-instrumentation
        - mountPath: /opt/datadog-packages/datadog-apm-inject
          name: datadog-auto-instrumentation
        - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
          name: kube-api-access-w4jrd
          readOnly: true
          recursiveReadOnly: Disabled
  phase: Running
  podIP: 10.110.128.42
  podIPs:
    - ip: 10.110.128.42
  qosClass: Burstable
  startTime: '2026-02-27T17:51:59Z'

No DD_TAGS and they (the tags) are not available in Datadog either.

According to documentation here: https://docs.datadoghq.com/tracing/trace_collection/single-step-apm/kubernetes/?tab=agentv764recommended#configure-usts-explicitly-with-ddtraceconfigs

DD_ENV inherited from cluster-level tags above

Leads me to believe that all tags under the datadog.tags attribute should be inherited.

Reproduction Code

No response

@BeyondEvil

@BeyondEvil, thanks for reporting this. This isn't a .NET tracer issue (this repository), and I am not very familiar with these Helm charts, but I tried looking through it to see what I could find 😅

It looks like the DD_TAGS environment variable is only set in the Datadog infrastructure pods (Node Agent, Cluster Agent, Cluster Checks Runner). See the Helm chart's template.

The SSI admission controller, which injects env vars into your application pods, does not include DD_TAGS in its injection set, which is why you don't see it in your instrumented workloads. The admission controller's default config only injects:

DD_TRACE_ENABLED=true
DD_LOGS_INJECTION=true
DD_TRACE_HEALTH_METRICS_ENABLED=true
DD_RUNTIME_METRICS_ENABLED=true

However, you shouldn't need DD_TAGS in the the instrumented workload because the Datadog Agent injects those tags into traces, metrics, etc. Was there a reason you wanted DD_TAGS in the instrumented workload?

As a workaround, it looks like you can use ddTraceConfigs to explicitly inject DD_TAGS into your workloads:

apm:
  instrumentation:
    ddTraceConfigs:
      - name: DD_TAGS
        value: "region:eu-north-1,provider:aws,collector:datadog"

@lucaspimentel

Thanks @lucaspimentel for the detailed response, I really appreciate it! 🙌

@BeyondEvil, thanks for reporting this. This isn't a .NET tracer issue (this repository), and I am not very familiar with these Helm charts, but I tried looking through it to see what I could find 😅

It looks like the DD_TAGS environment variable is only set in the Datadog infrastructure pods (Node Agent, Cluster Agent, Cluster Checks Runner). See the Helm chart's template.

The SSI admission controller, which injects env vars into your application pods, does not include DD_TAGS in its injection set, which is why you don't see it in your instrumented workloads. The admission controller's default config only injects:
DD_TRACE_ENABLED=true
DD_LOGS_INJECTION=true
DD_TRACE_HEALTH_METRICS_ENABLED=true
DD_RUNTIME_METRICS_ENABLED=true

What's weird is that in our other workloads (NodeJS based), the DD_TAGS env var does get injected:

Details

apiVersion: v1
kind: Pod
metadata:
  annotations:
    ad.datadoghq.com/core-api.logs: |-
      [{
        "source": "core-api",
        "service": "core-api",
        "log_processing_rules": [{
          "type": "mask_sequences",
          "name": "mask_emails",
          "replace_placeholder": "[REDACTED EMAIL]",
          "pattern" : "(?:[a-z0-9!#$%&'*+/=?^_`{|}~-]+(?:\\.[a-z0-9!#$%&'*+/=?^_`{|}~-]+)*|\"(?:[\\x01-\\x08\\x0b\\x0c\\x0e-\\x1f\\x21\\x23-\\x5b\\x5d-\\x7f]|\\\\[\\x01-\\x09\\x0b\\x0c\\x0e-\\x7f])*\")(?:@|%40)(?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\\[(?:(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9]))\\.){3}(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9])|[a-z0-9-]*[a-z0-9]:(?:[\\x01-\\x08\\x0b\\x0c\\x0e-\\x1f\\x21-\\x5a\\x53-\\x7f]|\\\\[\\x01-\\x09\\x0b\\x0c\\x0e-\\x7f])+)\\])"
        },
        {
          "type": "mask_sequences",
          "name": "simple_ip_address",
          "replace_placeholder": "[REDACTED IP]",
          "pattern": "(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)"
        }]
      }]
  labels:
    app: core-api
    env: dev
    pod-template-hash: 558b4cd67d
    region: eu-north-1
    tags.datadoghq.com/service: core-api
  name: core-api-558b4cd67d-4wj2v
  namespace: cosafe-dev
spec:
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - podAffinityTerm:
            labelSelector:
              matchLabels:
                app: core-api
            topologyKey: topology.kubernetes.io/zone
          weight: 100
  containers:
    - env:
        - name: DD_TRACE_ENABLED
          value: 'true'
        - name: DD_ENV
          value: dev
        - name: DD_LOGS_INJECTION
          value: 'true'
        - name: DD_TRACE_AGENT_HOSTNAME
          value: datadog.datadog.svc.cluster.local
        - name: DD_TRACE_AGENT_PORT
          value: '8126'
        - name: DD_REMOTE_CONFIGURATION_ENABLED
          value: 'false'
        - name: DD_SERVICE
          value: core-api
      envFrom:
        - secretRef:
            name: core-api
        - secretRef:
            name: token-jwks
      image: ghcr.io/cosafe/core-api:sha-71958b4
      imagePullPolicy: IfNotPresent
      livenessProbe:
        failureThreshold: 60
        httpGet:
          path: /core/v1/health
          port: 8080
          scheme: HTTP
        periodSeconds: 5
        successThreshold: 1
        timeoutSeconds: 1
      name: core-api
      readinessProbe:
        failureThreshold: 3
        httpGet:
          path: /core/v1/health
          port: 8080
          scheme: HTTP
        periodSeconds: 5
        successThreshold: 1
        timeoutSeconds: 2
      resources:
        limits:
          ephemeral-storage: 50Mi
          memory: 512Mi
        requests:
          cpu: 10m
          ephemeral-storage: 25Mi
          memory: 160Mi
      startupProbe:
        failureThreshold: 24
        httpGet:
          path: /core/v1/health
          port: 8080
          scheme: HTTP
        initialDelaySeconds: 5
        periodSeconds: 5
        successThreshold: 1
        timeoutSeconds: 2
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
        - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
          name: kube-api-access-ff7hh
          readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  imagePullSecrets:
    - name: ghcr-login
  nodeName: ip-10-110-128-81.eu-north-1.compute.internal
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
    - effect: NoExecute
      key: node.kubernetes.io/not-ready
      operator: Exists
      tolerationSeconds: 300
    - effect: NoExecute
      key: node.kubernetes.io/unreachable
      operator: Exists
      tolerationSeconds: 300
  topologySpreadConstraints:
    - labelSelector:
        matchLabels:
          app: core-api
      maxSkew: 1
      topologyKey: topology.kubernetes.io/zone
      whenUnsatisfiable: ScheduleAnyway
  volumes:
    - name: kube-api-access-ff7hh
      projected:
        defaultMode: 420
        sources:
          - serviceAccountToken:
              expirationSeconds: 3607
              path: token
          - configMap:
              items:
                - key: ca.crt
                  path: ca.crt
              name: kube-root-ca.crt
          - downwardAPI:
              items:
                - fieldRef:
                    apiVersion: v1
                    fieldPath: metadata.namespace
                  path: namespace
status:
  conditions:
    - lastProbeTime: null
      lastTransitionTime: '2026-02-27T17:38:57Z'
      status: 'True'
      type: PodReadyToStartContainers
    - lastProbeTime: null
      lastTransitionTime: '2026-02-27T17:38:54Z'
      status: 'True'
      type: Initialized
    - lastProbeTime: null
      lastTransitionTime: '2026-02-27T17:39:09Z'
      status: 'True'
      type: Ready
    - lastProbeTime: null
      lastTransitionTime: '2026-02-27T17:39:09Z'
      status: 'True'
      type: ContainersReady
    - lastProbeTime: null
      lastTransitionTime: '2026-02-27T17:38:54Z'
      status: 'True'
      type: PodScheduled
  containerStatuses:
    - containerID: >-
        containerd://8dbdd5a92ed6f60daca24aa8f92ed2ff43a8f02877ed15bed19a1df8b17b1a0e
      image: ghcr.io/company/core-api:sha-71958b4
      imageID: >-
        ghcr.io/company/core-api@sha256:ffe874e95199afe42c6a711b57d432e6af7346ab1bcc46960667c071697327fb
      lastState: {}
      name: core-api
      ready: true
      restartCount: 0
      started: true
      state:
        running:
          startedAt: '2026-02-27T17:38:57Z'
      volumeMounts:
        - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
          name: kube-api-access-ff7hh
          readOnly: true
          recursiveReadOnly: Disabled
  hostIP: 10.110.128.81
  hostIPs:
    - ip: 10.110.128.81
  phase: Running
  podIP: 10.110.130.26
  podIPs:
    - ip: 10.110.130.26
  qosClass: Burstable
  startTime: '2026-02-27T17:38:54Z'

However, you shouldn't need DD_TAGS in the the instrumented workload because the Datadog Agent injects those tags into traces, metrics, etc. Was there a reason you wanted DD_TAGS in the instrumented workload?

Well, that's the thing. It doesn't inject it, so when looking at for example the logs for the dotnet services the DD_TAGS tags (like region) isn't available. But they are available for our NodeJS based workloads.

Hence why I thought the issue we're seeing was related to dd-tracer. I've also found old issues/PRs here related to DD_TAGS parsing.

As a workaround, it looks like you can use ddTraceConfigs to explicitly inject DD_TAGS into your workloads:
apm:
  instrumentation:
    ddTraceConfigs:
      - name: DD_TAGS
        value: "region:eu-north-1,provider:aws,collector:datadog"

Yeah, I was hoping to avoid that as it effectively means I have to repeat what's already under the datadog.tags key.

Thanks for the additional detail! I will pass this along to the team that handles SSI injection and get back to you.

What's weird is that in our other workloads (NodeJS based), the DD_TAGS env var does get injected
[...]

I don't see DD_TAGS there. Am I missing something? 😕

edit: also there are none of the LD_PRELOAD or DD_INSTRUMENTATION_* env vars. Is this Node.js app instrumented using Single Step Instrumentation (SSI)? It looks like the .NET app and the Node.js app are using different deployment/instrumentation approaches.

What's weird is that in our other workloads (NodeJS based), the DD_TAGS env var does get injected
[...]

I don't see DD_TAGS there. Am I missing something? 😕

You're right. I could've swore it was there.

Regardless, the nodejs apps have the datadog.tags tags in Datadog logs (like region).

edit: also there are none of the LD_PRELOAD or DD_INSTRUMENTATION_* env vars. Is this Node.js app instrumented using SSI?

No, the nodejs apps are not instrumented using SSI. Which is why I suspect the issue lies with APM/dd-tracer.

No APM (nodejs): datadog.tags present on logs.
APM (dotnet): datadog.tags not present on logs.

So, yesterday I spent about 2 hours in my IDE with Claude Opus 4.6 trying to figure out why the hell our .NET services won't get the datadog.tags tags.

And I would've never figured this out on my own. 😅

Here's the log4net.config file we use for all of our .NET services:

<?xml version="1.0" encoding="utf-8" ?>
<log4net>
	<!-- Console Appender for logging to stdout -->
	<appender name="ConsoleAppender" type="log4net.Appender.ConsoleAppender">
		<layout type='log4net.Layout.SerializedLayout, log4net.Ext.Json'>
			<decorator type='Grannkompaniet.Infrastructure.Extensions.Log4NetBunyanDecorator, Grannkompaniet.Email' />
			<member value="name" />
		    <member value="hostname" />
            <member value="pid" />
			<member value="route" />
			<member value="req_id" />
			<member value="level" />
			<member value="levelStr" />
			<member value="userId" />

			<member value="req" />
			<member value="res" />

			<member value="message" />
			<member value="exception" />
			<member value="date" />
			<member value="time" />
			<member value="requestStart" />
			<member value="duration" />
			<member value="v" />
		</layout>
	</appender>
	<root>
		<appender-ref ref="ConsoleAppender" />
	</root>
</log4net>

Can you spot the problem?

spoiler:

<member value="hostname" />

That line means that the hostname for the log lines from the services becomes the pod name (ex. email-5cfdf6bd49-5dhxd), while the datadog.tags tags are matched with the underlying hostname (the Kubernetes node). So Datadog doesn't know they "belong together".

So simply removing that line from the config solved it!

Navigation Menu

Repository navigation

[BUG]: DD_TAGS from Cluster Agent not injected #8250

Tracer Version(s)

Operating system and platform

Instrumentation Mode

TFM

Bug Report

DD_ENV inherited from cluster-level tags above

Reproduction Code

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[BUG]: DD_TAGS from Cluster Agent not injected #8250

Description

Tracer Version(s)

Operating system and platform

Instrumentation Mode

TFM

Bug Report

DD_ENV inherited from cluster-level tags above

Reproduction Code

Activity

lucaspimentel commented on Feb 27, 2026

BeyondEvil commented on Feb 27, 2026

lucaspimentel commented on Feb 27, 2026

lucaspimentel commented on Feb 27, 2026

BeyondEvil commented on Feb 27, 2026

BeyondEvil commented on Feb 28, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions