[logstash] Enable TSDB for metrics data streams by AndersonQ · Pull Request #17401 · elastic/integrations

AndersonQ · 2026-02-12T16:51:25Z

Proposed commit message

[logstash] Enable TSDB for metrics data streams

Enable TSDB for health_report and node_cel data streams in the Logstash 
integration and add metric_type annotations to pipeline and plugins.

Dimensions added:
- health_report: logstash.node.name, logstash.node.uuid, logstash.pipeline.id
- node_cel: logstash.node.stats.logstash.name
- pipeline: logstash.pipeline.host.name
- plugins: logstash.pipeline.host.name

Annotate numeric fields with appropriate metric_type for the health_report,
node_cel, pipeline, plugins data streams.

metric_type corrections (counter → gauge):
- node_cel: jvm.threads.count, jvm.threads.peak_count, jvm.mem.heap_max_in_bytes,
queue.events_count
- pipeline: logstash.pipeline.queues.events
- plugins: beats.peak_connections, beats.current_connections

Assisted by Cursor

Data stream changes summary

Data Stream: `health_report`

Dimensions Added

Field	Type
`logstash.node.name`	keyword
`logstash.node.uuid`	keyword
`logstash.pipeline.id`	keyword

Fields with metric_type Added

Field	Type	metric_type
`logstash.pipeline.impacts.severity`	short	gauge
`logstash.pipeline.flow.worker_utilization.current`	float	gauge
`logstash.pipeline.flow.worker_utilization.last_1_hour`	float	gauge
`logstash.pipeline.flow.worker_utilization.last_5_minutes`	float	gauge
`logstash.pipeline.flow.worker_utilization.last_15_minutes`	float	gauge
`logstash.pipeline.flow.worker_utilization.lifetime`	float	gauge
`logstash.pipeline.flow.worker_utilization.last_1_minute`	float	gauge
`logstash.pipeline.flow.worker_utilization.last_24_hours`	float	gauge

Data Stream: `node_cel`

Dimensions Added

Field	Type
`logstash.node.stats.logstash.name`	keyword

Fields with metric_type Added

Field	Type	metric_type
`logstash.node.stats.pipelines.reloads.failures`	long	counter
`logstash.node.stats.pipelines.reloads.successes`	long	counter
`logstash.node.stats.pipelines.queue.events_count`	long	gauge
`logstash.node.stats.pipelines.queue.queue_size_in_bytes`	long	gauge
`logstash.node.stats.pipelines.queue.max_queue_size_in_bytes`	long	gauge
`logstash.node.stats.pipelines.events.in`	long	counter
`logstash.node.stats.pipelines.events.out`	long	counter
`logstash.node.stats.pipelines.events.filtered`	long	counter
`logstash.node.stats.pipelines.events.duration_in_millis`	long	counter
`logstash.node.stats.pipelines.events.queue_push_duration_in_millis`	long	counter

Fields with metric_type Changed

Field	Type	Old metric_type	New metric_type
`logstash.node.stats.jvm.threads.count`	long	counter	gauge
`logstash.node.stats.jvm.threads.peak_count`	long	counter	gauge
`logstash.node.stats.jvm.mem.heap_max_in_bytes`	long	counter	gauge
`logstash.node.stats.queue.events_count`	long	counter	gauge

Data Stream: `pipeline`

Dimensions Added

Field	Type
`logstash.pipeline.host.name`	keyword

Fields with metric_type Added

Field	Type	metric_type
`logstash.pipeline.info.batch_size`	long	gauge
`logstash.pipeline.info.batch_delay`	long	gauge
`logstash.pipeline.info.workers`	long	gauge

Fields with metric_type Changed

Field	Type	Old metric_type	New metric_type
`logstash.pipeline.queues.events`	long	counter	gauge

Data Stream: `plugins`

Dimensions Added

Field	Type
`logstash.pipeline.host.name`	keyword

Fields with metric_type Changed

Field	Type	Old metric_type	New metric_type
`logstash.pipeline.plugin.input.metrics.beats.peak_connections`	long	counter	gauge
`logstash.pipeline.plugin.input.metrics.beats.current_connections`	long	counter	gauge

Open questions

As the data streams are migrated to TSDB they do not support metrics which documents
containing the error when fetching a metric as those would not have the necessary
dimensions. When testing, I started the agent before logstash was ready, getting
a few errors, which when testing the migration with https://github.com/elastic/TSDB-migration-test-kit
could not be migrated.

If we want to keep this behaviour, I don't think the data stream can be migrated
to TSDB.

On my tests, the error events were present on health_report and on node_cel
data streams.

Here is an example of the error documents:

{"error": {"type": "illegal_argument_exception", "reason": "Error extracting routing: source didn't contain any routing fields"}, "status": 400, "document": {"cloud": {"availability_zone": "us-central1-f", "instance": {"name": "anderson-logstash", "id": "8077647130981829769"}, "provider": "gcp", "service": {"name": "GCE"}, "machine": {"type": "e2-standard-4"}, "project": {"id": "elastic-observability"}, "region": "us-central1", "account": {"id": "elastic-observability"}}, "input": {"type": "cel"}, "agent": {"name": "anderson-logstash", "id": "27ad10fc-cef3-424a-879c-23721c867517", "type": "filebeat", "ephemeral_id": "c2cfc104-c50f-4906-af6e-7ddcf911012a", "version": "8.17.10"}, "@timestamp": "2026-02-12T11:32:41.562Z", "ecs": {"version": "8.0.0"}, "data_stream": {"namespace": "default", "type": "metrics", "dataset": "logstash.node"}, "elastic_agent": {"id": "27ad10fc-cef3-424a-879c-23721c867517", "version": "8.17.10", "snapshot": false}, "host": {"hostname": "anderson-logstash", "os": {"kernel": "6.1.0-43-cloud-amd64", "codename": "bookworm", "name": "Debian GNU/Linux", "type": "linux", "family": "debian", "version": "12 (bookworm)", "platform": "debian"}, "containerized": false, "ip": ["10.128.0.72", "fe80::4001:aff:fe80:48"], "name": "anderson-logstash", "id": "370ef8b742434f90a470fd961035344e", "mac": ["42-01-0A-80-00-48"], "architecture": "x86_64"}, "error": {"message": "failed eval: ERROR: <input>:7:7: Get \"http://localhost:9600/_node/stats?graph=true&vertices=true\": dial tcp [::1]:9600: connect: connection refused\n |       ? {\n | ......^"}, "event": {"agent_id_status": "verified", "ingested": "2026-02-12T11:32:50Z", "dataset": "logstash.node"}}}

One option could be to have the agent.id as dimension, that way the error events
could be indexed, having their TSDB ID from the agent.id+timestamp, what seems
ok. However, it means additional mappings and dimensions for this corner case.

I checked and the errors are still logged and appear on the agent dashboards
which show errors, like the "concerning agents". So, the errors aren't lost,
nevertheless, it's still a breaking change if anyone would rely on documents with
the error key to know something is wrong.

Checklist

~~[ ] I have reviewed tips for building integrations and this pull request is aligned with them.~~
I have verified that all data streams collect metrics or logs.
~~[ ] I have added an entry to my package's changelog.yml file.~~
~~[ ] I have verified that Kibana version constraints are current according to guidelines.~~
~~[ ] I have verified that any added dashboard complies with Kibana's Dashboard good practices~~

How to test this PR locally

I used a modified version
of the TSDB migration test kit
to test the migration of the data streams to TSDB. It reindex the source index
to the destination index. This approach fails if there are documents that cannot
be ingested into the TSDB. For example the "error" events I mentioned above.
Thus, I modified it to scan the source index and use the bulk API to index the
documents, saving the failed and duplicated documents into 2 different files.

You may try both versions of the test kit.

click to show instructions

set up an elastic stack version 8.17.10 (8.17 is the lowest version supported)

Agent + Logstash output setup

Let's setup 2 nodes. All paths are relative to the Logstash directory

logstash: node 1:

agent.conf:

input {
  elastic_agent {
    port => 5044
    enrich => none
    ssl_enabled => false
  }
}

output {
  elasticsearch {
    cloud_id => "cloudID"
    data_stream => true
    ssl_enabled => true
    user => "elastic"
    password => "changeme"
  }
}

config/logstash.yml:

node.name: logstash-node-1

log.level: debug 
log.format.json.fix_duplicate_message_fields: true

path.logs: /tmp/logstash-node-1/logs

cd logstasn-node-1
./bin/logstash -f agent.conf

logstash: node 2:

agent.conf:

input {
  elastic_agent {
    port => 5045
    enrich => none
    ssl_enabled => false
  }
}

output {
  elasticsearch {
    cloud_id => "cloudID"
    data_stream => true
    ssl_enabled => true
    user => "elastic"
    password => "changeme"
  }
}

config/logstash.yml:

node.name: logstash-node-2
api.http.port: 9601

log.level: debug 
log.format.json.fix_duplicate_message_fields: true

path.logs: /tmp/logstash-node-2/logs

cd logstasn-node-2
./bin/logstash -f agent.conf

Elastic Agent

create a agent policy
add a logstash output with hosts: ["localhost:5044", "localhost:5045"]
set the output as the integrations output
generate some logs: flog -t log -o /tmp/agent/in/log.ndjson -w -f json -l -p 1048576 -d 500ms
setup a filestream input to collect the generated logs
install the agent with --namespace logstash-output

Elastic Agent with Logstash integration

create an agent policy
add 2 logstash integrations, each to monitor a node:
- integration 1:
  - Metrics Metrics (Elastic Agent) -> Logstash URL: http://localhost:9600
  - Logs paths: /tmp/logstash-node-1/logs/logstash-plain*.log
  - Slowlogs paths: /tmp/logstash-node-1/logs/logstash-slowlog-plain*.log
  - Metrics (Stack Monitoring) -> Hosts http://localhost:9600
- integration 1:
  - Metrics Metrics (Elastic Agent) -> Logstash URL: http://localhost:9601
  - Logs paths: /tmp/logstash-node-2/logs/logstash-plain*.log
  - Slowlogs paths: /tmp/logstash-node-2/logs/logstash-slowlog-plain*.log
  - Metrics (Stack Monitoring) -> Hosts http://localhost:9601
- install the agent with --namespace monitoring

Verify

verify the agent-logstash-output is sending data, check there are logs from
path /tmp/agent/in/log.ndjson
verify the agent-monitoring is collecting data. Check the logstash dashboards.
I recomend to let it run for a good while, so there will be a good amount of data
https://github.com/elastic/TSDB-migration-test-kit can use when testing the TSDB
migration.
build and install the integration from this PR: elastic-package build -v && elastic-package install -v
run TSDB-migration-test-kit for each one of the changed data streams
check the dashboards are working fine with the old and new data

Related issues

Closes [Logstash integration] Migrate datas treams to TSDB. #16512

Enable TSDB for health_report and node_cel data streams in the Logstash integration and add metric_type annotations to pipeline and plugins. Dimensions added: - health_report: logstash.node.name, logstash.node.uuid, logstash.pipeline.id - node_cel: logstash.node.stats.logstash.name - pipeline: logstash.pipeline.host.name - plugins: logstash.pipeline.host.name Annotate numeric fields with appropriate metric_type for the health_report, node_cel, pipeline, plugins data streams. metric_type corrections (counter → gauge): - node_cel: jvm.threads.count, jvm.threads.peak_count, jvm.mem.heap_max_in_bytes, queue.events_count - pipeline: logstash.pipeline.queues.events - plugins: beats.peak_connections, beats.current_connections Assisted by Cursor

elasticmachine · 2026-02-12T17:10:49Z

💔 Build Failed

Buildkite Build
Commit: 6cbf6df

Failed CI Steps

Check integrations logstash

History

cc @AndersonQ

AndersonQ self-assigned this Feb 12, 2026

AndersonQ added Integration:logstash Logstash Team:Elastic-Agent-Data-Plane Agent Data Plane team [elastic/elastic-agent-data-plane] labels Feb 12, 2026

AndersonQ force-pushed the 16512-logstash-metrics-TSDB branch from 1b1af21 to 6cbf6df Compare February 12, 2026 16:53

AndersonQ changed the title ~~WIP: [logstash] Enable TSDB for metrics data streams~~ [logstash] Enable TSDB for metrics data streams Feb 13, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[logstash] Enable TSDB for metrics data streams#17401

[logstash] Enable TSDB for metrics data streams#17401
AndersonQ wants to merge 1 commit intoelastic:mainfrom
AndersonQ:16512-logstash-metrics-TSDB

AndersonQ commented Feb 12, 2026 •

edited

Loading

Uh oh!

elasticmachine commented Feb 12, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

AndersonQ commented Feb 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Data Stream: health_report

Dimensions Added

Fields with metric_type Added

Data Stream: node_cel

Dimensions Added

Fields with metric_type Added

Fields with metric_type Changed

Data Stream: pipeline

Dimensions Added

Fields with metric_type Added

Fields with metric_type Changed

Data Stream: plugins

Dimensions Added

Fields with metric_type Changed

Open questions

Checklist

How to test this PR locally

Agent + Logstash output setup

logstash: node 1:

logstash: node 2:

Elastic Agent

Elastic Agent with Logstash integration

Verify

Related issues

Uh oh!

elasticmachine commented Feb 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💔 Build Failed

Failed CI Steps

History

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

AndersonQ commented Feb 12, 2026 •

edited

Loading

Data Stream: `health_report`

Data Stream: `node_cel`

Data Stream: `pipeline`

Data Stream: `plugins`

elasticmachine commented Feb 12, 2026 •

edited

Loading