Elasticsearch sink: `ComponentEventsDropped`/`CallError` reports the full pre-retry batch size instead of the actual dropped-item count after `RetryPartial`

[vladislavlopatin255]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

When an Elasticsearch bulk request has a mix of successful, retriable, and non-retriable items, and the sink correctly narrows retries down to just the non-retriable subset via the RetryPartial mechanism (added in d9bef7e), the eventual "Events dropped" internal log/metric (vector_common::internal_event::component_events_dropped, emitted via CallError in lib/vector-stream/src/driver.rs) reports the size of the original, pre-retry batch, not the actual number of items dropped in the narrowed request.

Root cause (traced in the v0.51.1 source), in lib/vector-stream/src/driver.rs, Driver::run():

let mut req = batch.pop_front()...;
...
let event_count = req.get_metadata().event_count();  // captured from the ORIGINAL request, before any retries

let fut = svc.call(req)
    .err_into()
    .map(move |result| Self::handle_response(
        result, request_id, finalizers, event_count, ...
    ))

event_count (and finalizers) are captured once, before svc.call(req) is invoked. All of the RetryPartial narrowing logic (see src/sinks/elasticsearch/retry.rs, which reconstructs a smaller request from failed_events) happens inside that single svc.call() call - the tower-wrapped retry policy loops internally and returns only one final Result to the Driver. The Driver has no visibility into how much the request was narrowed during retries, so handle_response() -> emit_call_error() always reports the original event_count, even when only a handful of items in a 1500-2000 event batch were actually undeliverable.

We confirmed empirically, in a real pipeline, that real document loss tracks 1:1 with individual non-retryable items (each with its own ES document _id in the per-item "Not retriable; dropping the request." log), not with the full batch size - comparing Kafka consumer-group offset advance against real Elasticsearch document-count growth over the same window showed no meaningful gap. So delivery is correct (thanks to the RetryPartial fix), but the dropped-event telemetry wasn't updated to match, and badly overstates real data loss for any sink using this retry pattern.

Expected: the reported dropped-event count should reflect only the events actually dropped after partial-retry narrowing, not the original full batch size.

Configuration

sinks:
  elasticsearch_out:
    type: elasticsearch
    inputs: ["some_transform"]
    endpoints: ["https://es.example.internal:9200"]
    mode: data_stream
    api_version: v8
    bulk:
      action: create
    batch:
      max_events: 2000
    request:
      concurrency: 40
      retry_attempts: 20
      retry_initial_backoff_secs: 2
      retry_max_duration_secs: 1800

Version

0.51.1

Debug Output

Example Data

Example internal log lines from the same occurrence (redacted):

{"text":"Not retriable; dropping the request.","reason":"error type: document_parsing_exception, reason: [1:104] failed to parse field [attr.user] of type [keyword] in document with id '...'. Preview of field's value: '{user=..., db=}'","vector_component_id":"elasticsearch_out","vector_component_kind":"sink"}

{"text":"Events dropped","count":"1473","intentional":"false","reason":"Service call failed. No retries or retries exhausted.","vector_component_id":"elasticsearch_out","vector_component_kind":"sink"}

Each "Not retriable" line references exactly one specific document _id. The paired "Events dropped" line's count is consistently close to the configured batch.max_events (2000), not 1 - and per-minute, the row-count of "Not retriable" occurrences matches the row-count of "Events dropped" occurrences almost exactly (1:1 pairing), which is how we traced the two together.

Additional Context

Not running in Kubernetes, no unusual env vars/CLI flags. We noticed this while building an internal ES-reject dashboard and initially assumed we had a massive real data-loss problem (the count-based aggregate implied roughly half of all traffic was being dropped). We ruled that out by comparing Kafka consumer-group offset advance against real Elasticsearch document-count growth (via _cat/indices) over the same time window on the affected hosts - the two tracked each other closely, with no meaningful gap, confirming actual delivery is fine and the discrepancy is purely in the reported count.

References

#18451
#10870