Skip to content

feat(new sink): add YDB sink support#25219

Open
1NepuNep1 wants to merge 18 commits intovectordotdev:masterfrom
1NepuNep1:ydb-sink
Open

feat(new sink): add YDB sink support#25219
1NepuNep1 wants to merge 18 commits intovectordotdev:masterfrom
1NepuNep1:ydb-sink

Conversation

@1NepuNep1
Copy link
Copy Markdown

Summary

This PR introduces a new YDB (Yandex Database) sink for Vector, enabling users to send log and trace data directly to YDB tables using the official ydb-rs-sdk.

Key features:

  • Automatic table schema detection and type mapping
  • Insertion strategy selection (bulk_upsert for high throughput, UPSERT for indexed tables)
  • Automatic schema refresh on table structure changes
  • Health check with table existence and permissions validation

YDB is a distributed SQL database with horizontal scalability and strong consistency, ideal for high-throughput log storage and real-time analytics.

Vector configuration

sources:
  demo_logs:
    type: demo_logs
    format: json
    interval: 1.0

transforms:
  remap:
    type: remap
    inputs: ["demo_logs"]
    source: |
      .timestamp = now()
      .level = "info"
      .message = string!(.message)

sinks:
  ydb_sink:
    type: ydb
    inputs: ["remap"]
    endpoint: "grpc://localhost:2136?database=/local"
    table: "/local/logs"
    batch:
      max_events: 1000
      timeout_secs: 10

How did you test this PR?

Local Testing

  • Unit tests: cargo test --lib --features sinks-ydb ydb::
  • Code quality: make fmt, cargo clippy, make build-licenses
  • Documentation: make generate-component-docs, CI=true make check-docs
  • Changelog: ./scripts/check_changelog_fragments.sh

Integration Testing

Tested against ydbplatform/local-ydb:25.3 Docker image with various table schemas (simple, indexed, NOT NULL constraints). Verified both insertion strategies, health checks, and schema refresh functionality.

All integration tests configured for CI and passing.

Change Type

  • Bug fix
  • New feature
  • Dependencies
  • Non-functional (chore, refactoring, docs)
  • Performance

Is this a breaking change?

  • Yes
  • No

Does this PR include user facing changes?

  • Yes. Please add a changelog fragment based on our guidelines.
  • No. A maintainer will apply the no-changelog label to this PR.

References

Notes

Known Limitations:

  1. VRL Optional Values: Optional fields from VRL transforms are skipped with warning logs due to YDB SDK internal type representation. Does not affect normal operation.

  2. Event Types: Supports Log and Trace events only. Metric events are not supported.

@1NepuNep1 1NepuNep1 requested review from a team as code owners April 18, 2026 10:08
@github-actions github-actions bot added work in progress domain: sinks Anything related to the Vector's sinks domain: ci Anything related to Vector's CI environment domain: external docs Anything related to Vector's external, public documentation labels Apr 18, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 18, 2026

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@1NepuNep1
Copy link
Copy Markdown
Author

I have read the CLA Document and I hereby sign the CLA

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

domain: ci Anything related to Vector's CI environment domain: external docs Anything related to Vector's external, public documentation domain: sinks Anything related to the Vector's sinks work in progress

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant