feat(enrichment tables): add cuckoo filter to memory table

[esensar]

Summary

This adds support for cuckoo filters in memory enrichment tables, to support use cases where only presence of a key needs to be checked and false positives are acceptable, greatly improving memory usage compared to regular memory tables.

Bloom filters should be fairly easy to add as well (and will be done in a separate PR), but cuckoo is a better fit, because it supports deletion, especially using cuckoo-clock lib which extends cuckoo filter with TTL and more, which fits memory enrichment tables.

Vector configuration

enrichment_tables:
  cuckoo_table:
    type: memory
    ttl: 60
    flush_interval: 5
    scan_interval: 10
    inputs: ["cuckoo_generator"]
    filter:
      type: cuckoo
      max_entries: 100000
      ttl_enabled: true
      counter_enabled: true
      counter_bits: 4
      # since export keys back out of cuckoo filter is not possible, 
      # this component provides a way to persist whole filter state and reload it on restart
      persistence_path: /etc/vector/persisted-cuckoo-filter
      export_interval: 1

sources:
  data_for_table:
    type: file
    # Just a list of JSON lines in format {"key":"something"}
    include: ["/etc/vector/vector-cuckoo-memory-example-input.jsonl"]

  stdin_data:
    type: stdin

transforms:
  cuckoo_reader:
    type: "remap"
    inputs: ["stdin_data"]
    source: |
      key = .message

      existing, err = get_enrichment_table_record("cuckoo_table", { "key": key })

      if err == null {
        . = existing
      } else {
        .message = "Key not found"
      }

  cuckoo_generator:
    type: "remap"
    inputs: ["data_for_table"]
    source: |
      data = parse_json!(.message)
      . = set!(value: {}, path: [get!(data, path: ["key"])], data: { "ttl": 500 })

sinks:
  console:
    inputs: ["cuckoo_reader"]
    target: "stdout"
    type: "console"
    encoding:
      codec: "json"

How did you test this PR?

Ran the above configuration and looked up the keys using stdin source, by entering the keys to look up. Some unit tests were added as well.

Change Type

• Bug fix
• New feature
• Dependencies
• Non-functional (chore, refactoring, docs)
• Performance

Is this a breaking change?

• Yes
• No

Does this PR include user facing changes?

• Yes. Please add a changelog fragment based on our guidelines.
• No. A maintainer will apply the no-changelog label to this PR.

References

• Related: Cuckoo filter + LRU - seems like a natural match for Vector? #23595 (reply in thread)

Notes

• Please read our Vector contributor resources.
• Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
• Some CI checks run only after we manually approve them.
  • We recommend adding a pre-push hook, please see this template.
  • Alternatively, we recommend running the following locally before pushing to the remote branch:
    • make fmt
    • make check-clippy (if there are failures it's possible some of them can be fixed with make clippy-fix)
    • make test
• After a review is requested, please avoid force pushes to help us review incrementally.
  • Feel free to push as many commits as you want. They will be squashed into one before merging.
  • For example, you can run git merge origin master and git push.
• If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
  run make build-licenses to regenerate the license inventory and commit the changes (if any). More details on the dd-rust-license-tool.

---

Sponsored by Quad9

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c4e16f8ff3

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4fe95ea743

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Handle failed cuckoo inserts before acking events

handle_value discards the return value of insert_if_not_present_with_update/insert_if_not_present, so the sink marks events as delivered and emits memory_enrichment_table_insertions_total even when the filter cannot accept a key (for example once max_entries is reached without LRU). This creates silent data loss and misleading internal metrics under normal high-cardinality workloads; the insert result should be checked and failures surfaced (and counted) before acknowledging success.

Useful? React with 👍 / 👎.

[esensar]

The insertion will always succeed, it may just evict another item (which will be returned, but we can't do anything useful with it), meaning insertion of that item is successful, it is just possible that some other item is now missing, which is expected from cuckoo filter.