Skip to content

[Chore] Keep Stream filtering and projection on batches until output #14067

Description

@hanahmily

Summary

Stop constructing protobuf Element objects merely to filter rows or remove internally projected tags. This follows the legacy-engine removal child #14066 under #13998.

Size audit and dependencies

Classification: tracking parent, not an executable leaf or automated implementation task.
Boundary: Stream batch filtering and client projection through banyand/query/processor.go, with eligible frame egress and required protobuf egress.
It spans filter placement, projection, and response-path activation. Split into executable tasks against the merged implementation rather than assigning the entire workstream as one PR.
Blocked by: #14066, implemented by BanyanDB PR apache/skywalking-banyandb#1326 (still open when this issue was drafted).
Prerequisite already merged: BanyanDB PR apache/skywalking-banyandb#1331 provides pre-merge filtering for index-order queries.

Requirements

R1. Reuse apache/skywalking-banyandb#1331's pre-merge index-order filtering and remove the redundant element-level recheck. Preserve filter-first duplicate winners and the bounded merge.
R2. Apply timestamp-order filtering to batches at its existing position after the cap; do not change its existing under-fill behavior.
R3. Strip criteria-only and hidden ordering tags through batch projection. Retain internal ordering keys until ordering is finished, without exposing hidden tags in the client's projection.
R4. Enable frame output for eligible distributed queries. Preserve required protobuf responses, including traced responses and final client output.

Concrete oracle

Input:

ID=A, order=1, state=closed, service=old
ID=A, order=2, state=open,   service=new
ID=B, order=3, state=open,   service=other

With criteria state=open, ascending index order, and projection service, return A/new then B/other. Neither state nor the ordering tag may appear in client tag families. Offset 1, limit 1 returns B/other. Expected values are specified here, not derived from a legacy engine.

Completion criteria and RED-test direction

  • Real processor tests prove eligible filtered queries emit frames. Today their filter gate selects protobuf output, so this is an observable failing path assertion, not only output parity.
  • Exact selected values, duplicate winners, hidden-tag removal, null behavior, and offset/limit are tested across batches.
  • Timestamp-order filtering remains behind the cap, and traced responses retain tracing information.
  • Each implementation leaf activates a production caller and includes its own failing test; batch ownership is correct on success, cancellation, and errors.
go test ./pkg/query/vectorized/stream/... ./pkg/query/logical/stream ./banyand/query
go test ./test/integration/standalone/query/... ./test/integration/distributed/query/...

Out of scope

Reimplementing apache/skywalking-banyandb#1331; changing timestamp under-fill semantics; local cross-group or distributed result merging (a separate workstream).

Parent: #13998.

Compatibility and review policy

  • Preserve existing on-disk and wire formats and mixed-version compatibility. Any necessary protocol change requires a separately reviewed compatibility contract.
  • Use explicit, hand-calculated expected results; do not depend on the removed row engine as the test oracle.
  • Human review enforces the no-new-row-path policy. A row-call-site lint guard is not required.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    databaseBanyanDB - SkyWalking native database

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions