feat: add support for Firestore Pipeline - #292
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces Firestore Pipeline operations to the google_cloud_firestore package, enabling server-side projections, expressions, aggregates, and vector search, complete with comprehensive E2E and unit tests. The review feedback highlights several key improvement opportunities: reverting a breaking change to the DistanceMeasure enum by converting values to lowercase locally within the pipeline execution, optimizing performance by extracting a frequently compiled regular expression into a file-level constant, and adding as well as exporting missing top-level comparison helpers (lessThan and greaterThan) to ensure API completeness.
Coverage Report✅ Coverage 73.15% meets 40% threshold Total Coverage: 73.15% Package Breakdown
Minimum threshold: 40% |
There was a problem hiding this comment.
PipelineFunctions accepts Object? but only Expression.field() actually works without a fieldOrExpression-style check — untested since every call site already used field(). We should either commit to requiring Expression.field() explicitly or handle the coercion like Node's fieldOrExpression.
Arguments in a "field or expression" position kept String values as string
literals, so `PipelineFunctions.startsWith('title', 'Harry')` compared the
literal text "title" against "Harry" instead of reading the `title` field.
Add `_fieldOrExpression`, mirroring the Node SDK's `fieldOrExpression`, and
apply it across the function catalog. Value positions still keep Strings as
literals, and document paths stay values as they do in Node.
Also:
- Add `PipelineSource.createFrom()` to convert a `Query` or `VectorQuery`
into an equivalent Pipeline, translating filters, projections, implicit
orderings, cursors, limit/limitToLast and offset.
- Add expressions released since the initial port: `coalesce`, `length`,
`reverse`, `concat`, `getField`, `geoDistance`, `documentMatches` and
`score`, plus `logicalMinimum`/`logicalMaximum`.
- Align `PipelineExpression.length()` and `.concat()` with the Node SDK's
generic `length`/`concat` backend functions, and expose the
string-specific `charLength()`/`stringConcat()` alongside them.
- Rename field-position parameters to `fieldName` to match Node.
- Hide `greaterThan`/`lessThan` from the Firestore import in two suites that
want the `matcher` versions.
…dings Four Pipeline stages diverged from the backend contract, verified against the canonical Node SDK stage definitions in dev/src/pipelines/stage.ts: - `unnest` sent only the array expression, so there was no way to name the emitted element. It now sends `[expr, field(alias)]`, taking the alias from the selectable, and encodes `index_field` as a field reference rather than a string. - `replace_with` omitted the required mode argument; it now sends `[map, 'full_replace']`. - `sample` passed the rate as a `documents`/`percentage` option; it now sends `[rate, mode]` with mode `documents` or `percent`. - `distinct` sent a positional list of expressions; it now sends a single map keyed by alias, reusing the same projection map as `select` and `aggregate`. `unnest` and `sample` are breaking signature changes. Also make `PipelineFunctions.minimum`/`maximum` aggregate-only, matching Node, now that `logicalMinimum`/`logicalMaximum` cover the element-wise form. Adds golden proto tests asserting each stage's arguments and options, so this class of wire-format drift is caught without an Enterprise database.
The README covered 4 of the ~16 Pipeline stages and the example directory had no pipeline content at all. - Add `example/pipeline_example.dart` covering every stage: source/filter/ sort/project/limit, aggregates with and without grouping, `unnest`, `replaceWith` with `addFields`/`removeFields`, `distinct`, `sample`, `union`, `findNearest`, and `createFrom`. Each example seeds and cleans up its own documents, and carries `[START]`/`[END]` region markers for docs ingestion. - Expand the README: every source and stage with a runnable snippet, a function reference table mapping Dart helpers to backend function names, the field-argument vs value-argument rule, execution options, the `PipelineSnapshot`/`PipelineResult` surface, and a Query migration guide. - Note the Enterprise-edition requirement and the failure behavior on `Pipeline.execute()` and `Firestore.pipeline()`. - Add `example/README.md` so both examples are discoverable, and so pub.dev's Example tab shows representative code.
`equalAny('dart', Expression.field('tags'))` passed a bare String in a
field position, where a String means a field reference rather than a
string literal (mirroring the Node SDK's fieldOrExpression). It therefore
asked about a non-existent `dart` field, so equalAny was always false --
and the adjacent notEqualAny passed for exactly the same wrong reason, a
missing field trivially not equalling anything. Both now target `title`
with array literals so they exercise the real semantics.
Also require FIRESTORE_PIPELINE_E2E_DATABASE_ID explicitly rather than
falling back to '(default)'. CI credential helpers such as
google-github-actions/auth export GOOGLE_CLOUD_PROJECT, so the previous
project-only guard could silently arm the suite against a default
database that lacks Pipelines support or this suite's vector index.
Adds e2e_pipeline.yml, which runs test/e2e/pipeline_e2e_test.dart against the Enterprise-edition firestore-pipeline-test database that FlutterFire's e2e_tests_pipeline.yaml also targets, so both SDKs are validated against one shared Pipelines database. The suite existed but nothing ran it. Triggers on pull requests touching packages/google_cloud_firestore, on pushes to main, nightly, and on demand. Fork and dependabot pull requests are skipped because they receive no secrets. It is a separate workflow rather than a build.yml job so the live-quota cost is only paid for changes that can affect it. `dart test` exits 0 when every test is skipped, which is exactly what a missing project or database ID produces, so the job fails if the secret is empty and again if the run reports "All tests skipped" -- otherwise a misconfigured job would report green for tests that never executed. Authenticates as a dedicated service account holding roles/datastore.user in the Pipelines project. The key is written to $RUNNER_TEMP rather than the checkout and removed in an always() step.
The `documents` source stage routed each DocumentReference through
_encodePipelineValue, which emits the full resource name
(projects/{p}/databases/{d}/documents/books/book-1). Source stages name
resources by their path relative to the database instead: the `collection`
stage one branch above already emitted `/${path}`, and Node's
DocumentsSource encodes `'/' + ref.path`. The full name is only correct
for references in a value position, so the path is now built at the stage
rather than in the shared value encoder.
Extracts the shared `_relativeReference` helper, updates the unit test,
and adds E2E coverage for the stage, which was previously untested against
a live database.
Add support for Firestore Pipeline
How to use
Aggregates
Aggregate stages use aliased aggregate expressions:
Expressions
Use
Expression.field,Expression.constant, andExpression.variabletobuild expressions. Most helpers are also available as fluent methods: