Skip to content

feat: add support for Firestore Pipeline - #292

Merged
demolaf merged 12 commits into
mainfrom
feat/pipeline
Aug 31, 2026
Merged

feat: add support for Firestore Pipeline#292
demolaf merged 12 commits into
mainfrom
feat/pipeline

Conversation

@Lyokone

@Lyokone Lyokone commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Add support for Firestore Pipeline

How to use

final snapshot = await firestore
    .pipeline()
    .collection('books')
    .where(Expression.field('active').equalValue(true))
    .sort([Expression.field('price').ascending()])
    .select([
      Expression.field('title'),
      Expression.field('price'),
      Expression.field('title').toUpperCase().as('upperTitle'),
      Expression.field('tags').arrayLength().as('tagCount'),
    ])
    .limit(10)
    .execute();
for (final result in snapshot.results) {
  print(result.data());
}

Aggregates

Aggregate stages use aliased aggregate expressions:

final snapshot = await firestore
    .pipeline()
    .collection('books')
    .where(Expression.field('active').equalValue(true))
    .aggregate([
      Expression.field('price').sum().as('totalPrice'),
      Expression.field('rating').average().as('averageRating'),
      PipelineFunctions.count().as('bookCount'),
    ])
    .execute();
final data = snapshot.results.single.data();
print(data);

Expressions

Use Expression.field, Expression.constant, and Expression.variable to
build expressions. Most helpers are also available as fluent methods:

final expression = Expression.field('createdAt')
    .timestampSubtract('day', 7)
    .timestampToUnixSeconds()
    .as('createdSeconds');

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces Firestore Pipeline operations to the google_cloud_firestore package, enabling server-side projections, expressions, aggregates, and vector search, complete with comprehensive E2E and unit tests. The review feedback highlights several key improvement opportunities: reverting a breaking change to the DistanceMeasure enum by converting values to lowercase locally within the pipeline execution, optimizing performance by extracting a frequently compiled regular expression into a file-level constant, and adding as well as exporting missing top-level comparison helpers (lessThan and greaterThan) to ensure API completeness.

Comment thread packages/google_cloud_firestore/lib/src/pipeline.dart
Comment thread packages/google_cloud_firestore/lib/src/pipeline.dart Outdated
Comment thread packages/google_cloud_firestore/lib/src/pipeline.dart
Comment thread packages/google_cloud_firestore/lib/google_cloud_firestore.dart
@github-actions

github-actions Bot commented Jun 29, 2026

Copy link
Copy Markdown

Coverage Report

✅ Coverage 73.15% meets 40% threshold

Total Coverage: 73.15%
Lines Covered: 5859/8010

Package Breakdown

Package Coverage
google_cloud_firestore 73.21%
firebase_admin_sdk 73.07%

Minimum threshold: 40%

@demolaf
demolaf self-requested a review June 30, 2026 15:03
Comment thread packages/google_cloud_firestore/lib/src/pipeline.dart

@demolaf demolaf left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PipelineFunctions accepts Object? but only Expression.field() actually works without a fieldOrExpression-style check — untested since every call site already used field(). We should either commit to requiring Expression.field() explicitly or handle the coercion like Node's fieldOrExpression.

Comment thread packages/google_cloud_firestore/lib/src/pipeline.dart Outdated
Comment thread packages/google_cloud_firestore/lib/src/pipeline.dart Outdated
Comment thread packages/google_cloud_firestore/lib/src/pipeline.dart
Lyokone and others added 9 commits August 4, 2026 11:01
Arguments in a "field or expression" position kept String values as string
literals, so `PipelineFunctions.startsWith('title', 'Harry')` compared the
literal text "title" against "Harry" instead of reading the `title` field.
Add `_fieldOrExpression`, mirroring the Node SDK's `fieldOrExpression`, and
apply it across the function catalog. Value positions still keep Strings as
literals, and document paths stay values as they do in Node.

Also:

- Add `PipelineSource.createFrom()` to convert a `Query` or `VectorQuery`
  into an equivalent Pipeline, translating filters, projections, implicit
  orderings, cursors, limit/limitToLast and offset.
- Add expressions released since the initial port: `coalesce`, `length`,
  `reverse`, `concat`, `getField`, `geoDistance`, `documentMatches` and
  `score`, plus `logicalMinimum`/`logicalMaximum`.
- Align `PipelineExpression.length()` and `.concat()` with the Node SDK's
  generic `length`/`concat` backend functions, and expose the
  string-specific `charLength()`/`stringConcat()` alongside them.
- Rename field-position parameters to `fieldName` to match Node.
- Hide `greaterThan`/`lessThan` from the Firestore import in two suites that
  want the `matcher` versions.
…dings

Four Pipeline stages diverged from the backend contract, verified against
the canonical Node SDK stage definitions in dev/src/pipelines/stage.ts:

- `unnest` sent only the array expression, so there was no way to name the
  emitted element. It now sends `[expr, field(alias)]`, taking the alias
  from the selectable, and encodes `index_field` as a field reference
  rather than a string.
- `replace_with` omitted the required mode argument; it now sends
  `[map, 'full_replace']`.
- `sample` passed the rate as a `documents`/`percentage` option; it now
  sends `[rate, mode]` with mode `documents` or `percent`.
- `distinct` sent a positional list of expressions; it now sends a single
  map keyed by alias, reusing the same projection map as `select` and
  `aggregate`.

`unnest` and `sample` are breaking signature changes.

Also make `PipelineFunctions.minimum`/`maximum` aggregate-only, matching
Node, now that `logicalMinimum`/`logicalMaximum` cover the element-wise
form.

Adds golden proto tests asserting each stage's arguments and options, so
this class of wire-format drift is caught without an Enterprise database.
The README covered 4 of the ~16 Pipeline stages and the example directory
had no pipeline content at all.

- Add `example/pipeline_example.dart` covering every stage: source/filter/
  sort/project/limit, aggregates with and without grouping, `unnest`,
  `replaceWith` with `addFields`/`removeFields`, `distinct`, `sample`,
  `union`, `findNearest`, and `createFrom`. Each example seeds and cleans up
  its own documents, and carries `[START]`/`[END]` region markers for docs
  ingestion.
- Expand the README: every source and stage with a runnable snippet, a
  function reference table mapping Dart helpers to backend function names,
  the field-argument vs value-argument rule, execution options, the
  `PipelineSnapshot`/`PipelineResult` surface, and a Query migration guide.
- Note the Enterprise-edition requirement and the failure behavior on
  `Pipeline.execute()` and `Firestore.pipeline()`.
- Add `example/README.md` so both examples are discoverable, and so pub.dev's
  Example tab shows representative code.
`equalAny('dart', Expression.field('tags'))` passed a bare String in a
field position, where a String means a field reference rather than a
string literal (mirroring the Node SDK's fieldOrExpression). It therefore
asked about a non-existent `dart` field, so equalAny was always false --
and the adjacent notEqualAny passed for exactly the same wrong reason, a
missing field trivially not equalling anything. Both now target `title`
with array literals so they exercise the real semantics.

Also require FIRESTORE_PIPELINE_E2E_DATABASE_ID explicitly rather than
falling back to '(default)'. CI credential helpers such as
google-github-actions/auth export GOOGLE_CLOUD_PROJECT, so the previous
project-only guard could silently arm the suite against a default
database that lacks Pipelines support or this suite's vector index.
Adds e2e_pipeline.yml, which runs test/e2e/pipeline_e2e_test.dart against
the Enterprise-edition firestore-pipeline-test database that FlutterFire's
e2e_tests_pipeline.yaml also targets, so both SDKs are validated against
one shared Pipelines database. The suite existed but nothing ran it.

Triggers on pull requests touching packages/google_cloud_firestore, on
pushes to main, nightly, and on demand. Fork and dependabot pull requests
are skipped because they receive no secrets. It is a separate workflow
rather than a build.yml job so the live-quota cost is only paid for
changes that can affect it.

`dart test` exits 0 when every test is skipped, which is exactly what a
missing project or database ID produces, so the job fails if the secret
is empty and again if the run reports "All tests skipped" -- otherwise a
misconfigured job would report green for tests that never executed.

Authenticates as a dedicated service account holding roles/datastore.user
in the Pipelines project. The key is written to $RUNNER_TEMP rather than
the checkout and removed in an always() step.
The `documents` source stage routed each DocumentReference through
_encodePipelineValue, which emits the full resource name
(projects/{p}/databases/{d}/documents/books/book-1). Source stages name
resources by their path relative to the database instead: the `collection`
stage one branch above already emitted `/${path}`, and Node's
DocumentsSource encodes `'/' + ref.path`. The full name is only correct
for references in a value position, so the path is now built at the stage
rather than in the shared value encoder.

Extracts the shared `_relativeReference` helper, updates the unit test,
and adds E2E coverage for the stage, which was previously untested against
a live database.
@demolaf
demolaf merged commit 9652fc9 into main Aug 31, 2026
22 checks passed
@demolaf
demolaf deleted the feat/pipeline branch August 31, 2026 13:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants