feat(firestore): add BSON read deserialization support by ohmayr · Pull Request #18402 · googleapis/google-cloud-python

ohmayr · 2026-09-16T20:30:29Z

Adds opt-in BSON read deserialization support to the Google Cloud Firestore Python SDK.
When enabled via decode_bson=True, document fields containing BSON wire map structures returned by Firestore (such as {"__oid__": "507f191e810c19729de860ea"}) are automatically deserialized into their corresponding Python BSON container instances (BSONObjectId, BSONDecimal128, BSONTimestamp, BSONRegex, BSONBinary, BSONInt32, BSONMinKey, BSONMaxKey).

💻 Usage

Default Behavior (`decode_bson=False`)

Existing applications continue to receive raw map dictionaries by default to preserve 100% backward compatibility:

client = firestore.Client()
doc = client.collection("users").document("doc1").get()

Opt-in Behavior (`decode_bson=True`)

client = firestore.Client(decode_bson=True)
doc = client.collection("users").document("doc1").get()
# Returns deserialized BSON instance: {"user_id": BSONObjectId("507f191e810c19729de860ea")}
data = doc.to_dict()

🏛️ Design Decisions

Opt-In decode_bson=False Default (Enterprise Backward Safety): Defaulting to decode_bson=False ensures existing production code accessing raw dictionary keys (dict["user_id"]["oid"]) will not break upon upgrading the SDK.
Subtype 0 Binary Deserialization: Wire maps representing Subtype 0 BSON Binary (v[0] == 0) are deserialized into native Python bytes (b"..."), while non-zero subtypes ($1 \le v[0] \le 255$) deserialize into BSONBinary(data, subtype=v[0]) objects.
Explicit Non-None Fallback Control: Updated decode_dict() to explicitly check if decoded is not None: rather than relying on Python truthiness (or), preventing false fallback on empty byte payloads (b"") or falsy objects.
Recursive Nested Map & Array Support: Added _decode_bson_dict_recursive() to ensure BSON wire maps inside nested dictionaries and array elements are deserialized properly.

Fixes b/562164140 🦕

gemini-code-assist

Code Review

This pull request introduces several new BSON types (BSONInt32, BSONBinary, BSONTimestamp, BSONRegex, and BSONDecimal128) to the Firestore Python client, along with their respective decoders, integration tests, and unit tests. Feedback on these changes highlights a violation of Python's hash contract in BSONDecimal128 due to mixed-type equality with decimal.Decimal without matching hashes. Additionally, the reviewer recommended replacing the boolean or fallback logic in decode_dict with an explicit None check to prevent potential bugs with falsy decoded BSON objects.

…ecode_value - Restore full Union return type with _BSONType on decode_value. - Restore Returns and Raises docstring sections in decode_value matching base branch. - Remove unused _BSON_DECODERS import from _helpers.py. - Revert extraneous changes to pipeline_result.py. Towards #18402

… types with _BSONType - Annotate decode_dict with Union[dict, Vector, _BSONType]. - Update PipelineResult.data to return dict | Vector | _BSONType | None. - Import _BSONType under TYPE_CHECKING in pipeline_result.py. Towards #18402

…nsions for librarian - Make client a required positional parameter in decode_value and decode_dict. - Format comprehensions in _helpers.py as single lines to satisfy librarian generation check. Towards #18402

daniel-sanche

Looking better, but a few more comments

daniel-sanche · 2026-09-21T23:21:18Z

+        """Deserializes a BSON wire map dictionary into a BSON instance or bytes.
+
+        Args:
+            data (Any): Potential BSON wire map dictionary.


Can this really return Any type? I would assume BSONType | bytes | None

(Try to avoid using Any wherever possible)

The decoders in only return concrete subclasses (such as , , , etc.) or if the dict doesn't match a BSON wire format. Updated the return type annotation to .

Was this left unpushed?

Yes, apologies! The commit was rebased onto bson-pr1g-decimal128 and is now pushed. BSONType._from_dict now returns Optional[Union["BSONType", bytes]].

daniel-sanche · 2026-09-21T23:23:49Z


-def decode_dict(value_fields, client) -> Union[dict, Vector]:
+def _decode_bson_dict_recursive(data: Any) -> Any:
+    """Recursively decodes BSON wire map dictionaries."""


IIUC, This method shouldn't be necessary. decode_dict is already recursive, and should hanle BSON on its own. But let me know if I'm missing something

decodes from protobuf where each value is a protobuf message. In pipelines (like ), results can return already-converted python dictionaries where nested maps need BSON wire dicts converted. Having it centralized allows decoding both paths consistently.

daniel-sanche · 2026-09-21T23:26:26Z

            return None
-        return copy.deepcopy(self._data)
+        data = copy.deepcopy(self._data)
+        return _helpers._decode_bson_dict_recursive(data)


This shouldn't need to change, self._data should already be in a good format (i.e., it would have run through _decode_dict before being saved to _data)

Agreed! Kept untouched since it is already decoded at ingestion time.

daniel-sanche · 2026-09-21T23:31:09Z

+        decoder = _BSON_DECODERS.get(key)
+        if decoder is None:
+            return None
+        return decoder(val)


should we catch exceptions here, so we don't crash when reading data? Maybe fall back to None?

Added a block around in so corrupted or unexpected payload shapes gracefully return rather than raising uncaught exceptions during read.

Do you have an unpublished commit? I'm not seeing the change

Pushed now! Wrapped the decoder(val) call in try...except Exception: return None so malformed or unexpected wire dictionary shapes fall back gracefully to None instead of raising unhandled exceptions during read.

daniel-sanche · 2026-09-21T23:37:28Z

+def decode_dict(
+    value_fields,
+    client,
+) -> Union[dict, Vector, _BSONType]:


decodes a protobuf (), which is only ever converted into a , a , or a (e.g. ). Primitive are only decoded directly by when encountering a standalone protobuf bytes field, so is not a possible return type of .

It looks like some words are missing from your response, but the binary field in _BSON_DECODERS can retrun bytes

(This is why I'd really like to get rid of the Any annotations. It makes it very hard to trace types)

Good catch! You're completely right. __binary__ with subtype 0 returns native bytes. Updated decode_dict and PipelineResult.data return type annotations and docstrings to include bytes.

…nsions for librarian - Make client a required positional parameter in decode_value and decode_dict. - Format comprehensions in _helpers.py as single lines to satisfy librarian generation check. Towards #18402

…ation - Perform automatic BSON deserialization in decode_dict and DocumentSnapshot.to_dict using _BSONType._from_dict. - Remove decode_bson configuration parameter across Client, AsyncClient, BaseClient, and DocumentSnapshot. - Preserve precise return type annotations in decode_dict and restore docstring Raises section. Towards #18402

…ecode_value - Restore full Union return type with _BSONType on decode_value. - Restore Returns and Raises docstring sections in decode_value matching base branch. - Remove unused _BSON_DECODERS import from _helpers.py. - Revert extraneous changes to pipeline_result.py. Towards #18402

… types with _BSONType - Annotate decode_dict with Union[dict, Vector, _BSONType]. - Update PipelineResult.data to return dict | Vector | _BSONType | None. - Import _BSONType under TYPE_CHECKING in pipeline_result.py. Towards #18402

…nsions for librarian - Make client a required positional parameter in decode_value and decode_dict. - Format comprehensions in _helpers.py as single lines to satisfy librarian generation check. Towards #18402

daniel-sanche · 2026-09-23T23:02:18Z

            return hash((type(self), self._value))
+
+
+_BSON_DECODERS: Dict[str, Callable[[Any], Any]] = {


Can we get rid of the Anys here? The types should be well defined

Done. Updated _BSON_DECODERS annotation to Dict[str, Callable[..., Optional[Union[BSONType, bytes]]]], eliminating all Any annotations.