Skip to content

feat(gemini): support combined json_schema + grounding via Interactions API for Gemini 3+ (Fixes #3426) - #3510

Open
hpiclaranet wants to merge 1 commit into
langgenius:mainfrom
hpiclaranet:feat/gemini-interactions-3426
Open

feat(gemini): support combined json_schema + grounding via Interactions API for Gemini 3+ (Fixes #3426)#3510
hpiclaranet wants to merge 1 commit into
langgenius:mainfrom
hpiclaranet:feat/gemini-interactions-3426

Conversation

@hpiclaranet

Copy link
Copy Markdown

Summary

Fixes #3426. Gemini 3+ models natively support combining Structured Output (json_schema) with built-in tools (grounding, url_context, code_execution), but only via the Interactions API (client.interactions.create()). The legacy generateContent endpoint silently drops grounding when both are active (HTTP 200, empty groundingMetadata) — see google-gemini/cookbook#1274.

The existing PR #3429 only relaxes the validator but does not implement the Interactions API routing — merging it as-is would replace the explicit InvokeError with a silent correctness bug. This PR provides the complete end-to-end implementation.

Changes

_is_gemini3_plus(model) — new static method. Robust to models/ path prefixes and other provider path segments, case-insensitive, rejects non-Gemini-3 names (e.g. gemini-30, gemini-3abc, claude-3-sonnet).

_validate_feature_compatibility — gains model: Optional[str] parameter. Rule 1 (json_schema exclusivity) is relaxed for Gemini 3+ with native tools and no custom tools. Other rules (Rule 3: url_context + code_execution, Rule 5: custom tools auto-disable) remain strict on all models.

_generate() routing — dispatched to _generate_via_interactions() when all conditions hold: gemini-3+ AND json_schema AND native tools AND no custom tools. All other paths use the existing generateContent endpoint unchanged.

_generate_via_interactions() — constructs and calls genai_client.interactions.create() with:

  • Input converted from types.Content to Interactions API Step format
  • system_instruction as a top-level string
  • Tools as Interaction dict format ([{"type": "google_search"}])
  • response_format as a single dict (not a list — prevents the "responseFormat must be set when responseMimeType is set" error)
  • generation_config using only the subset supported by this endpoint
  • store=False (Dify manages conversation history server-side)
  • Support for both inline_data (base64-embedded) and file_data (Files API URI) multimodal content

_handle_interactions_response() — extracts output text and token usage from the completed Interaction into LLMResult.

_handle_interactions_stream_response() — iterates Stream[InteractionSSEEvent], handles step.delta (text chunks), interaction.completed (final usage), and error events. Wrapped in a try/except to surface SDK exceptions as Dify InvokeError.

Design decisions

  • store=False: Dify is the conversation orchestrator and stores its own history. Passing store=True or using previous_interaction_id would duplicate state on Google servers, create desynchronization risks when Dify edits messages, and potentially violate data-sovereignty requirements. Stateless is the correct integration model for Dify.

  • Client caching: _generate_via_interactions currently constructs its own genai.Client(...). The separate PR [gemini] Performance: client cache, FileCache, token fallback, splitter local counting (Fixes #3333) #3509 adds a module-level cached client helper. The two PRs are independent and can merge in any order — once both land, _generate_via_interactions will be updated to use the cache in a follow-up.

  • Backward compatibility: Gemini 2.5, 2.0, and 1.5 models continue to use the strict validator + generateContent path unchanged.

Testing

  • 28 new unit tests covering _is_gemini3_plus (15 cases), validator relaxation (8 cases), and response/stream handlers (5 cases)
  • ruff check — All checks passed
  • ruff format — clean
  • python3 -m py_compile — pass
  • pytest models/tests/ (excluding live) — 207/207 passed

Version

manifest.yaml: 0.9.3 → 0.9.4

References

…ns API for Gemini 3+ (Fixes langgenius#3426)

Gemini 3+ models natively support combining Structured Output (json_schema)
with built-in tools (grounding, url_context, code_execution), but only via
the Interactions API (client.interactions.create()). The legacy generateContent
endpoint silently drops grounding when both are active (google-gemini/cookbook
langgenius#1274).

Changes:
- New _is_gemini3_plus(model) static method — robust to path prefixes,
  case-insensitive, rejects non-Gemini-3 models like gemini-30.
- _validate_feature_compatibility gains model param; Rule 1 (json_schema
  exclusivity) is relaxed for Gemini 3+ with native tools and no custom
  tools. Other rules (3, 5) remain strict on all models.
- _generate() routing: when gemini-3+ + json_schema + native tools + no
  custom tools → _generate_via_interactions() instead of generateContent.
- _generate_via_interactions() — calls client.interactions.create() with:
  - Input in Interactions API Step format (not types.Content)
  - system_instruction as top-level str
  - tools as dict format [{"type": "google_search"}]
  - response_format as single dict with mime_type inside
  - generation_config subset (no top_k/thinking_budget)
  - store=False (Dify manages history)
  - Supports inline_data and file_data (Files API URIs) for attachments
- _handle_interactions_response() — extracts output_text/usage → LLMResult
- _handle_interactions_stream_response() — iterates SSE events, handles
  step.delta, interaction.completed, error with proper exception wrapping
- New test_interactions.py — 28 unit tests (detection, validator, handlers)

Coexistence with PR langgenius#3509 (performance): this PR includes its own
genai.Client construction in _generate_via_interactions. If langgenius#3509 merges
first, the two are independent (no semantic conflict, only minor
inefficiency until performance landing is rebased).

Manifest: 0.9.3 → 0.9.4
Tests: 207/207 passed (ruff, py_compile clean)
@dosubot dosubot Bot added size:L This PR changes 100-499 lines, ignoring generated files. enhancement New feature or request labels Jul 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request size:L This PR changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[gemini] Structured output (json_schema) cannot be combined with grounding/url_context — needs Interactions API migration for Gemini 3+

1 participant