feat(gemini): support combined json_schema + grounding via Interactions API for Gemini 3+ (Fixes #3426) - #3510
Open
hpiclaranet wants to merge 1 commit into
Open
Conversation
…ns API for Gemini 3+ (Fixes langgenius#3426) Gemini 3+ models natively support combining Structured Output (json_schema) with built-in tools (grounding, url_context, code_execution), but only via the Interactions API (client.interactions.create()). The legacy generateContent endpoint silently drops grounding when both are active (google-gemini/cookbook langgenius#1274). Changes: - New _is_gemini3_plus(model) static method — robust to path prefixes, case-insensitive, rejects non-Gemini-3 models like gemini-30. - _validate_feature_compatibility gains model param; Rule 1 (json_schema exclusivity) is relaxed for Gemini 3+ with native tools and no custom tools. Other rules (3, 5) remain strict on all models. - _generate() routing: when gemini-3+ + json_schema + native tools + no custom tools → _generate_via_interactions() instead of generateContent. - _generate_via_interactions() — calls client.interactions.create() with: - Input in Interactions API Step format (not types.Content) - system_instruction as top-level str - tools as dict format [{"type": "google_search"}] - response_format as single dict with mime_type inside - generation_config subset (no top_k/thinking_budget) - store=False (Dify manages history) - Supports inline_data and file_data (Files API URIs) for attachments - _handle_interactions_response() — extracts output_text/usage → LLMResult - _handle_interactions_stream_response() — iterates SSE events, handles step.delta, interaction.completed, error with proper exception wrapping - New test_interactions.py — 28 unit tests (detection, validator, handlers) Coexistence with PR langgenius#3509 (performance): this PR includes its own genai.Client construction in _generate_via_interactions. If langgenius#3509 merges first, the two are independent (no semantic conflict, only minor inefficiency until performance landing is rebased). Manifest: 0.9.3 → 0.9.4 Tests: 207/207 passed (ruff, py_compile clean)
16 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #3426. Gemini 3+ models natively support combining Structured Output (
json_schema) with built-in tools (grounding,url_context,code_execution), but only via the Interactions API (client.interactions.create()). The legacygenerateContentendpoint silently drops grounding when both are active (HTTP 200, emptygroundingMetadata) — see google-gemini/cookbook#1274.The existing PR #3429 only relaxes the validator but does not implement the Interactions API routing — merging it as-is would replace the explicit
InvokeErrorwith a silent correctness bug. This PR provides the complete end-to-end implementation.Changes
_is_gemini3_plus(model)— new static method. Robust tomodels/path prefixes and other provider path segments, case-insensitive, rejects non-Gemini-3 names (e.g.gemini-30,gemini-3abc,claude-3-sonnet)._validate_feature_compatibility— gainsmodel: Optional[str]parameter. Rule 1 (json_schema exclusivity) is relaxed for Gemini 3+ with native tools and no custom tools. Other rules (Rule 3: url_context + code_execution, Rule 5: custom tools auto-disable) remain strict on all models._generate()routing — dispatched to_generate_via_interactions()when all conditions hold: gemini-3+ AND json_schema AND native tools AND no custom tools. All other paths use the existinggenerateContentendpoint unchanged._generate_via_interactions()— constructs and callsgenai_client.interactions.create()with:types.Contentto Interactions API Step formatsystem_instructionas a top-level string[{"type": "google_search"}])response_formatas a single dict (not a list — prevents the"responseFormat must be set when responseMimeType is set"error)generation_configusing only the subset supported by this endpointstore=False(Dify manages conversation history server-side)inline_data(base64-embedded) andfile_data(Files API URI) multimodal content_handle_interactions_response()— extracts output text and token usage from the completed Interaction intoLLMResult._handle_interactions_stream_response()— iteratesStream[InteractionSSEEvent], handlesstep.delta(text chunks),interaction.completed(final usage), anderrorevents. Wrapped in a try/except to surface SDK exceptions as DifyInvokeError.Design decisions
store=False: Dify is the conversation orchestrator and stores its own history. Passingstore=Trueor usingprevious_interaction_idwould duplicate state on Google servers, create desynchronization risks when Dify edits messages, and potentially violate data-sovereignty requirements. Stateless is the correct integration model for Dify.Client caching:
_generate_via_interactionscurrently constructs its owngenai.Client(...). The separate PR [gemini] Performance: client cache, FileCache, token fallback, splitter local counting (Fixes #3333) #3509 adds a module-level cached client helper. The two PRs are independent and can merge in any order — once both land,_generate_via_interactionswill be updated to use the cache in a follow-up.Backward compatibility: Gemini 2.5, 2.0, and 1.5 models continue to use the strict validator +
generateContentpath unchanged.Testing
_is_gemini3_plus(15 cases), validator relaxation (8 cases), and response/stream handlers (5 cases)ruff check— All checks passedruff format— cleanpython3 -m py_compile— passpytest models/tests/(excluding live) — 207/207 passedVersion
manifest.yaml: 0.9.3 → 0.9.4
References
generateContentendpoint with json_schema + google_search