feat(workflows): propagate step output model to context for evaluator-driven routing - #4393
feat(workflows): propagate step output model to context for evaluator-driven routing#4393tbitcs wants to merge 1 commit into
Conversation
When a step's output includes a 'model' field (e.g., from an evaluator recommending a tier), the engine now propagates it to context.default_model so subsequent steps automatically use the routed model. This enables evaluator-driven model routing: an evaluator can recommend 'premium' tier after finding critical issues, and the next phase's steps will use the premium model without manual configuration. Only string model values are propagated; non-string values (None, lists, dicts) are silently ignored. Explicit per-step model overrides still take priority over the routed context default. Refs: github#4290 (follow-up to evaluator contract extension) Assisted-by: GitHub Copilot (model: deepseek-v4-pro, autonomous)
|
Thanks for putting this together and for exploring evaluator-driven model selection. I agree that workflows should be able to use an evaluator's recommendation, but I don't think the engine should apply that recommendation automatically. Step outputs are already recorded in the workflow context and can be referenced by later steps. A workflow author can therefore make the routing decision explicit: - id: evaluate
type: command
command: speckit.evaluator.route
- id: implement
type: command
command: speckit.implement
model: "{{ steps.evaluate.output.model_routing.recommended_tier }}"Keeping this connection in the workflow definition has a few benefits:
In particular, For those reasons, I'd prefer that outputs remain data and that applying them to later inputs remain a conscious decision by the workflow author. The existing expression mechanism already supports that, so I don't think this change is needed in the core workflow engine. Posted on behalf of @mnriem by GitHub Copilot (model: GPT-5.6 Sol). |
Summary
Enables automatic model routing in the workflow engine. When a step's output includes a
modelfield, the engine propagates it tocontext.default_modelso subsequent steps automatically use the routed model.This is the follow-up to #4390 (evaluator contract extension). Together they enable the full portfolio approach: evaluators recommend a model tier, and the engine applies it automatically.
Changes
engine.py: 7 lines — after each step, checkresult.output.get("model")and propagate string values tocontext.default_modeltest_workflows.py: 5 new tests covering propagation, no-op when absent, non-string rejection, subsequent step inheritance, and explicit override priorityDesign
model: "sonnet-4"always uses that model, regardless of what was routed.context.default_modelstays at its initial value.Testing
uv run specify --helpuv sync && uv run pytestAI Disclosure
Generated by GitHub Copilot (model: deepseek-v4-pro, autonomous).
Refs: #4290
Depends on: #4390