Skip to content

Bind loop variables in the properties of generated Python operators #8669

Description

@aglinxinyuan

Feature Summary

Inside a control block (Loop Start ... Loop End), any operator property can hold a loop-variable reference such as $i (#8635, #8636). Operators whose Scala descriptor generates Python code from their properties, such as the sklearn trainers, cannot use it yet. Their code is generated when the workflow is compiled, before the loop runs, so the compiler rejects a reference on them. The trainers' hyperparameter values are the common case: the property panel offers $i there, but the run fails at compile time.

Loop Start (i = 1) -> SVC trainer (C = "$i") -> Loop End
today:  compile error, "its code is generated before the loop runs"
after:  iteration i trains with C = i

Proposed Solution or Design

A text property that the generated code decodes at run time is already read inside the operator, after the iteration's state has arrived. Bind the reference there, using the state that each Python operator registers before process_state (state, #8641):

Step Where What
1 Code generation (Scala) Inside a block, the decode expression of a text reference, self.decode_python_template('<"$i" encoded>'), becomes a lookup, self.loop_variable_text('i').
2 Compiler A generated operator is accepted when every reference it holds was rewritten. A numeric or boolean reference, or a text reference that the code embeds as plain text, is still a compile error that names the property.
3 Python operator loop_variable_text(name) returns the current value of name in state as text, and fails with a clear message when no state message carried it.
4 Code generation (Scala) A numeric or boolean property holds a placeholder when the code is generated, and its value is written into the code. Generate the code twice, from two probe values: where the two codes differ by exactly the value, and nowhere else, read it from the state instead (self.loop_variable_value('n', 'integer')). Where they differ otherwise, or the descriptor checks or adjusts the value, the reference stays a compile error.

Outside every block nothing changes: a text such as $AAPL stays text.

Depends on #8639 and #8642.

Affected Area

Workflow Engine (Amber)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions