Fix print_steps() and add optional progress/ETA logging to Pipeline - #417
Open
animmosmith wants to merge 3 commits into
Conversation
Fixes #415: print_steps() called steps_to_string(self.steps), but self.steps was never set anywhere - Pipeline only ever sets self.settings/self.stepnames in __init__. steps_to_string() also turned out to expect instantiated step objects (with a __dict__), not the raw settings dict, and is itself already marked deprecated - rather than resurrecting that, print_steps() now just pretty-prints the settings dict directly, which is simpler and shows the actually-configured arguments. Closes #416: adds an opt-in Pipeline.enable_progress_tracking(total_files, log_interval) that logs percent complete/elapsed/ETA every log_interval calls to run(). No effect unless called. With multiple chunks (pyopia process --num-chunks), each chunk's Pipeline instance tracks its own progress independently - this rides on the existing per-process logger, so it's automatically safe under the queue-based multiprocess logging already in place.
…ng-and-auxdata-errors' into 415-416-pipeline-progress-logging-and-print-steps-fix
This was
linked to
issues
Aug 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
print_steps()calledsteps_to_string(self.steps), butself.stepsis never set anywhere -Pipeline.__init__only ever setsself.settings/self.stepnames.steps_to_string()also turned out to expect instantiated step objects (with a__dict__), not the raw settings dict, and is itself already marked deprecated - rather than resurrecting that machinery,print_steps()now just pretty-prints the settings dict directly, which is simpler and shows the actually-configured arguments.Pipeline.enable_progress_tracking(total_files, log_interval)that logs percent complete/elapsed/ETA everylog_intervalcalls torun(). No effect unless called. With multiple chunks (pyopia process --num-chunks), each chunk'sPipelineinstance tracks its own progress independently (documented in the docstring) - this rides on the existing per-process logger, so it's automatically safe under the queue-based multiprocess logging already in place from Make multi-chunk logging queue-based and surface auxiliary data errors clearly #412.Version bumped to 2.16.22, ready to merge once the rest of the current chain is in.
Test plan
flake8 pyopiaclean.pytest -m "not slow and not training"): 36 passed.print_steps()now runs without error (previously crashed withAttributeError, then a follow-upTypeErroronce the obvious attribute-name fix was tried, confirmingsteps_to_string()needed to be bypassed rather than patched).🤖 Generated with Claude Code