Skip to content

Fix print_steps() and add optional progress/ETA logging to Pipeline - #417

Open
animmosmith wants to merge 3 commits into
331-350-safer-multiprocess-logging-and-auxdata-errorsfrom
415-416-pipeline-progress-logging-and-print-steps-fix
Open

Fix print_steps() and add optional progress/ETA logging to Pipeline#417
animmosmith wants to merge 3 commits into
331-350-safer-multiprocess-logging-and-auxdata-errorsfrom
415-416-pipeline-progress-logging-and-print-steps-fix

Conversation

@animmosmith

Copy link
Copy Markdown
Collaborator

Summary

  • Fixes Pipeline.print_steps() references a non-existent attribute #415: print_steps() called steps_to_string(self.steps), but self.steps is never set anywhere - Pipeline.__init__ only ever sets self.settings/self.stepnames. steps_to_string() also turned out to expect instantiated step objects (with a __dict__), not the raw settings dict, and is itself already marked deprecated - rather than resurrecting that machinery, print_steps() now just pretty-prints the settings dict directly, which is simpler and shows the actually-configured arguments.
  • Closes Add optional progress/ETA logging to Pipeline #416: adds an opt-in Pipeline.enable_progress_tracking(total_files, log_interval) that logs percent complete/elapsed/ETA every log_interval calls to run(). No effect unless called. With multiple chunks (pyopia process --num-chunks), each chunk's Pipeline instance tracks its own progress independently (documented in the docstring) - this rides on the existing per-process logger, so it's automatically safe under the queue-based multiprocess logging already in place from Make multi-chunk logging queue-based and surface auxiliary data errors clearly #412.

Version bumped to 2.16.22, ready to merge once the rest of the current chain is in.

Test plan

  • flake8 pyopia clean.
  • Fast test subset (pytest -m "not slow and not training"): 36 passed.
  • Manually verified with a synthetic pipeline: progress logs fire at the configured interval with correct percentages, and print_steps() now runs without error (previously crashed with AttributeError, then a follow-up TypeError once the obvious attribute-name fix was tried, confirming steps_to_string() needed to be bypassed rather than patched).

🤖 Generated with Claude Code

Fixes #415: print_steps() called steps_to_string(self.steps), but self.steps
was never set anywhere - Pipeline only ever sets self.settings/self.stepnames
in __init__. steps_to_string() also turned out to expect instantiated step
objects (with a __dict__), not the raw settings dict, and is itself already
marked deprecated - rather than resurrecting that, print_steps() now just
pretty-prints the settings dict directly, which is simpler and shows the
actually-configured arguments.

Closes #416: adds an opt-in Pipeline.enable_progress_tracking(total_files,
log_interval) that logs percent complete/elapsed/ETA every log_interval
calls to run(). No effect unless called. With multiple chunks
(pyopia process --num-chunks), each chunk's Pipeline instance tracks its own
progress independently - this rides on the existing per-process logger, so
it's automatically safe under the queue-based multiprocess logging already in
place.
…ng-and-auxdata-errors' into 415-416-pipeline-progress-logging-and-print-steps-fix
@animmosmith animmosmith self-assigned this Aug 5, 2026
@animmosmith animmosmith added bug Something isn't working patch / enhancement improved functionality or patch indented for changes that require bumping only the PATCH number priority-low labels Aug 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working patch / enhancement improved functionality or patch indented for changes that require bumping only the PATCH number priority-low

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add optional progress/ETA logging to Pipeline Pipeline.print_steps() references a non-existent attribute

1 participant