Skip to content

local data backend: optimise for abspath lookups#2844

Merged
bghira merged 1 commit into
mainfrom
perf/os-walk-optimisation
Jul 17, 2026
Merged

local data backend: optimise for abspath lookups#2844
bghira merged 1 commit into
mainfrom
perf/os-walk-optimisation

Conversation

@bghira

@bghira bghira commented Jul 17, 2026

Copy link
Copy Markdown
Owner

This pull request refactors the list_files method in simpletuner/helpers/data_backend/local.py to improve file extension filtering and simplify file path handling. The main changes focus on how file extensions are matched and how file paths are constructed and returned.

File extension filtering and path handling improvements:

  • Changed the file extension filtering logic to use a tuple of dot-prefixed, lowercased extensions for more robust matching with str.endswith, instead of a set of extensions without the dot.
  • Simplified file path handling by removing the use of Path objects and instead using os.path.abspath and os.path.join to build absolute file paths.
  • Removed the creation of Path objects for each file found, as noted in the updated docstring and code.

Code cleanup:

  • Removed the unused import of Path from the file.

@bghira
bghira merged commit 43ca935 into main Jul 17, 2026
2 checks passed
@bghira
bghira deleted the perf/os-walk-optimisation branch July 17, 2026 14:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant