Skip to content

webshart caption lookup - #3064

Merged
bghira merged 2 commits into
mainfrom
bugfix/webshart-caption-lookup
Aug 13, 2026
Merged

webshart caption lookup#3064
bghira merged 2 commits into
mainfrom
bugfix/webshart-caption-lookup

Conversation

@bghira

@bghira bghira commented Aug 13, 2026

Copy link
Copy Markdown
Owner

This pull request introduces support for optimizing caption loading in Webshart-backed datasets, reducing costly range reads for datasets that store captions in sidecar files. It adds a new webshart_optimize_captions option, updates configuration handling, and ensures that sidecar captions are consolidated into the local metadata cache when enabled. The documentation is updated in multiple languages to explain the new feature and its CLI usage. Additional minor improvements include more robust tag frequency memoization and a small fix in aspect bucket computation.

Webshart caption optimization support:

  • Added the webshart_optimize_captions option to dataset configs, CLI, and backend initialization, enabling automatic consolidation of sidecar captions into the local metadata cache for faster access. [1] [2] [3] [4] [5] [6] [7]
  • Implemented _optimize_caption_metadata() in WebshartDataBackend to perform the coalescing step at startup, with appropriate checks for required Webshart build features.

Documentation updates:

  • Updated the dataloader documentation in English, Spanish, Hindi, Japanese, Portuguese, and Chinese to describe the new caption optimization option, its benefits, and CLI usage for pre-coalescing captions. [1] [2] [3] [4] [5] [6]

Performance and bug fixes:

  • Memoized the tag frequency computation in training save hooks to avoid expensive recomputation on remote datasets.
  • Fixed a bug in aspect ratio bucket computation to properly check for caption sidecar files in Webshart metadata.

bghira added 2 commits August 12, 2026 18:11
…on-lookup

# Conflicts:
#	simpletuner/helpers/metadata/backends/webshart.py
#	tests/test_webshart_backend.py
@bghira
bghira merged commit ec6eeb6 into main Aug 13, 2026
4 checks passed
@bghira
bghira deleted the bugfix/webshart-caption-lookup branch August 13, 2026 02:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant