Skip to content

feat: add multimodal lego module - #72

Open
Ayakaee wants to merge 11 commits into
OpenDCAI:mainfrom
Ayakaee:feat/multimodal-lego
Open

Ayakaee wants to merge 11 commits into
OpenDCAI:mainfrom
Ayakaee:feat/multimodal-lego

Conversation

@Ayakaee

@Ayakaee Ayakaee commented Oct 26, 2026

Copy link
Copy Markdown

Summary

Enable multimodal selectors and weighters to work together through the LEGO
training pipeline, preserving sample identity, component refresh behavior,
and checkpoint state.

The pipeline composes:
Select → Mix → Reorder → Reweight.

Changes

  • Preserve processed sample/domain IDs through multimodal collation and training.
  • Integrate Loss, temporal DeltaLoss, and ADAPT-INF selectors;
    Static/Random/ODM mixers; existing reorder stages; and multimodal
    Loss, visual DeltaLoss, and ADAPT weighters.
  • Add compatible component contexts and weight-refresh lifecycle handling.
  • Support checkpoint/resume for scheduling state, including ODM.
  • Share compatible raw loss scores through ScoreBoard, with a
    shared / independent switch.
  • Preserve ADAPT-INF's actual CCS selection count in filter mode without
    filling or truncating its results.
  • Add a LLaVA training configuration and component configuration.

Validation

  • 79 local CPU regression tests passed.
  • Real LLaVA-1.5-7B validation on one A100 80GB, using 128 training
    examples and 16 separate anchors.
  • Four shared/independent comparisons passed.
  • Four continuous/resumed training comparisons passed, interrupting
    at step 3 and continuing to step 6.
  • Comparisons cover sample order, refresh steps, loss, recorded
    Loss/DeltaLoss weights, and final adapter parameters.
  • Five example tests passed after reorganizing the validation scripts.

Validation scripts and test files are retained locally and excluded
from this PR.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant