Add source-agnostic tokenized dataset with bedbase source - #20
Open
nsheff wants to merge 2 commits into
Open
Conversation
Add geniml/dataset/: a source-agnostic tokenized training dataset built on the RegionSet/BedSet abstraction, connecting bedbase (the read half) to HF-Trainer training (the train half) — the previously-missing adapter. - RegionSetSource Protocol + concrete sources: BedbaseSource, FileListSource, BedSetSource - Pure tokenize_regionset / sample_and_remove windowing core (unk filtering, context-size chunking, huge-file skip) - TokenizedRegionDataset: materialize mode (on-disk parquet cached on selection + universe + context_size) and stream mode (torch IterableDataset with .gtok cache) - select_bedbase_samples queries the live bedbase /v1/bed/list API into a dated, serializable SampleSelection manifest (replaces the lost all_hg38_good_meta.csv); BedbaseSource reads only via BBClient.load_bed - Provenance: cache keying + write_provenance (dataset_provenance.json) - Refactor region2vec's file-list BEDDataset onto FileListSource - How-to doc + mkdocs nav entry heavy deps (datasets, torch) are imported lazily so the source and tokenize halves work on a base install. 13 tests pass.
- setup.py: add geniml.dataset to the packages list so it ships with the install (it was missing, so pip install would not include the new package). - Guard module-level optional-dependency imports in test_scembed (scanpy / sc extra) and test_search / test_text2bednn (qdrant_client / search extra) with pytest.importorskip, matching test_dataset's importorskip(datasets). These modules previously errored at collection under an [ml]-only install; execution of the network/service tests remains gated by --bedbase/--qdrant/--huggingface.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
geniml/dataset/: a source-agnostic tokenized training dataset built on theRegionSet/BedSetabstraction, connecting bedbase (the read half) to HF-Trainertraining (the train half) — the previously-missing adapter. It separates source (where region sets come from) from dataset (tokenize + window + batch), so a new data origin is a small source, not a new dataset. Bedbase is the first concrete source.What's here
Source interface (
source.py)RegionSetSource— minimalProtocolyielding(RegionSet, meta)BedSetSource(wrapsgeniml.io.BedSet/BBClient.load_bedsetoutput),FileListSource(local BED paths)Tokenization core (
tokenize.py) — pure, unit-testabletokenize_regionset(unk filtering, huge-file skip) +sample_and_removewindowingDataset (
dataset.py)TokenizedRegionDataset:materializemode (on-disk parquet, cached on selection + universe + context_size, optional multiprocess tokenization) andstreammode (torchIterableDatasetwith.gtokcache, DataLoader-sharded)Bedbase source (
bedbase.py)select_bedbase_samplesqueries the live/v1/bed/listAPI into a dated, serializableSampleSelectionmanifest (replaces the lostall_hg38_good_meta.csv);BedbaseSourcereads each bed only viaBBClient.load_bedProvenance (
manifest.py) — cache keying +write_provenance(dataset_provenance.json)Integrations & docs
BEDDatasetrefactored ontoFileListSourceHeavy deps (
datasets,torch) are imported lazily so the source and tokenize halves work on a base install; only building a dataset requiresgeniml[ml].Notes
gtars-bbcacheis absent against gtars 0.9.1, so this stays onBBClient; the swap point is localized toBedbaseSource.--data-source {parquet,bedbase}training-script change lives in the separateatacformer-bedbaseanalysis repo, not here.Testing
tests/test_dataset.py): materialize + cache, stream + DataLoader, parallel tokenization, windowing, bedbase selection/manifest, cache keying. Live bedbase selection verified with--bedbase.