Skip to content

fix(storage): preserve streaming records and metadata - #586

Merged
SunnyHaze merged 2 commits into
OpenDCAI:mainfrom
JustinBinber:codex/fix-stream-record-contract
Nov 13, 2026
Merged

SunnyHaze merged 2 commits into
OpenDCAI:mainfrom
JustinBinber:codex/fix-stream-record-contract

Conversation

@JustinBinber

Copy link
Copy Markdown
Contributor

Fixes #585.

Make StreamBatchedFileStorage yield bounded DataFrame batches for JSON, Parquet and Pickle inputs instead of column-name strings, and expose matching top-level column metadata. Count logical CSV/JSONL records and reuse counts only while the current step and input-file identity are unchanged.

The existing output write/flush paths are unchanged. Fallback formats still materialize the source before slicing; this does not introduce native streaming readers for those formats. Chunk readers close when iteration ends early, and symlink format detection follows the configured source filename.

Validation on Windows/Python 3.11.7, pandas 3.1.1 and PyArrow 26.0.1:

  • 22 new CPU regressions pass, including three real compiled pipeline runs, bounded batches, nested Parquet fields, multiline/blank/empty records, count invalidation and symlink inputs. Against unmodified base b9836cb, these tests produce 18 failures and 4 passing controls.
  • Seven related storage, pipeline, cache, JSONL-policy and batch suites: 90 passed, 1 skipped (optional XLSX dependency).
  • Independent review reproduced and resolved a symlink regression; no open findings remain. git diff --check passes. No model-provider calls were used.

@SunnyHaze

Copy link
Copy Markdown
Collaborator

LGTM, thanks!

@SunnyHaze
SunnyHaze merged commit 84863d8 into OpenDCAI:main Nov 13, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Streaming storage yields column names and stale record metadata

2 participants