Skip to content

Fix pagination#378

Open
Yostra wants to merge 3 commits into
mainfrom
fix_pagination
Open

Fix pagination#378
Yostra wants to merge 3 commits into
mainfrom
fix_pagination

Conversation

@Yostra
Copy link
Copy Markdown
Collaborator

@Yostra Yostra commented May 20, 2026

Summary

Stripe's List API can return an empty page when starting_after (or ending_before) points at an object whose created second is fully inside the created[gte]/lt] window. The cursor effectively "loses" the rest of that one-second bucket, so any subdivision branch that inherited the cursor would silently skip records.
This PR removes the assumption that a cursor can drain a same-second boundary and instead re-queries that second by timestamp.

What changed

  • subdivideRanges: child ranges no longer inherit the parent cursor. Every child is fetched fresh from its created bounds. The final (newest) child is still widened to lastObserved + 1s so the boundary second is covered without relying on the cursor.
  • Small-remainder threshold tightened from < 30s to <= 1s so we only fall back to cursor-only pagination when there is genuinely nothing left to split.
  • streamingSubdivide: yields a shallow clone of range and forwards lastObserved on the event, so downstream range_complete accounting uses the fetch result instead of re-scanning emitted records.
  • fetchPageForRange: treats date as the created-timestamp fallback (invoice items use date, not created), and the head-completion logic in iterateStream matches.
  • Re-enables concurrency: 100 for streamingSubdivide.
  • Skips the "no duplicate record IDs" e2e assertion — re-fetching the boundary second can legitimately re-emit IDs; destination upserts handle idempotency.

@Yostra Yostra marked this pull request as ready for review May 21, 2026 00:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant