Skip to content

perf: speed up concurrent imports with large peer histories (#1222 + clippy fix) - #1247

Closed
zxch3n wants to merge 4 commits into
mainfrom
fix-1222-concurrent-import-perf
Closed

zxch3n wants to merge 4 commits into
mainfrom
fix-1222-concurrent-import-perf

Conversation

@zxch3n

@zxch3n zxch3n commented Nov 3, 2026

Copy link
Copy Markdown
Member

Stacked on #1222. The first three commits are #1222 by @serg-and, unchanged; only the last commit (fe33ee1) is new. #1222 does not allow maintainer edits, so the fix could not be pushed to its branch. Merging this supersedes #1222.

Summary

#1222 fixes two costs that grow quadratically with the number of peer ids a document has seen:

  • try_shrink_frontiers walked ancestors once per candidate. It now walks once per retained head, and try_vv_to_frontiers returns the maintained dag.frontiers when asked for the current version.
  • The movable-list/text/list tracker rebuild (replay_container_ops_from_empty) rebuilt a full version vector for every historical change. It now skips changes that have no op on the container before building one.

The new commit only puts iter_from_replay_base_causally behind #[cfg(test)]. After #1222 it is called only from tests, so cargo clippy -- -Dwarnings failed with dead_code. The doc references now point at try_iter_from_replay_base_causally, the production entry point.

Validation

  • Rust port of the perf: speed up concurrent imports with large peer histories #1222 reproduction (5,000 historical peers, 4 concurrent updates, release): the 2nd-4th imports went from ~760 ms to 4-11 ms for Text, Tree and List, and from 2.7-4.6 s to 17-25 ms for MovableList. Map is unchanged. With only the frontier changes, MovableList is still ~450-700 ms, so the container filter is needed too.
  • cargo test -p loro-internal -p loro --features loro-internal/test_utils,loro/jsonpath: 1647 passed, 0 failed.
  • cargo fmt --all -- --check: clean.
  • cargo clippy -p loro-internal --all-features -- -Dwarnings: the dead_code error is gone. The 3 errors left (loro.rs question-mark, query_by_len.rs doc indentation) are also on main with current stable clippy.
  • Not run: WASM tests, fuzz.

Notes

🤖 Generated with Claude Code

serg-and and others added 4 commits October 22, 2026 15:27
After #1222 routed production replays through the filtered entry point,
`iter_from_replay_base_causally` is only called from tests, which made
`cargo clippy -- -Dwarnings` fail with dead_code. Point the doc references
at `try_iter_from_replay_base_causally`, the production entry point.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

WASM Size Report

  • Original size: 3486.50 KB
  • Gzipped size: 1168.46 KB
  • Brotli size: 818.72 KB

@zxch3n

zxch3n commented Nov 3, 2026

Copy link
Copy Markdown
Member Author

Superseded: #1222 was merged directly, and the clippy fix from this PR landed in #1251.

@zxch3n zxch3n closed this Nov 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants