Optimize changelog diffing for large metadata sets#125
Merged
JakeShirley merged 7 commits intomainfrom Apr 17, 2026
Merged
Conversation
Replace hot-path algorithms in ChangelogGenerator with faster equivalents: - Value-case comparison: use JSON.stringify with sorted-key replacer instead of deepCopyJson + removePropertyRecursive + deepEqual. This eliminates millions of unnecessary object clones on large inputs. - compareArray nextSubobjects lookup: build a Map<key, index> once per call instead of .map().indexOf() per element (O(N*M) -> O(N+M)). - getChangelogForVersion: reverse linear scan instead of .map().indexOf(), avoiding an intermediate array allocation per call. - Per-module changelog copy: JSON.stringify once + JSON.parse N times instead of deepCopyJson per module. Benchmarked on 5 Minecraft releases (~278 MB of script module metadata): Before: 897s wall-clock, 6683 MB peak RSS After: 75s wall-clock, 6596 MB peak RSS (12x faster)
rlandav
approved these changes
Apr 17, 2026
JakeShirley
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The changelog diffing step in
@minecraft/api-docs-generatorwas extremely slow and memory-hungry when processing many Minecraft metadata versions. This PR replaces four hot-path algorithms with faster equivalents.Changes
1. Value-case comparison —
JSON.stringifywith sorted-key replacerReplaces
deepCopyJson+removePropertyRecursive+deepEqualwith a singleJSON.stringifypass per value using a sorted-key replacer. This is the main win, as it eliminates millions of unnecessary object clones on production-scale inputs.2.
compareArraynextSubobjects lookup —Mapinstead of.map().indexOf()Builds a
Map<key, index>overnextSubobjectsonce per call instead of allocating an intermediate array and scanning it per element. Reduces complexity from O(N·M) to O(N+M).3.
getChangelogForVersion— reverse scan instead of.map().indexOf()Scans from the end of the changelog array (where recently-added versions are) instead of mapping the entire array into a new one per call.
4. Per-module changelog copy — serialize once, parse N times
Replaces
deepCopyJson(sortedChangelogs)called once per module with a singleJSON.stringifyfollowed by NJSON.parsecalls.Benchmark Results
Tested on 5 Minecraft releases (~278 MB of real script module metadata including server-bindings up to 2.5 MB each):
Output is byte-identical to baseline across all 12 generated changelog JSON files.
Testing
tools/api-docs-generator)tools/api-docs-generator-test-snapshots)