Add performance-testing tooling for big synthetic wikis#1189
Draft
JeroenDeDauw wants to merge 3 commits into
Draft
Add performance-testing tooling for big synthetic wikis#1189JeroenDeDauw wants to merge 3 commits into
JeroenDeDauw wants to merge 3 commits into
Conversation
ADR 29 (docs/adr/029-scalability-targets.md) sets Subject-count targets, and the first measurements against them needed a throwaway harness to build big wikis. This makes that capability a committed tool: generate a deterministic N-Subject wiki, time the import, and snapshot the result so a large environment rehydrates into any worktree stack in minutes instead of being regenerated or kept running. ## What lands - `maintenance/GeneratePerformanceDump.php` writes a MediaWiki XML dump of synthetic Subject pages plus the Schema they use, so an import is self-contained. Options: `--pages` (required), `--subjects-per-page` (10), `--seed` (1), `--output` (stdout). The shape follows ADR 29's "typical" column: 10 Subjects per page (1 main + 9 children) and 12 Statements each, three of them relations to Subjects on other pages — two backward references and one forward reference, so both stub creation and in-place upgrade are exercised. Output is deterministic, and the id space is split into a seed part and an index part, so dumps from different seeds can share one wiki. - `make perf-generate`, `perf-import`, `perf-snapshot` and `perf-restore`, following the Makefile's existing dual-mode proxy conventions. `perf-import` runs `importDump.php --no-updates` and prints elapsed time, pages/sec and Subjects/sec; that line is the benchmark. The snapshot is a `mariadb-dump` plus a `neo4j-admin database dump`, and restore loads both back. - A README section under Development, and `perf/` gitignored. The generator is PHP rather than a script in another language so it runs in-container with no host dependencies. ## Verified end to end On a dev stack: generate 200 pages, import (34 s, 58 Subjects/sec), snapshot (3 s: 2.6 MB SQL plus 550 KB Neo4j), tear the stack down including volumes, reseed clean, restore. The restored wiki matches the pre-wipe state exactly — 2085 Subject nodes, 8170 relationships, 321 pages — and the Subject REST endpoint answers for a generated id. Two findings from that run, both handled: - `neo4j-admin` has to run as the `neo4j` user, not the root that `docker compose exec` gives you. Loading as root leaves a store the server cannot write, and the database then stays offline while `START DATABASE` still reports success. `perf-restore` therefore proves the database answers a query before reporting success. - Neo4j's index state travels inside the snapshot, so a stack that has never run `RebuildGraphDatabases.php` has no `Subject(id)` index to measure against. QLever is outside the snapshot entirely, since its index is filled at runtime over SPARQL. Both are documented next to the targets. ## Tests `GeneratePerformanceDumpTest` covers the dump structure, Schema validity through `SchemaContentValidator`, statement counts through NeoWiki's own slot deserialization, relation-target closure and off-page targeting, and determinism. Checked by mutation: breaking the relation offsets, the seed's contribution to ids, one statement, or the Schema page each fails the matching test and nothing else. ## Considered, omitted - A `--statements-per-subject` knob. ADR 29 treats Statements per Subject as a dimension, but varying it means generating variable Property Definitions, and no measurement needs it yet. - A round-trip import integration test. The import path is covered by the end-to-end run above; running MediaWiki's importer inside PHPUnit would be slow for what it adds. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Only the Subject and relation ids carried the seed, so every seed produced the same page titles: `Perf test 0000000`. Importing a second dump into the same wiki therefore collided on every title and added no pages, which is the opposite of what the seed is for. Titles are now `Perf test <seed>-<index>`. Verified by importing seed 1 and seed 2 into one wiki: 20 pages each, Subject count up by the full 400. Also tightens the README section and the `make help` strings, and drops two constants from the public surface: nothing outside the script reads them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`perf-snapshot` and `perf-restore` bracket `neo4j-admin` with STOP/START DATABASE and then probe the database with a query, because START reports success even when the store it mounted is unusable. The probe ran but nothing gated on it: `; exit $status` discarded the sub-make's status, so a restore that left the database offline still printed "Restored." and exited 0 — exactly the failure the probe was added for. Also from a review pass over the branch: - Page offsets are clamped to the run size. `--pages 13` sent the `-13` offset back onto its own page for every page in the run, so the off-page relation invariant that the code comment, the README and the test all assert did not hold. Covered now for run sizes 20, 13 and 2. - A failed `perf-snapshot` no longer leaves a half-updated snapshot that `perf-restore` would load as a mismatched pair: the old files go first, and the guard requires non-empty ones. - `perf-generate` works from inside the container, where the `mkdir -p perf` had not run. - The content model, slot role and format literals come from the constants the rest of the codebase uses, so a rename cannot leave this script emitting dumps that import into the wrong slot. - The manifest comment says `total-subjects`, not `subjects`, which the `subjects=` make variable already uses for the per-page count. - Short writes and a failing close are detected, so a dump cut off by a full disk is reported rather than left looking complete. - New tests: the Schema defines exactly the properties the Subjects use, every generated property type is registered, the highest accepted seed still mints ids of the accepted length, and both rejected-option paths. - README: the rebuild that creates the Neo4j constraints is now step one rather than an aside, and the section says what `--no-updates` leaves behind and that the two halves of a snapshot are not captured at the same instant. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ADR 29 sets Subject-count targets, and the first
measurements against them needed a throwaway harness to build big wikis. This makes that
capability a committed tool: generate a deterministic N-Subject wiki, time the import, and
snapshot the result so a large environment rehydrates into any worktree stack in minutes instead
of being regenerated or kept running.
What lands
maintenance/GeneratePerformanceDump.phpwrites a MediaWiki XML dump of synthetic Subject pagesplus the Schema they use, so an import is self-contained. Options:
--pages(required),--subjects-per-page(10),--seed(1),--output(stdout). The shape sits at the scale ADR 29calls typical: 10 Subjects per page (1 main + 9 children) and 12 Statements each, three of them
relations to Subjects on other pages. Mixing negative and positive page offsets means the import
meets both targets that already exist and targets it has to stub-create and later upgrade in
place. Output is deterministic, and page titles and ids both carry the seed, so dumps from
different seeds can be imported into one wiki.
make perf-generate,perf-import,perf-snapshotandperf-restore. The first two follow theMakefile's dual-mode proxy convention and work from the host or inside the container; the other
two drive the containers, so they are host-only.
perf-importrunsimportDump.php --no-updatesand prints elapsed time, pages/sec and Subjects/sec; that line isthe benchmark. The snapshot is a
mariadb-dumpplus aneo4j-admin database dump, and restoreloads both back.
perf/gitignored.The generator is PHP rather than a script in another language so it runs in-container with no host
dependencies.
Verified end to end
On a dev stack: generate, import, snapshot, tear the stack down including volumes, reseed clean,
restore. The restored wiki matches the pre-wipe state exactly — 2085 Subject nodes, 8170
relationships, 321 pages — and the Subject REST endpoint answers for a generated id. Deleting 100
Subject nodes and restoring brings the count back. Three seeds imported into one wiki keep their
own page sets (20 + 20 + 50 pages).
Three behaviours the targets have to handle, all covered in the code and documented:
neo4j-adminhas to run as theneo4juser, not the root thatdocker compose execgives you.Loading as root leaves a store the server cannot write, and the database then stays offline while
START DATABASEstill reports success — so both targets probe the database with a query and failif it does not answer.
RebuildGraphDatabases.php, so a stack thathas never run it measures an unindexed graph. The README makes that rebuild step one.
across a restore. A restored wiki therefore keeps whatever the SPARQL store already held.
Tests
GeneratePerformanceDumpTestcovers the dump structure, Schema validity throughSchemaContentValidator, that the Schema defines exactly the properties the Subjects use and thatevery property type is registered, statement counts through NeoWiki's own slot deserialization,
relation-target closure and off-page targeting at three run sizes, determinism, seed disjointness,
the highest accepted seed, and both rejected-option paths. Checked by mutation: breaking the
relation offsets or their clamp, the seed's contribution to ids or titles, one statement, or the
Schema page each fails the matching test and nothing else.
Considered, omitted
--statements-per-subjectknob. ADR 29 treats Statements per Subject as a dimension, butvarying it means generating variable Property Definitions, and no measurement needs it yet.
running MediaWiki's importer inside PHPUnit would be slow for what it adds.
maintenance/to the phpcs and phpstan targets. It is outside both today, so this scriptgets neither, but bringing the directory in scope would surface the two existing scripts as well.