Skip to content

Add performance-testing tooling for big synthetic wikis#1189

Draft
JeroenDeDauw wants to merge 3 commits into
masterfrom
feature/perf-testing-tooling
Draft

Add performance-testing tooling for big synthetic wikis#1189
JeroenDeDauw wants to merge 3 commits into
masterfrom
feature/perf-testing-tooling

Conversation

@JeroenDeDauw

@JeroenDeDauw JeroenDeDauw commented Jul 26, 2026

Copy link
Copy Markdown
Member

ADR 29 sets Subject-count targets, and the first
measurements against them needed a throwaway harness to build big wikis. This makes that
capability a committed tool: generate a deterministic N-Subject wiki, time the import, and
snapshot the result so a large environment rehydrates into any worktree stack in minutes instead
of being regenerated or kept running.

What lands

  • maintenance/GeneratePerformanceDump.php writes a MediaWiki XML dump of synthetic Subject pages
    plus the Schema they use, so an import is self-contained. Options: --pages (required),
    --subjects-per-page (10), --seed (1), --output (stdout). The shape sits at the scale ADR 29
    calls typical: 10 Subjects per page (1 main + 9 children) and 12 Statements each, three of them
    relations to Subjects on other pages. Mixing negative and positive page offsets means the import
    meets both targets that already exist and targets it has to stub-create and later upgrade in
    place. Output is deterministic, and page titles and ids both carry the seed, so dumps from
    different seeds can be imported into one wiki.
  • make perf-generate, perf-import, perf-snapshot and perf-restore. The first two follow the
    Makefile's dual-mode proxy convention and work from the host or inside the container; the other
    two drive the containers, so they are host-only. perf-import runs
    importDump.php --no-updates and prints elapsed time, pages/sec and Subjects/sec; that line is
    the benchmark. The snapshot is a mariadb-dump plus a neo4j-admin database dump, and restore
    loads both back.
  • A README section under Development, and perf/ gitignored.

The generator is PHP rather than a script in another language so it runs in-container with no host
dependencies.

Verified end to end

On a dev stack: generate, import, snapshot, tear the stack down including volumes, reseed clean,
restore. The restored wiki matches the pre-wipe state exactly — 2085 Subject nodes, 8170
relationships, 321 pages — and the Subject REST endpoint answers for a generated id. Deleting 100
Subject nodes and restoring brings the count back. Three seeds imported into one wiki keep their
own page sets (20 + 20 + 50 pages).

Three behaviours the targets have to handle, all covered in the code and documented:

  • neo4j-admin has to run as the neo4j user, not the root that docker compose exec gives you.
    Loading as root leaves a store the server cannot write, and the database then stays offline while
    START DATABASE still reports success — so both targets probe the database with a query and fail
    if it does not answer.
  • Neo4j's uniqueness constraints are created only by RebuildGraphDatabases.php, so a stack that
    has never run it measures an unindexed graph. The README makes that rebuild step one.
  • QLever is outside the snapshot, since its index is filled at runtime over SPARQL and persists
    across a restore. A restored wiki therefore keeps whatever the SPARQL store already held.

Tests

GeneratePerformanceDumpTest covers the dump structure, Schema validity through
SchemaContentValidator, that the Schema defines exactly the properties the Subjects use and that
every property type is registered, statement counts through NeoWiki's own slot deserialization,
relation-target closure and off-page targeting at three run sizes, determinism, seed disjointness,
the highest accepted seed, and both rejected-option paths. Checked by mutation: breaking the
relation offsets or their clamp, the seed's contribution to ids or titles, one statement, or the
Schema page each fails the matching test and nothing else.

Considered, omitted

  • A --statements-per-subject knob. ADR 29 treats Statements per Subject as a dimension, but
    varying it means generating variable Property Definitions, and no measurement needs it yet.
  • A round-trip import integration test. The import path is covered by the end-to-end run above;
    running MediaWiki's importer inside PHPUnit would be slow for what it adds.
  • Adding maintenance/ to the phpcs and phpstan targets. It is outside both today, so this script
    gets neither, but bringing the directory in scope would surface the two existing scripts as well.

AI-authored — Claude Code, Opus 5 (max); detailed spec from @JeroenDeDauw, no redirects; diff not yet human-reviewed; full PHPUnit suite and make cs green locally, CI green, plus generate/import/snapshot/wipe/restore runs against a live dev stack, a five-mutation check of the new tests, and an adversarial code and prose review whose findings are in the second and third commits.

ADR 29 (docs/adr/029-scalability-targets.md) sets Subject-count targets, and the first
measurements against them needed a throwaway harness to build big wikis. This makes that
capability a committed tool: generate a deterministic N-Subject wiki, time the import, and
snapshot the result so a large environment rehydrates into any worktree stack in minutes
instead of being regenerated or kept running.

## What lands

- `maintenance/GeneratePerformanceDump.php` writes a MediaWiki XML dump of synthetic Subject
  pages plus the Schema they use, so an import is self-contained. Options: `--pages`
  (required), `--subjects-per-page` (10), `--seed` (1), `--output` (stdout). The shape follows
  ADR 29's "typical" column: 10 Subjects per page (1 main + 9 children) and 12 Statements each,
  three of them relations to Subjects on other pages — two backward references and one forward
  reference, so both stub creation and in-place upgrade are exercised. Output is deterministic,
  and the id space is split into a seed part and an index part, so dumps from different seeds
  can share one wiki.
- `make perf-generate`, `perf-import`, `perf-snapshot` and `perf-restore`, following the
  Makefile's existing dual-mode proxy conventions. `perf-import` runs
  `importDump.php --no-updates` and prints elapsed time, pages/sec and Subjects/sec; that line
  is the benchmark. The snapshot is a `mariadb-dump` plus a `neo4j-admin database dump`, and
  restore loads both back.
- A README section under Development, and `perf/` gitignored.

The generator is PHP rather than a script in another language so it runs in-container with no
host dependencies.

## Verified end to end

On a dev stack: generate 200 pages, import (34 s, 58 Subjects/sec), snapshot (3 s: 2.6 MB SQL
plus 550 KB Neo4j), tear the stack down including volumes, reseed clean, restore. The restored
wiki matches the pre-wipe state exactly — 2085 Subject nodes, 8170 relationships, 321 pages —
and the Subject REST endpoint answers for a generated id.

Two findings from that run, both handled:

- `neo4j-admin` has to run as the `neo4j` user, not the root that `docker compose exec` gives
  you. Loading as root leaves a store the server cannot write, and the database then stays
  offline while `START DATABASE` still reports success. `perf-restore` therefore proves the
  database answers a query before reporting success.
- Neo4j's index state travels inside the snapshot, so a stack that has never run
  `RebuildGraphDatabases.php` has no `Subject(id)` index to measure against. QLever is outside
  the snapshot entirely, since its index is filled at runtime over SPARQL. Both are documented
  next to the targets.

## Tests

`GeneratePerformanceDumpTest` covers the dump structure, Schema validity through
`SchemaContentValidator`, statement counts through NeoWiki's own slot deserialization,
relation-target closure and off-page targeting, and determinism. Checked by mutation: breaking
the relation offsets, the seed's contribution to ids, one statement, or the Schema page each
fails the matching test and nothing else.

## Considered, omitted

- A `--statements-per-subject` knob. ADR 29 treats Statements per Subject as a dimension, but
  varying it means generating variable Property Definitions, and no measurement needs it yet.
- A round-trip import integration test. The import path is covered by the end-to-end run above;
  running MediaWiki's importer inside PHPUnit would be slow for what it adds.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@JeroenDeDauw JeroenDeDauw added the performance Performance and scalability work (see ADR 29) label Jul 26, 2026
JeroenDeDauw and others added 2 commits July 27, 2026 01:10
Only the Subject and relation ids carried the seed, so every seed produced the same page
titles: `Perf test 0000000`. Importing a second dump into the same wiki therefore collided on
every title and added no pages, which is the opposite of what the seed is for. Titles are now
`Perf test <seed>-<index>`.

Verified by importing seed 1 and seed 2 into one wiki: 20 pages each, Subject count up by the
full 400.

Also tightens the README section and the `make help` strings, and drops two constants from the
public surface: nothing outside the script reads them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`perf-snapshot` and `perf-restore` bracket `neo4j-admin` with STOP/START DATABASE and then
probe the database with a query, because START reports success even when the store it mounted
is unusable. The probe ran but nothing gated on it: `; exit $status` discarded the sub-make's
status, so a restore that left the database offline still printed "Restored." and exited 0 —
exactly the failure the probe was added for.

Also from a review pass over the branch:

- Page offsets are clamped to the run size. `--pages 13` sent the `-13` offset back onto its
  own page for every page in the run, so the off-page relation invariant that the code comment,
  the README and the test all assert did not hold. Covered now for run sizes 20, 13 and 2.
- A failed `perf-snapshot` no longer leaves a half-updated snapshot that `perf-restore` would
  load as a mismatched pair: the old files go first, and the guard requires non-empty ones.
- `perf-generate` works from inside the container, where the `mkdir -p perf` had not run.
- The content model, slot role and format literals come from the constants the rest of the
  codebase uses, so a rename cannot leave this script emitting dumps that import into the
  wrong slot.
- The manifest comment says `total-subjects`, not `subjects`, which the `subjects=` make
  variable already uses for the per-page count.
- Short writes and a failing close are detected, so a dump cut off by a full disk is reported
  rather than left looking complete.
- New tests: the Schema defines exactly the properties the Subjects use, every generated
  property type is registered, the highest accepted seed still mints ids of the accepted
  length, and both rejected-option paths.
- README: the rebuild that creates the Neo4j constraints is now step one rather than an
  aside, and the section says what `--no-updates` leaves behind and that the two halves of a
  snapshot are not captured at the same instant.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

performance Performance and scalability work (see ADR 29)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant