Add multi-Ractor memory benchmarks

README.md

@@ -138,7 +138,9 @@ There are two Ractor-related categories:
 
 * **`--category ractor`** - Runs both regular benchmarks marked with `ractor:
   true` in `benchmarks.yml` AND all benchmarks from the `benchmarks-ractor`
-  directory. The `harness-ractor` harness is used for both types of benchmark.
+  directory. The `harness-ractor` harness is used unless a benchmark sets
+  `default_harness` in `benchmarks.yml` (the Ractor memory benchmarks set
+  `harness-ractor-mem`).
 
 * **`--category ractor-only`** - Runs ONLY benchmarks from the
   `benchmarks-ractor` directory, ignoring regular benchmarks even if they are
@@ -165,6 +167,50 @@ intended to be used with any harness except `harness-ractor`.
 Note: The `harness-ractor` harness is automatically selected when using these
 categories, so there's no need to specify `--harness` manually.
 
+### Ractor Memory Benchmarks
+
+Three benchmarks measure pathological memory behaviour with multiple ractors,
+for GC work that reclaims ractor-local memory. They use the
+`harness-ractor-mem` harness, which measures retained RSS against a pristine
+process baseline after a full GC, plus a sampled peak RSS:
+
+* **`ractor-dead-set`** - Every ractor builds a large live set and terminates.
+  Retention shows how much of the dead ractors' final live sets a full GC
+  leaves resident.
+* **`ractor-idle-garbage`** - Every ractor builds a large set, drops all
+  references, then idles without allocating. The garbage cannot be swept
+  while the ractor idles.
+* **`ractor-msg-backlog`** - Unshareable payloads flood the queues of gated
+  consumer ractors, duplicating the payload data per consumer.
+
+```bash
+ruby -Iharness-ractor-mem benchmarks/ractor-dead-set/benchmark.rb
+```
+
+The harness prints `BENCH_METRIC retained_mib=<worst count median>` and
+`BENCH_METRIC peak_mib=...` lines, plus one pair per ractor count. The ractor
+counts and trials are controlled with `RUBY_BENCH_RACTORS` (default `1,2,4,6,8`)
+and `MIN_BENCH_ITRS` (default 3).
+
+The harness collects with `GC.start(global: true)` when the target Ruby's
+`GC.start` accepts the `global:` keyword. Some Ruby 4.1 builds do not accept it.
+On a target with Ractor-local GC, a plain `GC.start` collects only the main
+Ractor's object space. The JSON field `ractor_mem_settle` records `global` or
+`default`.
+
+With `--ractor-gc` (`RUBY_BENCH_RACTOR_GC=1`), every worker samples its own
+Ractor-local GC activity. The results then carry the same `gc_by_ractors` data
+as `harness-ractor`. A scenario wraps each worker body in
+`measure_worker_gc { ... }`, which returns `[result, sample]`. The main Ractor
+passes each sample to `record_worker_gc(worker_index, sample)`. A trial fails
+when its recorded worker indexes are not `0...count`.
+
+Worker samples cover only the workers' own object spaces during the scenario.
+They do not include allocation by the main Ractor, such as the payloads that
+`ractor-msg-backlog` sends. They also do not include the GCs that the harness
+runs to measure retention. The JSON field `gc_controller_samples` covers the
+main Ractor during the scenario.
+
 ## Ruby options
 
 By default, ruby-bench benchmarks the Ruby used for `run_benchmarks.rb`.
@@ -222,6 +268,7 @@ You can find several test harnesses in this repository:
 * harness-stats - count method calls and loop iterations
 * harness-vernier - a harness to profile the benchmark with vernier
 * harness-warmup - a harness which runs as long as needed to find warmed up (peak) performance
+* harness-ractor-mem - a harness for ractor memory pathology benchmarks, measuring retained and peak RSS across ractor counts
 
 To use it, run a benchmark script directly, specifying a harness directory with `-I`:
 
@@ -303,7 +350,8 @@ process's lifetime peak from `getrusage`.
 ## Measuring Ractor GC activity
 
 The `--ractor-gc` option of `run_benchmarks.rb` collects Ractor-local GC
-metrics for benchmarks that use the Ractor harness (`--category ractor`).
+metrics for benchmarks that use `harness-ractor` or `harness-ractor-mem`
+(`--category ractor`).
 The target must use Ruby 4.1 or newer with per-Ractor global GC attribution
 ([ruby/ruby#19147](https://github.com/ruby/ruby/pull/19147)); older targets
 fail before warmup.

benchmarks.yml

@@ -127,6 +127,21 @@ protoboeuf-encode:
   ractor: true
 rack:
   desc: test the performance of the Rack framework with barely any routing.
+ractor-dead-set:
+  desc: multiple ractors each build a large live set and terminate, then memory retention after their death is measured with the ractor memory harness.
+  ractor: true
+  ractor_only: true
+  default_harness: harness-ractor-mem
+ractor-idle-garbage:
+  desc: multiple ractors each build and drop a large garbage set, then idle uncollected while retention is measured with the ractor memory harness.
+  ractor: true
+  ractor_only: true
+  default_harness: harness-ractor-mem
+ractor-msg-backlog:
+  desc: unshareable message payloads flood the queues of gated consumer ractors, duplicating data per consumer and driving peak RSS and retention.
+  ractor: true
+  ractor_only: true
+  default_harness: harness-ractor-mem
 ruby-json:
   desc: an optimized version of the json_pure gem's pure Ruby JSON parser.
   ractor: true

benchmarks/ractor-dead-set/benchmark.rb

@@ -0,0 +1,34 @@
+# Multiple ractors each build a large live set and then terminate.
+# The final live sets of dead ractors are garbage after death. Measured
+# retention shows how much of that memory a full GC fails to reclaim.
+
+Warning[:experimental] = false
+
+require_relative "../../harness/loader"
+
+DEAD_SET_ITEMS = Integer(ENV.fetch("RACTOR_DEAD_SET_ITEMS", 100_000))
+
+run_benchmark(3) do |num_ractors|
+  workers = num_ractors.times.map do |worker|
+    Ractor.new(worker, DEAD_SET_ITEMS) do |worker_id, items|
+      measure_worker_gc do
+        keep = []
+        i = 0
+        while i < items
+          keep << "worker #{worker_id} item #{i} " + ("y" * 100)
+          i += 1
+        end
+        keep.size
+      end
+    end
+  end
+
+  total = 0
+  workers.each_with_index do |worker, worker_id|
+    size, sample = worker.value
+    record_worker_gc(worker_id, sample)
+    total += size
+  end
+  raise "unexpected dead-set size" unless total == num_ractors * DEAD_SET_ITEMS
+  nil
+end

benchmarks/ractor-idle-garbage/benchmark.rb

@@ -0,0 +1,46 @@
+# Multiple ractors each build a large live set, drop all references, then park
+# on Ractor.receive without allocating again. Their garbage cannot be swept by
+# the main Ractor's GC while they idle, so it is retained until each worker
+# is released. The scenario returns a cleanup proc that the harness calls
+# after the retention measurement.
+
+Warning[:experimental] = false
+
+require_relative "../../harness/loader"
+
+IDLE_GARBAGE_ITEMS = Integer(ENV.fetch("RACTOR_IDLE_GARBAGE_ITEMS", 100_000))
+
+run_benchmark(3) do |num_ractors|
+  drained = Ractor.new(num_ractors) do |count|
+    Array.new(count) { Ractor.receive }
+  end
+
+  workers = num_ractors.times.map do |worker|
+    Ractor.new(drained, worker, IDLE_GARBAGE_ITEMS) do |ack, worker_id, items|
+      keep = []
+      _, sample = measure_worker_gc do
+        i = 0
+        while i < items
+          keep << "worker #{worker_id} garbage #{i} " + ("y" * 100)
+          i += 1
+        end
+      end
+      message = Ractor.make_shareable([worker_id, sample])
+      keep = nil
+      ack.send message
+      Ractor.receive
+      :worker_done
+    end
+  end
+
+  drained_workers = drained.value
+  raise "unexpected drain barrier result" unless drained_workers.size == num_ractors
+  drained_workers.each { |worker_id, sample| record_worker_gc(worker_id, sample) }
+
+  proc do
+    workers.each do |worker|
+      worker.send :stop
+      raise "unexpected worker result" unless worker.value == :worker_done
+    end
+  end
+end

benchmarks/ractor-msg-backlog/benchmark.rb

@@ -0,0 +1,49 @@
+# The main Ractor floods the incoming queues of multiple gated consumer
+# ractors with unshareable string payloads. The copies made on send pile up
+# in the queues while the consumers sleep, which duplicates the payload data
+# per consumer and drives peak RSS. Measured retention after the queues are
+# drained shows how much of the copied memory a full GC fails to reclaim.
+# This file intentionally omits frozen_string_literal: frozen payloads are
+# shareable and would not be copied on send.
+
+Warning[:experimental] = false
+
+require_relative "../../harness/loader"
+
+BACKLOG_MESSAGES = Integer(ENV.fetch("RACTOR_BACKLOG_MESSAGES", 20_000))
+BACKLOG_GATE_SLEEP = Float(ENV.fetch("RACTOR_BACKLOG_GATE_SLEEP", 1.0))
+
+run_benchmark(3) do |num_ractors|
+  consumers = num_ractors.times.map do |consumer|
+    Ractor.new(consumer, BACKLOG_GATE_SLEEP) do |consumer_id, gate|
+      measure_worker_gc do
+        sleep gate
+        taken = 0
+        loop do
+          message = Ractor.receive
+          break if message == :done
+          taken += 1
+        end
+        taken
+      end
+    end
+  end
+
+  consumers.each_with_index do |consumer, consumer_id|
+    i = 0
+    while i < BACKLOG_MESSAGES
+      consumer.send "consumer #{consumer_id} message #{i} " + ("x" * 200)
+      i += 1
+    end
+    consumer.send :done
+  end
+
+  total = 0
+  consumers.each_with_index do |consumer, consumer_id|
+    taken, sample = consumer.value
+    record_worker_gc(consumer_id, sample)
+    total += taken
+  end
+  raise "unexpected backlog drain" unless total == num_ractors * BACKLOG_MESSAGES
+  nil
+end