Robust analysis of memory consumption in Chonk

[ledwards2225]

Where exactly is memory consumed in Chonk. We need an up to date analysis now that the system has largely stabilized. We can use this to determine whether there is any additional work we can do to drive down memory consumption.

[johnathan79717]

Prover memory profile: where the bytes go

Profiled the AMM flow (ecdsar1+amm_add_liquidity_1_recursions+sponsored_fpc, 19 circuits) using BB_POLY_STATS=1. Here's the memory breakdown and where savings opportunities exist.

Overall memory trace (RSS)

Start:                           172 MiB  (CRS + msgpack parsing)
After EcdsaRAccount (122K):      305 MiB  (+133 MiB)
After private_kernel_init (47K): 342 MiB  (+37 MiB)
... several small circuits ...   342 MiB  (reusing freed memory)
After Token:transfer (180K):     559 MiB  (+217 MiB)  ← largest circuit
After next kernel_inner:         613 MiB  (peak RSS)

RSS is a high-water mark — glibc doesn't return freed pages to the OS. Circuits are processed sequentially; each circuit's ProverInstance is freed after folding. The peak is set by the largest circuit being accumulated.

Per-polynomial breakdown (EcdsaRAccount, 122K trace, representative)

Category	Count	Per-poly MB	Total MB	Values fit in
sigma_1-4	4	3.73	14.9	≤4 bytes (100%)
id_1-4	4	3.73	14.9	≤4 bytes (100%)
w_l/w_r/w_o/w_4	4	3.73	14.9	mixed (30-50% ≤4B, rest ≤32B)
q_m/q_c/q_l/q_r/q_o/q_4	6	3.73	22.4	mixed (sparse, many zeros)
Gate selectors	8	0-1.3	~3.8	≤4 bytes
Tables	4	1.10	4.4	mixed
Other (z_perm, lookup, databus, etc.)	~20	small	~7.2	various
Total	~55		82.5 MB	85.7% compressible

The Compr(MB) column in BB_POLY_STATS shows ideal compressed size: 11.8 MB vs 82.5 MB actual. Most of the gap is from storing small values (≤4 bytes) in 32-byte Fr elements, and from zero elements still occupying 32 bytes.

Where peak memory occurs

During each circuit's accumulation, the peak is at sumcheck round 0:

Original prover polynomials:      82.5 MB (100%)
+ PartiallyEvaluatedMultivariates: ~41 MB (50%, half-size copy)
= Peak during sumcheck:          ~124 MB (150% of polynomial set)

After sumcheck, PEM is freed and batch_polynomials reuses existing polynomial storage (the first polynomial becomes the accumulation target, others are added-scaled into it).

Concrete reduction proposals

1. Compact sigma/id storage (uint32_t compression) — WIP in PR #21124

• Compresses 8 polynomials from 32B to 4B per element during oink+sumcheck
• Savings: ~26 MB for the largest circuit (122K trace), proportionally more for larger traces
• Transparent to readers via Polynomial::operator[] returning Fr by value
• Currently only in UltraProver path; needs to also go in HyperNova folding path

2. Sparse/zero-aware allocation

• Many polynomials are >50% zeros (e.g., q_m: 77% zeros, q_c: 80% zeros)
• Virtual-zero extension already avoids allocating trailing zeros
• Further savings from run-length encoding or bitmap-based sparse storage

3. Free early, allocate late

• Circuit block memory is already freed after copy to prover polynomials (done in chonk::accumulate)
• Selectors could potentially be freed after commitment (VK construction) since they're not needed for sumcheck in all flavors — need to check relation dependencies
• z_perm, lookup_inverses are allocated but zero until computed — could defer allocation

4. malloc_trim between circuits

• Calling malloc_trim(0) after each circuit's fold returns would release freed pages back to OS
• Wouldn't reduce peak but would prevent glibc arena inflation from artificially raising RSS
• Useful for measurement and for reducing memory pressure on constrained devices

5. Polynomial streaming / on-demand decode

• Instead of expand→batch, teach batch_polynomials to read compact sources directly
• add_scaled(other, scalar) already reads via other[index] which handles compact mode
• Only the accumulation target (first polynomial) needs raw data() — all others can stay compact

[ledwards2225]

@johnathan79717 FYI it seems that some of the changes/optimizations we're considering in the near future may effect the memory behavior a bit, e.g. where the peak lies etc. Just more motivation to think about some kind of tooling to make it easy for us to continually reassess the memory situation in the near term

[johnathan79717]

/claudebox suggest tooling options to make it easy for us to continually reassess the memory situation, especially about the peak memory usage, how high it is, what contributes to the peak, etc.

[AztecBot]

ClaudeBox: suggest tooling options to make it easy for us to continually reassess the memory situation, especially about the peak m ... workflow run