Skip to content

Compute --stats stdDev in display space - #351

Merged
slimbuck merged 2 commits into
playcanvas:mainfrom
slimbuck:stats-dev
Oct 7, 2026
Merged

slimbuck merged 2 commits into
playcanvas:mainfrom
slimbuck:stats-dev

Conversation

@slimbuck

@slimbuck slimbuck commented Oct 7, 2026 •

Copy link
Copy Markdown
Member

--stats pushed the raw-space mean and stdDev through each column's display transform (exp for scale, sigmoid for opacity, the f_dc color transform). That only works for order statistics, so for these columns the reported mean and spread were wrong. For example, raw scales [0, ln 4] showed a mean of exp(mean) = 2 instead of 2.5, and a constant log-scale column showed a stdDev of exp(0) = 1 instead of 0.

StatsAccumulator now also computes a running mean and variance of the transformed values (Welford), exposed as displayMean and displayStdDev on LodStatsData and rounded to 6 significant digits. The running values are kept in units of an exact power-of-two scale, so squared deviations can't overflow while the values are finite; Infinity is reported only when a display value itself overflows (e.g. exp of a very large log-scale). The text table and JSON use them directly, and the JSON reports them as mean and stdDev, so the JSON shape doesn't change. Min, max and median still go through the transform, which is exact because the transforms are monotonic.

Adds tests comparing display-space mean and stdDev with exact values for opacity, scale, f_dc and x, covering the constant-column, overflow, near-overflow (raw scales 699/700) and arithmetic-mean cases.

@slimbuck
slimbuck requested a balanced review from Copilot October 7, 2026 17:22
@slimbuck slimbuck self-assigned this Oct 7, 2026
@slimbuck slimbuck added the enhancement New feature or request label Oct 7, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Nonlinear display means remain incorrect, and finite transformed values can incorrectly produce infinite standard deviations.

2 open findings
What changed in this PR

Adds display-space standard deviation tracking for transformed splat columns and preserves the existing JSON schema.

Changes:

  • Computes transformed-value variance during streaming.
  • Uses display-space standard deviation in text and JSON output.
  • Adds accuracy, constant-column, and overflow tests.
File Description
src/​lib/​ops/​stats.ts Accumulates display-space standard deviation.
src/​lib/​source-info.ts Reports the new value in formatted output.
test/​stats.test.mjs Tests transformed statistics and overflow.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/lib/ops/stats.ts Outdated
Comment thread src/lib/source-info.ts

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Tiny finite display values can underflow during variance accumulation, incorrectly producing zero standard deviation.

0 open findings

2 resolved since last review
Previously missed (1)

In code that hasn't changed since last review

Medium severity Initialize scaling from nonzero display magnitude to prevent underflow

src/​lib/​ops/​stats.ts:166

The scaling only grows from 1, so tiny finite display values can still underflow during the Welford product. For example, raw scales [-700, -699] transform to finite values, but this code reports displayStdDev = 0 instead of about 8.47e-305. Seed the scale from the first nonzero display magnitude, then continue growing it, so the accumulation protects against underflow as well as overflow.

🧠 Review effort: Balanced

@slimbuck
slimbuck requested a review from a team October 7, 2026 17:35
@slimbuck
slimbuck merged commit 0732d21 into playcanvas:main Oct 7, 2026
3 checks passed
@slimbuck
slimbuck deleted the stats-dev branch October 7, 2026 17:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants