Skip to content

fix(outliers): spikes and dips in one segment no longer mask each other - #31

Merged
dgnaglo merged 2 commits into
mainfrom
fix/outliers-no-masking
Oct 12, 2026
Merged

dgnaglo merged 2 commits into
mainfrom
fix/outliers-no-masking

Conversation

@dgnaglo

@dgnaglo dgnaglo commented Oct 12, 2026

Copy link
Copy Markdown
Contributor

Problem

With preprocessing: {"outliers": true}, several spikes or dips in the same series mask each other. Seen on agent-dev (engine 0.1.5) with 24 months of trend (100 -> 184), spikes at 950 and 900 and a dip at 5: only the 900 was corrected, and it was brought to 212 while the trend is at ~169 there.

outliers_detection(method_ls = "cpt") tests |x - mean| > 3 sd inside each changepoint segment, with the mean and sd computed with the spikes in them. Recomputed on that series (cut at 11): segment 1-11 has sd 257, so 950 sits at 2.99 sd and 5 at 0.69 sd; segment 11-24 catches 900 at 3.46 sd and replaces it with the segment mean, 212.4 (the API returned 212).

Fix

  • Inside each segment, a point is an outlier when it is more than 3 sd from the segment mean (unchanged rule), or more than 5 sd from the mean of the other points, sd of the other points too, the farthest point first, until none is left (.segment_outliers()). Leave-one-out scores are computed in O(n) per pass (.deleted_z()), so long segments stay cheap.
  • Outliers are replaced by a linear interpolation of their non-outlier neighbours, instead of the segment mean, which the spike pulled up and which ignored the trend.
  • docs/API.md and NEWS.md updated. .merge_short_segments() is unchanged.

On the series above: 3 corrections, 950 -> 114, 5 -> 132, 900 -> 166.

Bench

timetk::m4_monthly, defects injected at random positions, 100 draws per case, same script and seed before/after (absolute numbers differ from #26's table, which used another script; the deltas are comparable). "corrected" = the point ends up at less than half of its injected deviation from the true value; "error" = median |corrected - true| / true over corrected points; "other changed" = points changed that were not injected, per series. multi = a x3 spike, a -80 % dip and a x2 spike in the same series.

Window Case Corrected before Corrected after Error before Error after Other changed before Other changed after
36 months spike x3 93 % 95 % 11.6 % 6.4 % 0.03 0.08
36 months spike x2 75 % 82 % 8.9 % 4.2 % 0.02 0.06
36 months dip -80 % 73 % 76 % 5.7 % 4.9 % 0.04 0.05
36 months multi 43 % 63 % 11.3 % 3.6 % 0.01 0.06
36 months clean series 0.04 0.07
60 months spike x3 91 % 95 % 10.8 % 3.6 % 0.06 0.11
60 months spike x2 82 % 86 % 7.3 % 5.1 % 0.06 0.07
60 months dip -80 % 78 % 79 % 7.7 % 4.3 % 0.07 0.08
60 months multi 52 % 69 % 8.2 % 5.3 % 0.05 0.07
60 months clean series 0.11 0.12
full series spike x3 97 % 97 % 11.2 % 4.5 % 0.83 1.07
full series spike x2 79 % 80 % 10.9 % 4.5 % 0.90 1.18
full series dip -80 % 78 % 79 % 9.0 % 6.1 % 0.90 1.14
full series multi 78 % 83 % 9.5 % 6.0 % 0.73 1.02
full series clean series 0.95 1.20

Detection never goes down; false corrections go up by 0.03 per clean 36-month series and 0.25 per full series.

Variants tried and dropped (same bench):

  • robust STL + super smoother residuals, 3 IQR (the forecast::tsoutliers() approach, base stats only): 1.36 false corrections per clean 36-month series;
  • leave-one-out only, without the 3 sd rule: at 3 sd, 1.2 false corrections per series; at 5 sd, below the current rule on single spikes and dips (73 % / 69 % vs 75 % / 73 %); at 6 sd and above, the multi gain is gone.
  • The 5 sd threshold of the combined rule: 4 sd doubles false corrections (0.18 per clean 36-month series), 6 sd loses 7 points on multi.

Tests

  • tests/testthat/test-outliers-spikes.R: the 24-month series above (3 points changed, replacements close to the neighbours), the API report lists the 3 dates, and .deleted_z() matches a direct leave-one-out computation. Against main (0.1.5 image, pkgload::load_all()): 7 failures/errors; after: all pass.
  • Full suite against this branch installed in a private library of the apowerb/th2forecast:0.1.5 image, as CI tests the installed package: test_dir(..., stop_on_failure = TRUE) exit 0.

Not changed

  • The trend is still not removed before the test (a spike smaller than the seasonal swing can still be missed; anomalies is the step for that).
  • preprocessing_data() also calls outliers_detection(method_ls = "cpt") and gets the same new behaviour.

The cpt outlier test computed the segment mean and sd with the spikes in
them: a big spike inflated the sd and hid itself and its neighbours. A
point is now also an outlier when it is more than 5 sd from the mean of
the other points (leave-one-out, farthest first, O(n) per pass), and
outliers are replaced by a linear interpolation of their neighbours
instead of the segment mean.

[review: diff relu (seuils, segments courts, séries intermittentes, coût O(n)) ; tests ciblés rouge→vert ; suite complète verte sur la branche installée]
uvr sync --frozen failed in the image build: CRAN published globals 0.20.0, so the lock no longer matched the manifest. Regenerated with uvr 0.4.6 (same version as the Dockerfile); only globals changes.

[review: diff du lock relu, 1 paquet (globals 0.19.1 -> 0.20.0)]
@dgnaglo
dgnaglo merged commit 0e4e416 into main Oct 12, 2026
6 checks passed
@dgnaglo
dgnaglo deleted the fix/outliers-no-masking branch October 12, 2026 11:35
@dgnaglo dgnaglo mentioned this pull request Oct 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant