Smoothing feature contribution and spatial pattern copying from dominant feature

[fcaleca1]

Hi all,
I am working on a simple binary classification task using ExplainableBoostingClassifier, and I have a couple of questions about smoothing feature contributions and an artifact I'm seeing in spatial predictions.

I am trying to get smoother feature contribution plots, especially by reducing jagged contributions across bins. With my default setup of EBM, the contribution of a variable of my interest is

I then tried setting smoothing_rounds == max_rounds (or very large smoothing_rounds) to force smoother versions. The contribution of the same variable becomes smoother, but I have also noticed a change in the scores, particularly in the upper bins: the contribution shifts up compared to the jagged version.

1. Is it expected that increasing smoothing_rounds changes the magnitude of the contribution?
2. Is smoothing_rounds designed to obtain smoother plots, or is it better to force smoothness with other parameters? Are there suggested ranges of values of smoothing_rounds/other parameters to get smoother plots?
3. Should I also try with regularization (reg_alpha, reg_lambda) for this purpose?

My dataset is spatial, and after fitting the model, I display predicted probabilities across space. In both classes, the resulting probability map reproduces spatial pattern of such a predictor (the same one on which I am interested in getting a smoother contribution). This predictor also reports the highest mean absolute score. It looks like the model is relying heavily on this feature (a sort of overfitting). However, when using a "smoother" EBM version, I observed that these artifacts become less pronounced. By artifacts, I mean spatial changes in probabilities that closely follow the changes of that variable. I short, I observe sharp increases or decreases in probabilities that align with where this variable changes.

1. Are there suggested strategies to mitigate this "spatial pattern" copying beyond just increasing smoothing?

[paulbkoch]

Very interesting @fcaleca1 -- We have an example in our docs on smoothing that you might find answers some of your questions: https://interpret.ml/docs/python/examples/interpretable-regression-synthetic.html

The changes in scores you're observing are happening because smoothing rounds converge slower, so if you make all boosting rounds smoothing rounds, then you also need to substantially increase the number of rounds. Your model will underfit otherwise as it doesn't have the time it takes to push the graphs into the +2 range that you're seeing in the unsmoothed graphs. Try doubling both max_rounds and smoothing_rounds and see how that changes things. One nice property of using only smoothing rounds is that you basically can't overfit the model anymore, so boost as long as you can afford. Just for fun, I once ran the sythnethetic dataset in the example above for 10 million smoothing rounds. It took several days, but the graphs were amazing. To be clear, you shouldn't need anything nearly this excessive.

In general, I've been fairly surprised at how unsmooth real-world data is. When trying EBMs on synthetic data with known smooth generation functions like in the documentation above, we're able to recover remarkably smooth shape functions. I've noticed that people trying EBMs for the first time on real world data often initially suspect unsmooth graphs to be overfitting. Most of the time when I've subsequently forced graphs to be smoother, it ends up hurting the model performance though.

smoothing_rounds were initially introduced because boosted EBMs originally start from the null-model where all the graphs are at zero. Often in the first few rounds a single feature dominates in terms of gain, and also a single part of the feature graph dominates. What can happen is that the first few cuts occur at the same location. This might happen even if the underlying generation function is smooth at that location. Perhaps you have a situation where the left half of the graph is negative and the right half is positive and the middle is just a good place to put the initial cuts. What will end up happening in this situation is that you get jumpy graphs at that initial cut location, and the bumpy section tends to remain. Subsequent boosting rounds often correct the larger region, but not necessarily the fine detail because the gain for doing so is small once the large region has been corrected. smoothing rounds prevent this by forcing the first few rounds to cut at random positions. Normally after a set number of smoothing rounds we switch to greedy boosting to add the fine detail that we probably missed during the smoothing rounds.

Let's say that you have a dataset where the underlying data-generation function isn't smooth, but you want the feature to be smooth in the model anyways. In this case you can't allow greedy boosting to run unhindered since the bumpiness in this scenario is a high-gain improvement. The simplest way to achieve this is to make smoothing_rounds == max_rounds as you suggested. A more involved way would be to use model-editing to force the smoothness after fitting. A third way you could do this would be to freeze the feature that you want to keep smooth after a set number of smoothing_rounds, and continue boosting the other features using greedy boosting. This third method is a little difficult to do with the current package, but some of the required components are illustrated in https://interpret.ml/docs/python/examples/custom-interactions.html. You can also try a few other parameters like reg_alpha, reg_lambda, or max_delta_step, although in my experience these have much less effect than smoothing_rounds.

For your specific dataset, if any of your features are lat/long equivalents, I'd recommend you try something like a Hierarchical Triangular Mesh, since that should allow far better clustering of the spatial features. I'd recommend trying this out in-fact with any model.