Fix masking for fill_value and missing_value for numpy 2.4.0#304
Fix masking for fill_value and missing_value for numpy 2.4.0#304valeriupredoi wants to merge 8 commits intomainfrom
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #304 +/- ##
==========================================
+ Coverage 86.04% 86.21% +0.17%
==========================================
Files 7 7
Lines 645 653 +8
==========================================
+ Hits 555 563 +8
Misses 90 90 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
activestorage/storage.py
Outdated
| if missing_value is not None: | ||
| data = np.ma.masked_equal(data, missing_value) | ||
| if isinstance(missing_value, np.ndarray) or isinstance(missing_value, list): | ||
| data = np.ma.masked_where(data == missing_value, data) |
There was a problem hiding this comment.
Surely this only works when missing_value broadcasts to data, which in general it wont.
There was a problem hiding this comment.
well I'm running out of any valid numpy function that can mask with an array then 😁
There was a problem hiding this comment.
sorry, I got distracted by the news, how do you even apply a missing_value mask array if it's not broadcastable? I am really not familiar with anything missing value that's dim > 1 (or not scalar) - please suggest code, buds 🍺
There was a problem hiding this comment.
@davidhassell sorry - only just now that I refactored - given this is for storage.py, and that's just an emulator of Reductionist, I decided to pop a ValueError instead of going through with your albeit solid suggestion to go through the individual missing values - you reckon that'll do it? 🍺
|
frenly ping @davidhassell when you have a moment for this one, buds 🍻 |
Description
I merged too fast #303 (like a .303 bullet hehe) but @davidhassell mentioned it's not complete since missing_value could be a multi-dim array, not just a size one, as I thought. This should fix that.
Reminder: numpy 2.4.0 doesn't allow stuff like ma.masked_equal or other such funcs that set fill_values, to have arrays passed for fill_value, eg their deprecation:
This implementation here should be backwards compatible.
Before you get started
Checklist