BUG: fix stale fill_value after ufuncs change MaskedArray dtype by jdavies-st · Pull Request #32423 · numpy/numpy

jdavies-st · 2026-08-25T10:44:32Z

PR summary

MaskedArray._update_from copied _fill_value from the input array unconditionally, with no check that it was still valid for the result's dtype. Ufuncs that change dtype (e.g. np.strings.find on a string masked array, which returns an integer array) left the result holding a fill_value of the old, invalid dtype. The mismatch was silent until something later called _check_fill_value (e.g. .view()), then raising an exception.

_update_from is the common path for both direct ufunc calls (via __array_wrap__) and np.ma's own wrapped ufuncs, so fixing it here covers all dtype-changing ufuncs. When the copied fill_value is no longer valid for the new dtype, fall back to the default fill_value for that dtype instead of propagating the stale value; when it's still valid, keep it unmodified.

Fixes #32401

First time committer introduction

I develop code for the astropy ecosystem, and this bug was found downstream in astropy.table.MaskedColumn which subclasses numpy.ma.MaskedArray, where string searches in column data are common. We are fixing it over it on the astropy side (astropy/astropy#20265), but it would be nice to fix it here as well.

AI Disclosure

AI models were used to help diagnose the bug and fix it.

…y#32423)

ngoldbaum · 2026-08-25T14:39:12Z

It looks like the test you added is failing on 32-bit builds.

jdavies-st · 2026-08-25T16:10:41Z

Thanks! I had forgot that some systems have int32 as the default integer type, so my test was a bit too strict. I've made the unit test more general to handle those cases.

ngoldbaum · 2026-08-25T16:39:39Z

I haven't yet reviewed for correctness, but on a brief glance: can you seriously trim down the comments? I use the following prompt to keep Claude from doing the commenting style in this PR:

Comments shouldn't narrate the code, shouldn't memorialize old bugs, and should be as brief as possible and should document non-local facts that cannot be inferred by reading the accompanying code.

jdavies-st · 2026-08-26T12:25:55Z

Thanks for the reviews @CAOShurong and @ngoldbaum. I've updated the code comments to reflect your concerns.

ngoldbaum

Here's a script that is still returning fishy results under this PR but works fine with my suggestion instead:

import numpy as np
r = np.strings.find(np.ma.array(['ab', 'cd'], mask=[0, 1], fill_value='-1'), 'a')
print(repr(r.filled()))
x = np.ma.array([1.5, np.nan, 2.5], mask=[0, 0, 1]); x.set_fill_value(1e20)
print(repr(np.isnan(x).filled()))
d = np.ma.array(np.array(['2020-01-01', '2020-01-02'], 'M8[D]'), mask=[0, 1])
d.set_fill_value(np.datetime64('NaT', 'D'))
print(repr((d - np.datetime64('2020-01-01', 'D')).filled()))

All of these cases are returned as object arrays because it happens to be castable, so the fill is still stored as the old dtype.

Also, the discarded cast re-runs on every later loop, so you'll see unnecessary repeated warnings in this example:

import warnings, numpy as np
z = np.ma.array([1 + 1j, 2j], mask=[0, 1]); repr(z)
h = np.ma.array([1.5, 2.5], dtype=np.float16, mask=[0, 1]); repr(h)
with warnings.catch_warnings(record=True) as w:
    warnings.simplefilter('always')
    r = np.abs(z); r[1:]; r.copy(); h[1:]; h.copy()
print([c.category.__name__ for c in w])

On this PR, this prints ['ComplexWarning', 'ComplexWarning', 'ComplexWarning', 'RuntimeWarning', 'RuntimeWarning'], while on main and with my suggestion it prints an empty list.

Note that I used an AI to look this over and it found the corner cases.

@CAOShurong drive-by, long, AI-generated comments that more-or-less say "I used an AI model to look this over, it didn't see any issues" are not helpful. Just comment with the one-sentence version.

Co-authored-by: Nathan Goldbaum <nathan.goldbaum@gmail.com>

ngoldbaum · 2026-08-27T16:44:32Z

Sorry for the noise: I posted and deleted some incorrect comments.

ngoldbaum

Sorry for the back-and-forth with myself. Changing masked array is complicated because it's poorly tested but still quite commonly used in the ecosystem and I'm worried about unintended regressions.

What about this diff on top of your PR instead?

diff --git a/numpy/ma/core.py b/numpy/ma/core.py
index 6b14634390..6bd96b1187 100644
--- a/numpy/ma/core.py
+++ b/numpy/ma/core.py
@@ -3039,12 +3039,10 @@ def _update_from(self, obj):
             _optinfo.update(getattr(obj, '__dict__', {}))
         _fill_value = getattr(obj, '_fill_value', None)
         if _fill_value is not None and getattr(obj, 'dtype', None) != self.dtype:
-            if _fill_value.dtype.kind == 'c' and self.dtype.kind in 'biuf':
-                _fill_value = _fill_value.real
             try:
                 _fill_value = _check_fill_value(_fill_value, self.dtype)
-            except (TypeError, ValueError):
-                _fill_value = None
+            except (TypeError, ValueError, OverflowError):
+                _fill_value=None
         _dict = {'_fill_value': _fill_value,
                      '_hardmask': getattr(obj, '_hardmask', False),
                      '_sharedmask': getattr(obj, '_sharedmask', False),

I think this more honestly shows where either floating point or complex number shenanigans are happening. It also catches OverflowError, which might also conceivably happen here.

There is one test (TestMaskedArrayArithmetic::test_basic_ufuncs) that fails because a warning starts being generated that wasn't happening before. IMO that's fine and the way to fix it is to add new warning suppression code to that test.

The point that users might see new warnings should also probably show up in the release note.

jdavies-st · 2026-08-28T10:34:23Z

Agree on using the existing casting infrastructure. I've updated the code per your suggestion, and I've added a case to the unit test exercising this. I also updaated the other tests to expect a ComplexWarning and updated the changelog entry to alert users to this fact.

mhvk

@jdavies-st - this looks good to me, thanks for tackling this!

p.s. Though I'm happy I decided that astropy's Masked class should not to carry a fill value. The obscure benefits really do not seem worth the trouble...

mhvk · 2026-09-02T15:36:45Z

With @ngoldbaum also having approved, I'll go ahead and merge. Thanks again!

Co-authored-by: Nathan Goldbaum <nathan.goldbaum@gmail.com>

BUG: fix stale fill_value after ufuncs change MaskedArray dtype (#32423)

This MR contains the following updates: | Package | Type | Update | Change | OpenSSF | |---|---|---|---|---| | [numpy](https://github.com/numpy/numpy) ([changelog](https://numpy.org/doc/stable/release)) | dependencies | patch | `2.5.2` → `2.5.3` | [![OpenSSF Scorecard](https://api.securityscorecards.dev/projects/github.com/numpy/numpy/badge)](https://securityscorecards.dev/viewer/?uri=github.com/numpy/numpy) | | [openai](https://github.com/openai/openai-python) | dependencies | patch | `2.52.0` → `2.52.1` | [![OpenSSF Scorecard](https://api.securityscorecards.dev/projects/github.com/openai/openai-python/badge)](https://securityscorecards.dev/viewer/?uri=github.com/openai/openai-python) | | [pydantic](https://github.com/pydantic/pydantic) ([changelog](https://docs.pydantic.dev/latest/changelog/)) | dependencies | patch | `2.13.4` → `2.13.5` | [![OpenSSF Scorecard](https://api.securityscorecards.dev/projects/github.com/pydantic/pydantic/badge)](https://securityscorecards.dev/viewer/?uri=github.com/pydantic/pydantic) | | [pydantic-ai](https://github.com/pydantic/pydantic-ai) ([changelog](https://github.com/pydantic/pydantic-ai/releases)) | dependencies | patch | `2.27.0` → `2.27.1` | [![OpenSSF Scorecard](https://api.securityscorecards.dev/projects/github.com/pydantic/pydantic-ai/badge)](https://securityscorecards.dev/viewer/?uri=github.com/pydantic/pydantic-ai) | | [types-requests](https://github.com/python/typeshed) ([changelog](https://github.com/typeshed-internal/stub_uploader/blob/main/data/changelogs/requests.md)) | dependencies | patch | `2.33.0.20260712` → `2.33.0.20260906` | [![OpenSSF Scorecard](https://api.securityscorecards.dev/projects/github.com/python/typeshed/badge)](https://securityscorecards.dev/viewer/?uri=github.com/python/typeshed) | | [uvicorn](https://github.com/Kludex/uvicorn) ([changelog](https://uvicorn.dev/release-notes)) | dependencies | patch | `0.52.3` → `0.52.4` | [![OpenSSF Scorecard](https://api.securityscorecards.dev/projects/github.com/Kludex/uvicorn/badge)](https://securityscorecards.dev/viewer/?uri=github.com/Kludex/uvicorn) | --- ### Release Notes <details> <summary>numpy/numpy (numpy)</summary> ### [`v2.5.3`](https://github.com/numpy/numpy/releases/tag/v2.5.3): (Sep 6, 2026) [Compare Source](numpy/numpy@v2.5.2...v2.5.3) ### NumPy 2.5.3 Release Notes The NumPy 2.5.3 is a patch release that fixes bugs discovered after the 2.5.2 release. Apart from the usual bug and maintenance work, there are a number of StringDType related fixes for problems discovered during the ongoing string work in the main branch. This release supports Python versions 3.12-3.15 #### Changes - Casting a fixed-width byte string array (`np.bytes_`) to `StringDType` now raises `TypeError` when the bytes are not valid UTF-8. Previously the invalid bytes were stored as-is and later caused undefined behavior in string operations. ([gh-32296](numpy/numpy#32296)) - `MaskedArray._fill_value` would become stale when ufuncs that change dtype left the result holding a fill\_value typed for the old dtype. The mismatch was silent until something later called `_check_fill_value`, such as `.view()`, and then a `TypeError` would be raised. Now, when the copied fill\_value is no longer valid for the new dtype, fall back to the default fill\_value for that dtype instead of propagating the stale value. This may raise a `ComplexWarning` if the fill\_value is complex and the new dtype is real. ([gh-32423](numpy/numpy#32423)) #### Contributors A total of 9 people contributed to this release. People with a "+" by their names contributed a patch for the first time. - Charles Harris - Iason Krommydas - James Davies + - Joren Hammudoglu - Maanas Arora - Matti Picus - Nathan Goldbaum - Shikhar Goel + - Yeonho Kim + #### Pull requests merged A total of 27 pull requests were merged for this release. - [#32235](numpy/numpy#32235): MAINT: Prepare 2.5.x for further development - [#32289](numpy/numpy#32289): BUG: raise ValueError when reading into record array with references... - [#32290](numpy/numpy#32290): BUG: avoid uninitialized memory access / NULL-pointer deref in... - [#32291](numpy/numpy#32291): TYP: fix `np.random.{get,set}_bit_generator` implicit re-exports... - [#32292](numpy/numpy#32292): BUG: don't assume strides are a multiple of itemsize in stringdtype... - [#32293](numpy/numpy#32293): CI: fix ccache CC override, add CXX in mac Conda CI ([#32285](numpy/numpy#32285)) - [#32303](numpy/numpy#32303): BUG: Fix ref leak in \[convert\_from\_type]{#convert\_from\_type} for custom scalar types... - [#32304](numpy/numpy#32304): BUG: fix a number of issues around iterators and StringDType... - [#32326](numpy/numpy#32326): TST: avoid allocating huge tuple of arrays in concatenate test... - [#32338](numpy/numpy#32338): BUG: avoid possible UB in 'safe' multiplication helpers ([#32294](numpy/numpy#32294)) - [#32339](numpy/numpy#32339): MAINT: use `PyObject_` functions instead of raw `PyArray_ ones` ([#32331](numpy/numpy#32331)) - [#32378](numpy/numpy#32378): BUG: validate UTF-8 and harden StringDType bounds handling ([#32296](numpy/numpy#32296)) - [#32380](numpy/numpy#32380): BUG: fix two error handling mistakes in stringdtype replace loop... - [#32381](numpy/numpy#32381): BUG: fix visibility annotations for functions in StringDType... - [#32384](numpy/numpy#32384): MAINT: Update ml\_dtypes pin to 8/21/2026. - [#32385](numpy/numpy#32385): MAINT: Update numpy/\_core/src/umath/svml - [#32410](numpy/numpy#32410): MAINT: skip failing cython limited API tests on Cython 3.3.0... - [#32427](numpy/numpy#32427): BUG: close duplicated file descriptor if fdopen fails ([#32386](numpy/numpy#32386)) - [#32428](numpy/numpy#32428): MAINT: add missing space in warning and error messages ([#32405](numpy/numpy#32405)) - [#32430](numpy/numpy#32430): BUG: fix error handling in StringDType to fixed-width bytes case... - [#32441](numpy/numpy#32441): MAINT: Only run nightly BLAS tests on main. - [#32471](numpy/numpy#32471): BUG: fix memory leak in StringDType creation error path ([#32470](numpy/numpy#32470)) - [#32477](numpy/numpy#32477): BUG: fix stale fill\_value after ufuncs change MaskedArray dtype... - [#32478](numpy/numpy#32478): MAINT: exit deadlock tests quickly on slow hardware ([#32466](numpy/numpy#32466)) - [#32481](numpy/numpy#32481): BUG: Backport StringDType byteorder fixes - [#32506](numpy/numpy#32506): DOC: use static scipy doc site for intershpinx ([#32503](numpy/numpy#32503)) - [#32509](numpy/numpy#32509): BUG: fix crash in ufunc.resolve\_dtypes with a Python scalar type... </details> <details> <summary>openai/openai-python (openai)</summary> ### [`v2.52.1`](https://github.com/openai/openai-python/blob/HEAD/CHANGELOG.md#2521-2026-07-31) [Compare Source](openai/openai-python@v2.52.0...v2.52.1) Full Changelog: [v2.52.0...v2.52.1](openai/openai-python@v2.52.0...v2.52.1) ##### Chores - **ci:** pin setup-uv v5 to its underlying commit ([#3560](openai/openai-python#3560)) ([cbdc98b](openai/openai-python@cbdc98b)) </details> <details> <summary>pydantic/pydantic (pydantic)</summary> ### [`v2.13.5`](https://github.com/pydantic/pydantic/releases/tag/v2.13.5) [Compare Source](pydantic/pydantic@v2.13.4...v2.13.5) #### v2.13.5 (2026-08-28) ##### What's Changed ##### Fixes - Allow reuse of validators when plugins are set by [@Viicos](https://github.com/Viicos) in [#13535](pydantic/pydantic#13535) - Fix missing GC traversal on some `pydantic-core` struct fields by [@Viicos](https://github.com/Viicos) in [#13624](pydantic/pydantic#13624) - Fix missing GC traversal in `pydantic-core` for `GeneralFieldsSerializer` by [@Viicos](https://github.com/Viicos) in [#13629](pydantic/pydantic#13629) - Count validated model fields once in smart unions by [@tamird](https://github.com/tamird) in [#13731](pydantic/pydantic#13731) </details> <details> <summary>pydantic/pydantic-ai (pydantic-ai)</summary> ### [`v2.27.1`](https://github.com/pydantic/pydantic-ai/releases/tag/v2.27.1): (2026-08-10) [Compare Source](pydantic/pydantic-ai@v2.27.0...v2.27.1) ##### 🛡️ Security This release fixed an information-disclosure issue: retry-prompt content (validation feedback sent back to the model, which can quote invalid values from its output) was not redacted by `InstrumentationSettings(include_content=False)` when the retry was not tied to a tool call. Now disclosed as [GHSA-3gh4-cghq-f8v4](GHSA-3gh4-cghq-f8v4) (low). Fixed here in `2.27.1` ([#7357](pydantic/pydantic-ai#7357)); v1 users should upgrade to `1.107.4` or later.  #### What's Changed ##### 🐛 Bug Fixes - Restore tool spans for failed argument validation by [@adtyavrdhn](https://github.com/adtyavrdhn) in [#6601](pydantic/pydantic-ai#6601) - Fix `XaiStreamedResponse` finish\_reason mapping for streaming responses by [@pydanty](https://github.com/pydanty)\[bot] in [#6814](pydantic/pydantic-ai#6814) - Point offline web UI hosting at the self-contained chat UI build by [@dsfaccini](https://github.com/dsfaccini) in [#7349](pydantic/pydantic-ai#7349) - Allow adaptive thinking with Tool Output and forced tool choice on Anthropic by [@pydanty](https://github.com/pydanty)\[bot] in [#7200](pydantic/pydantic-ai#7200) - Gate `RetryPromptPart` OpenTelemetry content on `include_content` by [@sean-kim05](https://github.com/sean-kim05) in [#7357](pydantic/pydantic-ai#7357) **Full Changelog**: <pydantic/pydantic-ai@v2.27.0...v2.27.1> </details> <details> <summary>Kludex/uvicorn (uvicorn)</summary> ### [`v0.52.4`](https://github.com/Kludex/uvicorn/releases/tag/0.52.4): Version 0.52.4 [Compare Source](Kludex/uvicorn@0.52.3...0.52.4) ##### Fixed - Remove duplicate `Date` headers from accepted WebSocket handshakes with `websockets-sansio` ([#3078](Kludex/uvicorn#3078)) **Full Changelog**: <Kludex/uvicorn@0.52.3...0.52.4> </details> --- - [ ] If you want to rebase/retry this MR, check this box --- This MR has been generated by [Mend Renovate CLI](https://github.com/renovatebot/renovate).  See merge request swiss-armed-forces/cyber-command/cea/loom!769 Co-authored-by: Loom MR Pipeline Trigger <group_103951964_bot_9504bb8dead6d4e406ad817a607f24be@noreply.gitlab.com> Co-authored-by: shrewd-laidback palace <shrewd-laidback-palace-736-c41-2c1-e464fc974@swiss-armed-forces-open-source.ch>

github-actions Bot added the 00 - Bug label Aug 25, 2026

jdavies-st added a commit to jdavies-st/numpy that referenced this pull request Aug 25, 2026

BUG: fix stale fill_value after ufuncs change MaskedArray dtype (nump…

00d6098

…y#32423)

jdavies-st force-pushed the bugfix-dtype-invalid-fill-value-ufuncs branch from b8fc904 to 00d6098 Compare August 25, 2026 10:46

BUG: fix stale fill_value after ufuncs change MaskedArray dtype (nump…

24e4fed

…y#32423)

jdavies-st force-pushed the bugfix-dtype-invalid-fill-value-ufuncs branch from 00d6098 to 24e4fed Compare August 25, 2026 11:08

charris added the 09 - Backport-Candidate PRs tagged should be backported label Aug 25, 2026

TST: Update fillvalue test to not assume default int is int64

09f088d

This comment was marked as low quality.

Sign in to view

CAOShurong mentioned this pull request Aug 26, 2026

BUG: ma.__array_wrap__ no longer propagates a dtype-invalid fill_value #32438

Closed

TST: Condense and clarify code comments

1a15d81

ngoldbaum reviewed Aug 26, 2026

View reviewed changes

Comment thread numpy/ma/core.py Outdated

jdavies-st and others added 2 commits August 27, 2026 13:09

BUG: Fix _fill_value corner cases were object or warnings are produced

884497e

Co-authored-by: Nathan Goldbaum <nathan.goldbaum@gmail.com>

TST: Add _fill_value corner case tests

050ca5f

This comment was marked as outdated.

Sign in to view

ngoldbaum reviewed Aug 27, 2026

View reviewed changes

jdavies-st added 2 commits August 28, 2026 11:56

BUG: Allow standard complex->real casting w/ComplexWarning

5d744cf

TST: Test OverflowWarning check on fill_value cast

38b644d

ngoldbaum approved these changes Aug 28, 2026

View reviewed changes

ngoldbaum added the triage review Issue/PR to be discussed at the next triage meeting label Aug 31, 2026

mhvk approved these changes Sep 2, 2026

View reviewed changes

mhvk merged commit b265412 into numpy:main Sep 2, 2026
90 checks passed

jdavies-st deleted the bugfix-dtype-invalid-fill-value-ufuncs branch September 2, 2026 16:04

charris mentioned this pull request Sep 2, 2026

BUG: fix stale fill_value after ufuncs change MaskedArray dtype (#32423) #32477

Merged

charris removed 09 - Backport-Candidate PRs tagged should be backported triage review Issue/PR to be discussed at the next triage meeting labels Sep 2, 2026

charris pushed a commit that referenced this pull request Sep 2, 2026

BUG: fix stale fill_value after ufuncs change MaskedArray dtype (#32423)

fc5820e

Co-authored-by: Nathan Goldbaum <nathan.goldbaum@gmail.com>

charris added a commit that referenced this pull request Sep 2, 2026

Merge pull request #32477 from charris/backport-32423

ab1b589

BUG: fix stale fill_value after ufuncs change MaskedArray dtype (#32423)

charris mentioned this pull request Sep 9, 2026

BUG: Reset an unrepresentable fill_value on a dtype change (#32508) #32557

Merged

danggeun mentioned this pull request Sep 9, 2026

BUG: Reset an inherited fill_value in MaskedArray.__new__ as well #32562

Merged

Navigation Menu

Uh oh!

BUG: fix stale fill_value after ufuncs change MaskedArray dtype - #32423

BUG: fix stale fill_value after ufuncs change MaskedArray dtype#32423
mhvk merged 7 commits into
numpy:mainfrom
jdavies-st:bugfix-dtype-invalid-fill-value-ufuncs

jdavies-st commented Aug 25, 2026

Uh oh!

ngoldbaum commented Aug 25, 2026

Uh oh!

jdavies-st commented Aug 25, 2026

Uh oh!

ngoldbaum commented Aug 25, 2026

Uh oh!

This comment was marked as low quality.

Uh oh!

jdavies-st commented Aug 26, 2026

Uh oh!

ngoldbaum left a comment •

edited

Loading

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

ngoldbaum commented Aug 27, 2026

Uh oh!

ngoldbaum left a comment •

edited

Loading

Uh oh!

jdavies-st commented Aug 28, 2026

Uh oh!

mhvk left a comment

Uh oh!

mhvk commented Sep 2, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

jdavies-st commented Aug 25, 2026

PR summary

First time committer introduction

AI Disclosure

Uh oh!

ngoldbaum commented Aug 25, 2026

Uh oh!

jdavies-st commented Aug 25, 2026

Uh oh!

ngoldbaum commented Aug 25, 2026

Uh oh!

This comment was marked as low quality.

Uh oh!

jdavies-st commented Aug 26, 2026

Uh oh!

ngoldbaum left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

ngoldbaum commented Aug 27, 2026

Uh oh!

ngoldbaum left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jdavies-st commented Aug 28, 2026

Uh oh!

mhvk left a comment

Choose a reason for hiding this comment

Uh oh!

mhvk commented Sep 2, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ngoldbaum left a comment •

edited

Loading

ngoldbaum left a comment •

edited

Loading