BUG: validate UTF-8 and harden StringDType bounds handling by ngoldbaum · Pull Request #32296 · numpy/numpy

ngoldbaum · 2026-08-14T18:54:31Z

PR summary

Both issues relied on injecting invalid UTF-8 via the bytes to string cast, which did not validate UTF-8.

This led to hangs and UB, which I've defensively hardened in a few spots but the main bugfix is adding UTF-8 validation to the bytes to string cast. I also saw a spot to avoid unnecessary re-scanning in the np.strings.slice loop, which is fixed to avoid possible UTF-8 validation issues.

Also documents that NpyString_Pack doesn't do UTF-8 validation, so input data must be validated as UTF-8 by users of the C API.

Additionally, fixes signed integer overflow issues and issues on 32-bit builds caused by using npy_intp internally but still accepting 64-bit data in the Python API.

AI Disclosure

I used an AI to do code review, address corner cases, and spot UB sites.

ikrommyd

Leaving a few comments. I think a release note is probably needed here too. However, I ran a review with Fable and it pointed out some things that look related and are not part of the exact changes you're making here but may be worth considering as part of this PR if they fix similar problems. I did not read them very carefully though.

🤖 AI text below 🤖

Some pre-existing issues I ran into while reviewing — none are regressions from this PR, they live in code these commits don't touch. Filing for visibility; the first two are Python-reachable instances of the same bug class this PR targets, so they may be worth folding in or spinning into follow-ups.

numpy/_core/src/umath/stringdtype_ufuncs.cpp

L168 · correctness — Same unvalidated-int64-drives-loop class this PR hardens for pad/zfill, still live in multiply_loop_core: a negative factor with an empty string passes the overflow check (0*huge=0) and loops (size_t)factor (~2⁶⁴) times. Confirmed: np.array([''], dtype=StringDType()) * -1 hangs the interpreter with the GIL held.
L176 · memory-leak — multiply_loop_core (L176) and add_strided_loop (L371) leak the PyMem_RawMalloc'd temp buffer when NpyString_pack fails on the in-place path — the exact leak this PR fixes in center/ljust/rjust (L1811) and zfill (L1940), but not in these two siblings.
L1797 · use-after-free — When out= aliases the fillchar array (not the input), descriptors[0]==descriptors[3] is false, so the loop calls load_new_string on the output element — freeing the aliased fill string — before *fill is dereferenced. Confirmed: _center(a, 20, fill, out=fill) is accepted and returns corrupted padding.
L1756 · efficiency — center/ljust/rjust scan each input's codepoints twice (num_codepoints() then string_pad recounts internally), zfill three times; the count could be threaded into string_pad.

numpy/_core/src/umath/string_buffer.h

L1636 · memory-safety — string_zfill reads (and may write) past the output buffer for an empty input: offset == final_width lands tmp one-past-end, *tmp reads 1–4 bytes OOB, and if that byte is '+'/'-' the buffer_memset writes a '0' out of bounds. Reachable via np.strings.zfill(np.array([''], dtype=StringDType()), 20); no test covers it.
L326 · memory-safety — Buffer<UTF8>::operator+= advances with for (int i=0; i<rhs; i++) — an int counter against an npy_int64 bound (same at L727) — so rhs > INT_MAX overflows and walks buf past the allocation. High bar (a single >2GB element), but this PR's int64-width plumbing makes such sizes reachable rather than impossible.

numpy/_core/src/multiarray/stringdtype/utf8_utils.c

L287 · memory-safety — The malformed-lead-byte hardening was added only to slice_strided_loop's manual scan; the shared UTF-8 walkers every other string ufunc uses (find_start_end_locs L287, utf8_character_index L314) still trust num_bytes_for_utf8_character, so the same hang/OOB class survives behind the now-documented-but-unenforced NpyString_pack C API.
L216 · documentation — utf8_buffer_size's doc comment still says it measures "the UTF-32 encoded string stored in s", but it now validates a UTF-8 byte buffer and is the load-bearing validation gate for the bytes/void casts; its trailing-NUL-trimming behavior (which bytes_to_string now depends on) is undocumented at the API boundary.

ikrommyd · 2026-08-17T19:44:03Z

Doesn't this need the same treatment as the "Only add step when another iteration remains" change in string dtype ufuncs below?

ikrommyd · 2026-08-17T19:47:17Z

+            npy_int64 pad_nbytes = safe_mul(
+                    (npy_int64)num_bytes_for_utf8_character((unsigned char *)s2.buf),
+                    width - (npy_int64)num_codepoints, &overflowed);


For a bad byte, num_bytes_for_utf8_character returns 0 right? If so, this would give wrong pad_nbytes.

ikrommyd · 2026-08-17T19:56:00Z

bytes_to_string and void_to_string can become one function tbh if you want.

ikrommyd · 2026-08-17T20:04:07Z

@@ -2310,10 +2341,15 @@ slice_strided_loop(PyArrayMethod_Context *context, char *const data[],
                /* explicitly discard const; initializing new buffer */
                char *buf = (char *)os.buf;


For symmetry with the step != 1 branch too

if (load_new_string(ops, &os, outsize, oallocator, "slice") < 0) { goto fail; } if (outsize > 0) { /* explicitly discard const; initializing new buffer */ char *buf = (char *)os.buf;

…and add

ngoldbaum · 2026-08-19T21:02:24Z

Wow, thanks @ikrommyd, you spotted a lot of user-visible bugs!

There were also some comments about bugs that require using private APIs or using C APIs incorrectly, so I didn't fix those. I did harden internals somewhat so that invalid UTF-8 can't lead to hangs or OOB access.

This has now grown a bit but I still think it's backportable.

ikrommyd

Thanks a lot! It was difficult for me to see what change corresponds to what comment due to the whole force push (I'm not a git expert, maybe there's ways to get better diffs) but I did skim through the code and I do not having anything more to say. I also re-used the fable session I used for the last review and pointed it at my previous review and the last commit of the branch to judge how things have been addressed and it was happy too.

LGTM!

charris · 2026-08-21T14:45:10Z

Let's give it a shot. Thanks Nathan and Iason.

BUG: validate UTF-8 and harden StringDType bounds handling (#32296)

…StringDType casts raises UnicodeDecodeError

This MR contains the following updates: | Package | Type | Update | Change | OpenSSF | |---|---|---|---|---| | [numpy](https://github.com/numpy/numpy) ([changelog](https://numpy.org/doc/stable/release)) | dependencies | patch | `2.5.2` → `2.5.3` | [![OpenSSF Scorecard](https://api.securityscorecards.dev/projects/github.com/numpy/numpy/badge)](https://securityscorecards.dev/viewer/?uri=github.com/numpy/numpy) | | [openai](https://github.com/openai/openai-python) | dependencies | patch | `2.52.0` → `2.52.1` | [![OpenSSF Scorecard](https://api.securityscorecards.dev/projects/github.com/openai/openai-python/badge)](https://securityscorecards.dev/viewer/?uri=github.com/openai/openai-python) | | [pydantic](https://github.com/pydantic/pydantic) ([changelog](https://docs.pydantic.dev/latest/changelog/)) | dependencies | patch | `2.13.4` → `2.13.5` | [![OpenSSF Scorecard](https://api.securityscorecards.dev/projects/github.com/pydantic/pydantic/badge)](https://securityscorecards.dev/viewer/?uri=github.com/pydantic/pydantic) | | [pydantic-ai](https://github.com/pydantic/pydantic-ai) ([changelog](https://github.com/pydantic/pydantic-ai/releases)) | dependencies | patch | `2.27.0` → `2.27.1` | [![OpenSSF Scorecard](https://api.securityscorecards.dev/projects/github.com/pydantic/pydantic-ai/badge)](https://securityscorecards.dev/viewer/?uri=github.com/pydantic/pydantic-ai) | | [types-requests](https://github.com/python/typeshed) ([changelog](https://github.com/typeshed-internal/stub_uploader/blob/main/data/changelogs/requests.md)) | dependencies | patch | `2.33.0.20260712` → `2.33.0.20260906` | [![OpenSSF Scorecard](https://api.securityscorecards.dev/projects/github.com/python/typeshed/badge)](https://securityscorecards.dev/viewer/?uri=github.com/python/typeshed) | | [uvicorn](https://github.com/Kludex/uvicorn) ([changelog](https://uvicorn.dev/release-notes)) | dependencies | patch | `0.52.3` → `0.52.4` | [![OpenSSF Scorecard](https://api.securityscorecards.dev/projects/github.com/Kludex/uvicorn/badge)](https://securityscorecards.dev/viewer/?uri=github.com/Kludex/uvicorn) | --- ### Release Notes <details> <summary>numpy/numpy (numpy)</summary> ### [`v2.5.3`](https://github.com/numpy/numpy/releases/tag/v2.5.3): (Sep 6, 2026) [Compare Source](numpy/numpy@v2.5.2...v2.5.3) ### NumPy 2.5.3 Release Notes The NumPy 2.5.3 is a patch release that fixes bugs discovered after the 2.5.2 release. Apart from the usual bug and maintenance work, there are a number of StringDType related fixes for problems discovered during the ongoing string work in the main branch. This release supports Python versions 3.12-3.15 #### Changes - Casting a fixed-width byte string array (`np.bytes_`) to `StringDType` now raises `TypeError` when the bytes are not valid UTF-8. Previously the invalid bytes were stored as-is and later caused undefined behavior in string operations. ([gh-32296](numpy/numpy#32296)) - `MaskedArray._fill_value` would become stale when ufuncs that change dtype left the result holding a fill\_value typed for the old dtype. The mismatch was silent until something later called `_check_fill_value`, such as `.view()`, and then a `TypeError` would be raised. Now, when the copied fill\_value is no longer valid for the new dtype, fall back to the default fill\_value for that dtype instead of propagating the stale value. This may raise a `ComplexWarning` if the fill\_value is complex and the new dtype is real. ([gh-32423](numpy/numpy#32423)) #### Contributors A total of 9 people contributed to this release. People with a "+" by their names contributed a patch for the first time. - Charles Harris - Iason Krommydas - James Davies + - Joren Hammudoglu - Maanas Arora - Matti Picus - Nathan Goldbaum - Shikhar Goel + - Yeonho Kim + #### Pull requests merged A total of 27 pull requests were merged for this release. - [#32235](numpy/numpy#32235): MAINT: Prepare 2.5.x for further development - [#32289](numpy/numpy#32289): BUG: raise ValueError when reading into record array with references... - [#32290](numpy/numpy#32290): BUG: avoid uninitialized memory access / NULL-pointer deref in... - [#32291](numpy/numpy#32291): TYP: fix `np.random.{get,set}_bit_generator` implicit re-exports... - [#32292](numpy/numpy#32292): BUG: don't assume strides are a multiple of itemsize in stringdtype... - [#32293](numpy/numpy#32293): CI: fix ccache CC override, add CXX in mac Conda CI ([#32285](numpy/numpy#32285)) - [#32303](numpy/numpy#32303): BUG: Fix ref leak in \[convert\_from\_type]{#convert\_from\_type} for custom scalar types... - [#32304](numpy/numpy#32304): BUG: fix a number of issues around iterators and StringDType... - [#32326](numpy/numpy#32326): TST: avoid allocating huge tuple of arrays in concatenate test... - [#32338](numpy/numpy#32338): BUG: avoid possible UB in 'safe' multiplication helpers ([#32294](numpy/numpy#32294)) - [#32339](numpy/numpy#32339): MAINT: use `PyObject_` functions instead of raw `PyArray_ ones` ([#32331](numpy/numpy#32331)) - [#32378](numpy/numpy#32378): BUG: validate UTF-8 and harden StringDType bounds handling ([#32296](numpy/numpy#32296)) - [#32380](numpy/numpy#32380): BUG: fix two error handling mistakes in stringdtype replace loop... - [#32381](numpy/numpy#32381): BUG: fix visibility annotations for functions in StringDType... - [#32384](numpy/numpy#32384): MAINT: Update ml\_dtypes pin to 8/21/2026. - [#32385](numpy/numpy#32385): MAINT: Update numpy/\_core/src/umath/svml - [#32410](numpy/numpy#32410): MAINT: skip failing cython limited API tests on Cython 3.3.0... - [#32427](numpy/numpy#32427): BUG: close duplicated file descriptor if fdopen fails ([#32386](numpy/numpy#32386)) - [#32428](numpy/numpy#32428): MAINT: add missing space in warning and error messages ([#32405](numpy/numpy#32405)) - [#32430](numpy/numpy#32430): BUG: fix error handling in StringDType to fixed-width bytes case... - [#32441](numpy/numpy#32441): MAINT: Only run nightly BLAS tests on main. - [#32471](numpy/numpy#32471): BUG: fix memory leak in StringDType creation error path ([#32470](numpy/numpy#32470)) - [#32477](numpy/numpy#32477): BUG: fix stale fill\_value after ufuncs change MaskedArray dtype... - [#32478](numpy/numpy#32478): MAINT: exit deadlock tests quickly on slow hardware ([#32466](numpy/numpy#32466)) - [#32481](numpy/numpy#32481): BUG: Backport StringDType byteorder fixes - [#32506](numpy/numpy#32506): DOC: use static scipy doc site for intershpinx ([#32503](numpy/numpy#32503)) - [#32509](numpy/numpy#32509): BUG: fix crash in ufunc.resolve\_dtypes with a Python scalar type... </details> <details> <summary>openai/openai-python (openai)</summary> ### [`v2.52.1`](https://github.com/openai/openai-python/blob/HEAD/CHANGELOG.md#2521-2026-07-31) [Compare Source](openai/openai-python@v2.52.0...v2.52.1) Full Changelog: [v2.52.0...v2.52.1](openai/openai-python@v2.52.0...v2.52.1) ##### Chores - **ci:** pin setup-uv v5 to its underlying commit ([#3560](openai/openai-python#3560)) ([cbdc98b](openai/openai-python@cbdc98b)) </details> <details> <summary>pydantic/pydantic (pydantic)</summary> ### [`v2.13.5`](https://github.com/pydantic/pydantic/releases/tag/v2.13.5) [Compare Source](pydantic/pydantic@v2.13.4...v2.13.5) #### v2.13.5 (2026-08-28) ##### What's Changed ##### Fixes - Allow reuse of validators when plugins are set by [@Viicos](https://github.com/Viicos) in [#13535](pydantic/pydantic#13535) - Fix missing GC traversal on some `pydantic-core` struct fields by [@Viicos](https://github.com/Viicos) in [#13624](pydantic/pydantic#13624) - Fix missing GC traversal in `pydantic-core` for `GeneralFieldsSerializer` by [@Viicos](https://github.com/Viicos) in [#13629](pydantic/pydantic#13629) - Count validated model fields once in smart unions by [@tamird](https://github.com/tamird) in [#13731](pydantic/pydantic#13731) </details> <details> <summary>pydantic/pydantic-ai (pydantic-ai)</summary> ### [`v2.27.1`](https://github.com/pydantic/pydantic-ai/releases/tag/v2.27.1): (2026-08-10) [Compare Source](pydantic/pydantic-ai@v2.27.0...v2.27.1) ##### 🛡️ Security This release fixed an information-disclosure issue: retry-prompt content (validation feedback sent back to the model, which can quote invalid values from its output) was not redacted by `InstrumentationSettings(include_content=False)` when the retry was not tied to a tool call. Now disclosed as [GHSA-3gh4-cghq-f8v4](GHSA-3gh4-cghq-f8v4) (low). Fixed here in `2.27.1` ([#7357](pydantic/pydantic-ai#7357)); v1 users should upgrade to `1.107.4` or later.  #### What's Changed ##### 🐛 Bug Fixes - Restore tool spans for failed argument validation by [@adtyavrdhn](https://github.com/adtyavrdhn) in [#6601](pydantic/pydantic-ai#6601) - Fix `XaiStreamedResponse` finish\_reason mapping for streaming responses by [@pydanty](https://github.com/pydanty)\[bot] in [#6814](pydantic/pydantic-ai#6814) - Point offline web UI hosting at the self-contained chat UI build by [@dsfaccini](https://github.com/dsfaccini) in [#7349](pydantic/pydantic-ai#7349) - Allow adaptive thinking with Tool Output and forced tool choice on Anthropic by [@pydanty](https://github.com/pydanty)\[bot] in [#7200](pydantic/pydantic-ai#7200) - Gate `RetryPromptPart` OpenTelemetry content on `include_content` by [@sean-kim05](https://github.com/sean-kim05) in [#7357](pydantic/pydantic-ai#7357) **Full Changelog**: <pydantic/pydantic-ai@v2.27.0...v2.27.1> </details> <details> <summary>Kludex/uvicorn (uvicorn)</summary> ### [`v0.52.4`](https://github.com/Kludex/uvicorn/releases/tag/0.52.4): Version 0.52.4 [Compare Source](Kludex/uvicorn@0.52.3...0.52.4) ##### Fixed - Remove duplicate `Date` headers from accepted WebSocket handshakes with `websockets-sansio` ([#3078](Kludex/uvicorn#3078)) **Full Changelog**: <Kludex/uvicorn@0.52.3...0.52.4> </details> --- - [ ] If you want to rebase/retry this MR, check this box --- This MR has been generated by [Mend Renovate CLI](https://github.com/renovatebot/renovate).  See merge request swiss-armed-forces/cyber-command/cea/loom!769 Co-authored-by: Loom MR Pipeline Trigger <group_103951964_bot_9504bb8dead6d4e406ad817a607f24be@noreply.gitlab.com> Co-authored-by: shrewd-laidback palace <shrewd-laidback-palace-736-c41-2c1-e464fc974@swiss-armed-forces-open-source.ch>

ngoldbaum added 00 - Bug 09 - Backport-Candidate PRs tagged should be backported labels Aug 14, 2026

ngoldbaum force-pushed the fix-stringdtype-utf-8 branch from 3cd53ac to a0c85a0 Compare August 14, 2026 22:34

ikrommyd reviewed Aug 17, 2026

View reviewed changes

ngoldbaum added 10 commits August 19, 2026 14:56

BUG: validate UTF-8 and harden StringDType bounds handling

d76eece

BUG: fix edge case for slicing with a step of min or max of intp

31563cb

BUG: harden StringDType UTF-8 walkers against invalid stored bytes

011fe97

BUG: fix StringDType multiply hang on empty or non-positive factor

e29b5d7

BUG: fix StringDType zfill out-of-bounds access when no padding is added

477e304

BUG: avoid pointer overflow in the fixed-width string slice loop

2f3d5d0

BUG: reject a zero slice step in the string slice loops

4db37e6

BUG: fix StringDType pad use-after-free when out aliases the fill array

c78ac5d

BUG: fix StringDType temp buffer leak when packing fails in multiply …

309f139

…and add

DOC: release note and comment fixes for StringDType UTF-8 validation

f147efb

ngoldbaum force-pushed the fix-stringdtype-utf-8 branch from a0c85a0 to f147efb Compare August 19, 2026 20:57

BUG: fix test for overflow on 32 bit platforms

0c548cf

ikrommyd approved these changes Aug 19, 2026

View reviewed changes

charris merged commit 923f0dc into numpy:main Aug 21, 2026
114 of 118 checks passed

charris mentioned this pull request Aug 21, 2026

BUG: validate UTF-8 and harden StringDType bounds handling (#32296) #32378

Merged

charris removed the 09 - Backport-Candidate PRs tagged should be backported label Aug 21, 2026

charris pushed a commit that referenced this pull request Aug 21, 2026

BUG: validate UTF-8 and harden StringDType bounds handling (#32296)

872e600

charris added a commit that referenced this pull request Aug 21, 2026

Merge pull request #32378 from charris/backport-32296

cb0923d

BUG: validate UTF-8 and harden StringDType bounds handling (#32296)

ngoldbaum added a commit to ngoldbaum/numpy that referenced this pull request Aug 24, 2026

MAINT: reconcile with numpygh-32296, invalid UTF-8 in fixed-width to …

73b95d1

…StringDType casts raises UnicodeDecodeError

ngoldbaum added a commit to ngoldbaum/numpy that referenced this pull request Sep 8, 2026

BUG: validate UTF-8 and harden StringDType bounds handling (numpy#32296)

99caee0

ngoldbaum added a commit to ngoldbaum/numpy that referenced this pull request Sep 8, 2026

BUG: validate UTF-8 and harden StringDType bounds handling (numpy#32296)

8852f9a

Navigation Menu

Uh oh!

BUG: validate UTF-8 and harden StringDType bounds handling - #32296

BUG: validate UTF-8 and harden StringDType bounds handling#32296
charris merged 11 commits into
numpy:mainfrom
ngoldbaum:fix-stringdtype-utf-8

ngoldbaum commented Aug 14, 2026 •

edited

Loading

Uh oh!

ikrommyd left a comment •

edited

Loading

Uh oh!

ikrommyd Aug 17, 2026

Uh oh!

ikrommyd Aug 17, 2026

Uh oh!

Uh oh!

Uh oh!

ikrommyd Aug 17, 2026

Uh oh!

ikrommyd Aug 17, 2026 •

edited

Loading

Uh oh!

ngoldbaum commented Aug 19, 2026

Uh oh!

ikrommyd left a comment

Uh oh!

Uh oh!

charris commented Aug 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		@@ -2310,10 +2341,15 @@ slice_strided_loop(PyArrayMethod_Context context, char const data[],
		/* explicitly discard const; initializing new buffer */
		char buf = (char )os.buf;

Uh oh!

Conversation

ngoldbaum commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR summary

AI Disclosure

Uh oh!

ikrommyd left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ikrommyd Aug 17, 2026

Choose a reason for hiding this comment

Uh oh!

ikrommyd Aug 17, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ikrommyd Aug 17, 2026

Choose a reason for hiding this comment

Uh oh!

ikrommyd Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ngoldbaum commented Aug 19, 2026

Uh oh!

ikrommyd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

charris commented Aug 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ngoldbaum commented Aug 14, 2026 •

edited

Loading

ikrommyd left a comment •

edited

Loading

ikrommyd Aug 17, 2026 •

edited

Loading