BUG: validate UTF-8 and harden StringDType bounds handling (#32296) - #32378
Merged
Merged
Conversation
736-c41-2c1-e464fc974
added a commit
to Swiss-Armed-Forces/Loom
that referenced
this pull request
Sep 15, 2026
This MR contains the following updates: | Package | Type | Update | Change | OpenSSF | |---|---|---|---|---| | [numpy](https://github.com/numpy/numpy) ([changelog](https://numpy.org/doc/stable/release)) | dependencies | patch | `2.5.2` → `2.5.3` | [](https://securityscorecards.dev/viewer/?uri=github.com/numpy/numpy) | | [openai](https://github.com/openai/openai-python) | dependencies | patch | `2.52.0` → `2.52.1` | [](https://securityscorecards.dev/viewer/?uri=github.com/openai/openai-python) | | [pydantic](https://github.com/pydantic/pydantic) ([changelog](https://docs.pydantic.dev/latest/changelog/)) | dependencies | patch | `2.13.4` → `2.13.5` | [](https://securityscorecards.dev/viewer/?uri=github.com/pydantic/pydantic) | | [pydantic-ai](https://github.com/pydantic/pydantic-ai) ([changelog](https://github.com/pydantic/pydantic-ai/releases)) | dependencies | patch | `2.27.0` → `2.27.1` | [](https://securityscorecards.dev/viewer/?uri=github.com/pydantic/pydantic-ai) | | [types-requests](https://github.com/python/typeshed) ([changelog](https://github.com/typeshed-internal/stub_uploader/blob/main/data/changelogs/requests.md)) | dependencies | patch | `2.33.0.20260712` → `2.33.0.20260906` | [](https://securityscorecards.dev/viewer/?uri=github.com/python/typeshed) | | [uvicorn](https://github.com/Kludex/uvicorn) ([changelog](https://uvicorn.dev/release-notes)) | dependencies | patch | `0.52.3` → `0.52.4` | [](https://securityscorecards.dev/viewer/?uri=github.com/Kludex/uvicorn) | --- ### Release Notes <details> <summary>numpy/numpy (numpy)</summary> ### [`v2.5.3`](https://github.com/numpy/numpy/releases/tag/v2.5.3): (Sep 6, 2026) [Compare Source](numpy/numpy@v2.5.2...v2.5.3) ### NumPy 2.5.3 Release Notes The NumPy 2.5.3 is a patch release that fixes bugs discovered after the 2.5.2 release. Apart from the usual bug and maintenance work, there are a number of StringDType related fixes for problems discovered during the ongoing string work in the main branch. This release supports Python versions 3.12-3.15 #### Changes - Casting a fixed-width byte string array (`np.bytes_`) to `StringDType` now raises `TypeError` when the bytes are not valid UTF-8. Previously the invalid bytes were stored as-is and later caused undefined behavior in string operations. ([gh-32296](numpy/numpy#32296)) - `MaskedArray._fill_value` would become stale when ufuncs that change dtype left the result holding a fill\_value typed for the old dtype. The mismatch was silent until something later called `_check_fill_value`, such as `.view()`, and then a `TypeError` would be raised. Now, when the copied fill\_value is no longer valid for the new dtype, fall back to the default fill\_value for that dtype instead of propagating the stale value. This may raise a `ComplexWarning` if the fill\_value is complex and the new dtype is real. ([gh-32423](numpy/numpy#32423)) #### Contributors A total of 9 people contributed to this release. People with a "+" by their names contributed a patch for the first time. - Charles Harris - Iason Krommydas - James Davies + - Joren Hammudoglu - Maanas Arora - Matti Picus - Nathan Goldbaum - Shikhar Goel + - Yeonho Kim + #### Pull requests merged A total of 27 pull requests were merged for this release. - [#​32235](numpy/numpy#32235): MAINT: Prepare 2.5.x for further development - [#​32289](numpy/numpy#32289): BUG: raise ValueError when reading into record array with references... - [#​32290](numpy/numpy#32290): BUG: avoid uninitialized memory access / NULL-pointer deref in... - [#​32291](numpy/numpy#32291): TYP: fix `np.random.{get,set}_bit_generator` implicit re-exports... - [#​32292](numpy/numpy#32292): BUG: don't assume strides are a multiple of itemsize in stringdtype... - [#​32293](numpy/numpy#32293): CI: fix ccache CC override, add CXX in mac Conda CI ([#​32285](numpy/numpy#32285)) - [#​32303](numpy/numpy#32303): BUG: Fix ref leak in \[convert\_from\_type]{#convert\_from\_type} for custom scalar types... - [#​32304](numpy/numpy#32304): BUG: fix a number of issues around iterators and StringDType... - [#​32326](numpy/numpy#32326): TST: avoid allocating huge tuple of arrays in concatenate test... - [#​32338](numpy/numpy#32338): BUG: avoid possible UB in 'safe' multiplication helpers ([#​32294](numpy/numpy#32294)) - [#​32339](numpy/numpy#32339): MAINT: use `PyObject_` functions instead of raw `PyArray_ ones` ([#​32331](numpy/numpy#32331)) - [#​32378](numpy/numpy#32378): BUG: validate UTF-8 and harden StringDType bounds handling ([#​32296](numpy/numpy#32296)) - [#​32380](numpy/numpy#32380): BUG: fix two error handling mistakes in stringdtype replace loop... - [#​32381](numpy/numpy#32381): BUG: fix visibility annotations for functions in StringDType... - [#​32384](numpy/numpy#32384): MAINT: Update ml\_dtypes pin to 8/21/2026. - [#​32385](numpy/numpy#32385): MAINT: Update numpy/\_core/src/umath/svml - [#​32410](numpy/numpy#32410): MAINT: skip failing cython limited API tests on Cython 3.3.0... - [#​32427](numpy/numpy#32427): BUG: close duplicated file descriptor if fdopen fails ([#​32386](numpy/numpy#32386)) - [#​32428](numpy/numpy#32428): MAINT: add missing space in warning and error messages ([#​32405](numpy/numpy#32405)) - [#​32430](numpy/numpy#32430): BUG: fix error handling in StringDType to fixed-width bytes case... - [#​32441](numpy/numpy#32441): MAINT: Only run nightly BLAS tests on main. - [#​32471](numpy/numpy#32471): BUG: fix memory leak in StringDType creation error path ([#​32470](numpy/numpy#32470)) - [#​32477](numpy/numpy#32477): BUG: fix stale fill\_value after ufuncs change MaskedArray dtype... - [#​32478](numpy/numpy#32478): MAINT: exit deadlock tests quickly on slow hardware ([#​32466](numpy/numpy#32466)) - [#​32481](numpy/numpy#32481): BUG: Backport StringDType byteorder fixes - [#​32506](numpy/numpy#32506): DOC: use static scipy doc site for intershpinx ([#​32503](numpy/numpy#32503)) - [#​32509](numpy/numpy#32509): BUG: fix crash in ufunc.resolve\_dtypes with a Python scalar type... </details> <details> <summary>openai/openai-python (openai)</summary> ### [`v2.52.1`](https://github.com/openai/openai-python/blob/HEAD/CHANGELOG.md#2521-2026-07-31) [Compare Source](openai/openai-python@v2.52.0...v2.52.1) Full Changelog: [v2.52.0...v2.52.1](openai/openai-python@v2.52.0...v2.52.1) ##### Chores - **ci:** pin setup-uv v5 to its underlying commit ([#​3560](openai/openai-python#3560)) ([cbdc98b](openai/openai-python@cbdc98b)) </details> <details> <summary>pydantic/pydantic (pydantic)</summary> ### [`v2.13.5`](https://github.com/pydantic/pydantic/releases/tag/v2.13.5) [Compare Source](pydantic/pydantic@v2.13.4...v2.13.5) #### v2.13.5 (2026-08-28) ##### What's Changed ##### Fixes - Allow reuse of validators when plugins are set by [@​Viicos](https://github.com/Viicos) in [#​13535](pydantic/pydantic#13535) - Fix missing GC traversal on some `pydantic-core` struct fields by [@​Viicos](https://github.com/Viicos) in [#​13624](pydantic/pydantic#13624) - Fix missing GC traversal in `pydantic-core` for `GeneralFieldsSerializer` by [@​Viicos](https://github.com/Viicos) in [#​13629](pydantic/pydantic#13629) - Count validated model fields once in smart unions by [@​tamird](https://github.com/tamird) in [#​13731](pydantic/pydantic#13731) </details> <details> <summary>pydantic/pydantic-ai (pydantic-ai)</summary> ### [`v2.27.1`](https://github.com/pydantic/pydantic-ai/releases/tag/v2.27.1): (2026-08-10) [Compare Source](pydantic/pydantic-ai@v2.27.0...v2.27.1) ##### 🛡️ Security This release fixed an information-disclosure issue: retry-prompt content (validation feedback sent back to the model, which can quote invalid values from its output) was not redacted by `InstrumentationSettings(include_content=False)` when the retry was not tied to a tool call. Now disclosed as [GHSA-3gh4-cghq-f8v4](GHSA-3gh4-cghq-f8v4) (low). Fixed here in `2.27.1` ([#​7357](pydantic/pydantic-ai#7357)); v1 users should upgrade to `1.107.4` or later. <!-- Release notes generated using configuration in .github/release.yml at main --> #### What's Changed ##### 🐛 Bug Fixes - Restore tool spans for failed argument validation by [@​adtyavrdhn](https://github.com/adtyavrdhn) in [#​6601](pydantic/pydantic-ai#6601) - Fix `XaiStreamedResponse` finish\_reason mapping for streaming responses by [@​pydanty](https://github.com/pydanty)\[bot] in [#​6814](pydantic/pydantic-ai#6814) - Point offline web UI hosting at the self-contained chat UI build by [@​dsfaccini](https://github.com/dsfaccini) in [#​7349](pydantic/pydantic-ai#7349) - Allow adaptive thinking with Tool Output and forced tool choice on Anthropic by [@​pydanty](https://github.com/pydanty)\[bot] in [#​7200](pydantic/pydantic-ai#7200) - Gate `RetryPromptPart` OpenTelemetry content on `include_content` by [@​sean-kim05](https://github.com/sean-kim05) in [#​7357](pydantic/pydantic-ai#7357) **Full Changelog**: <pydantic/pydantic-ai@v2.27.0...v2.27.1> </details> <details> <summary>Kludex/uvicorn (uvicorn)</summary> ### [`v0.52.4`](https://github.com/Kludex/uvicorn/releases/tag/0.52.4): Version 0.52.4 [Compare Source](Kludex/uvicorn@0.52.3...0.52.4) ##### Fixed - Remove duplicate `Date` headers from accepted WebSocket handshakes with `websockets-sansio` ([#​3078](Kludex/uvicorn#3078)) **Full Changelog**: <Kludex/uvicorn@0.52.3...0.52.4> </details> --- - [ ] <!-- rebase-check -->If you want to rebase/retry this MR, check this box --- This MR has been generated by [Mend Renovate CLI](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC4zOS4yIiwidXBkYXRlZEluVmVyIjoiNDQuOTAuMiIsInRhcmdldEJyYW5jaCI6Im1haW4iLCJsYWJlbHMiOlsiZGVwZW5kZW5jaWVzIiwicmVub3ZhdGUiXX0=--> See merge request swiss-armed-forces/cyber-command/cea/loom!769 Co-authored-by: Loom MR Pipeline Trigger <group_103951964_bot_9504bb8dead6d4e406ad817a607f24be@noreply.gitlab.com> Co-authored-by: shrewd-laidback palace <shrewd-laidback-palace-736-c41-2c1-e464fc974@swiss-armed-forces-open-source.ch>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport of #32296.
PR summary
Fixes #32287, Fixes #32288
Both issues relied on injecting invalid UTF-8 via the bytes to string cast, which did not validate UTF-8.
This led to hangs and UB, which I've defensively hardened in a few spots but the main bugfix is adding UTF-8 validation to the bytes to string cast. I also saw a spot to avoid unnecessary re-scanning in the np.strings.slice loop, which is fixed to avoid possible UTF-8 validation issues.
Also documents that
NpyString_Packdoesn't do UTF-8 validation, so input data must be validated as UTF-8 by users of the C API.Additionally, fixes signed integer overflow issues and issues on 32-bit builds caused by using
npy_intpinternally but still accepting 64-bit data in the Python API.AI Disclosure
I used an AI to do code review, address corner cases, and spot UB sites.