Skip to content

BUG: StringDType UTF-8 decoder may read out of bounds #32288

Description

@fweimer-rh

Describe the issue:

Red Hat Product Security has asked me to forward the following report.

RHEL-221682.html

Reproduce the code example:

import numpy as np
from numpy.dtypes import StringDType

# Element 0: 15 ASCII bytes + 0xF0 (truncated 4-byte UTF-8 lead)
# = 16 bytes total, stored in arena (> 15 byte short string limit on 64-bit)
#
# Element 1: 144 bytes (0x90 size byte in arena), starts with 0x80 0x80
#
# OOB read from \xF0: picks up arena size byte 0x90 + data bytes 0x80, 0x80
# Decodes as: U+10000 (valid codepoint, from leaked adjacent data)

x = np.array([
    b'A' * 15 + b'\xF0',               # 16 bytes after NUL trim, in arena
    b'\x80\x80' + b'A' * 142,           # 144 bytes, starts with 0x80 0x80
], dtype='S144')

s = x.astype(StringDType())
u = s.astype('U16')
print(repr(u[0]), [hex(ord(ch)) for ch in u[0]])

Error message:

sizeof size_t equivalent: 8
np.str_('AAAAAAAAAAAAAAA𐀀') ['0x41', '0x41', '0x41', '0x41', '0x41', '0x41', '0x41', '0x41', '0x41', '0x41', '0x41', '0x41', '0x41', '0x41', '0x41', '0x10000']

Python and NumPy Versions:

Python 3.14.6 running numpy 2.4.6

Runtime Environment:

No response

How does this issue affect you or how did you find it:

I've been advised to mention: Found by AISLE in partnership with Red Hat

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions