Skip to content

np.strings.find raises ValueError for a table MaskedColumn #20257

Description

@jdavies-st

Description

Using np.strings.find or np.char.find on a table MaskedColumn that has a string dtype raises an error. It used to work in astropy 6. It seems to be broken in astropy 7 and 8.

Expected behavior

If there is a column of strings, I expect to be able to find the string in the column. This is what worked in past versions of astropy, the latest being version 6.1.7:

In [2]: from astropy.table.column import MaskedColumn
   ...: import numpy as np
   ...: 
   ...: 
   ...: data = ["foo", "bar", "baz"]
   ...: col = MaskedColumn(data=data, mask=False, fill_value="N/A", dtype=np.dtype("<U8"))
   ...: np.char.find(col, "foo")
Out[2]: array([ 0, -1, -1])

And this was with:

In [3]: astropy.__version__
Out[3]: '6.1.7'

In [4]: np.__version__
Out[4]: '1.26.4'

This may seem like an obscure workflow, but when using astroquery to pull JWST data from MAST, the dataURI column in the returned astropy table is a MaskedColumn, and if one doesn't want to download every file that the API returns, one often wants to filter out some files, i.e. not download the *.jpg files or the *_rateints.fits files.

How to Reproduce

from astropy.table.column import MaskedColumn
import numpy as np


data = ["foo", "bar", "baz"]
col = MaskedColumn(data=data, mask=False, fill_value="N/A", dtype=np.dtype("<U8"))
np.strings.find(col, "foo")

And the traceback is:

In [33]: np.strings.find(col, "foo")
Out[33]: ---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
File ~/miniconda3/envs/euclid/lib/python3.12/site-packages/numpy/ma/core.py:510, in _check_fill_value(fill_value, ndtype)
    509 try:
--> 510     fill_value = np.asarray(fill_value, dtype=ndtype)
    511 except (OverflowError, ValueError) as e:
    512     # Raise TypeError instead of OverflowError or ValueError.
    513     # OverflowError is seldom used, and the real problem here is
    514     # that the passed fill_value is not compatible with the ndtype.

ValueError: invalid literal for int() with base 10: np.str_('N/A')

The above exception was the direct cause of the following exception:

TypeError                                 Traceback (most recent call last)
File ~/miniconda3/envs/euclid/lib/python3.12/site-packages/astropy/table/column.py:1338, in Column.__repr__(self)
   1337 def __repr__(self):
-> 1338     return self._base_repr_(html=False)

File ~/miniconda3/envs/euclid/lib/python3.12/site-packages/astropy/table/column.py:1326, in Column._base_repr_(self, html)
   1322     from astropy.utils.xml.writer import xml_escape
   1324     descr = xml_escape(descr)
-> 1326 data_lines, outs = self._formatter._pformat_col(
   1327     self, show_name=False, show_unit=False, show_length=False, html=html
   1328 )
   1330 out = descr + "\n".join(data_lines)
   1332 return out

File ~/miniconda3/envs/euclid/lib/python3.12/site-packages/astropy/table/pprint.py:295, in TableFormatter._pformat_col(self, col, max_lines, show_name, show_unit, show_dtype, show_length, html, align)
    283 col_strs_iter = self._pformat_col_iter(
    284     col,
    285     max_lines,
   (...)    290     outs=outs,
    291 )
    293 # Replace tab and newline with text representations so they display nicely.
    294 # Newline in particular is a problem in a multicolumn table.
--> 295 col_strs = [
    296     val.replace("\t", "\\t").replace("\n", "\\n") for val in col_strs_iter
    297 ]
    298 if len(col_strs) > 0:
    299     col_width = max(len(x) for x in col_strs)

File ~/miniconda3/envs/euclid/lib/python3.12/site-packages/astropy/table/pprint.py:562, in TableFormatter._pformat_col_iter(self, col, max_lines, show_name, show_unit, outs, show_dtype, show_length)
    560 else:
    561     try:
--> 562         yield format_col_str(idx)
    563     except ValueError:
    564         raise ValueError(
    565             f'Unable to parse format string "{col_format}" for '
    566             f'entry "{col[idx]}" in column "{col.info.name}" '
   (...)    569             "for possible format specifications."
    570         )

File ~/miniconda3/envs/euclid/lib/python3.12/site-packages/astropy/table/pprint.py:522, in TableFormatter._pformat_col_iter.<locals>.format_col_str(idx)
    520 def format_col_str(idx):
    521     if not multidims:
--> 522         return format_func(col_format, col if is_scalar else col[idx])
    524     # Prevents columns like Column(data=[[(1,)],[(2,)]], name='a')
    525     # with shape (n,1,...,1) from being printed as if there was
    526     # more than one element in a row
    527     if multidims_all_ones:

File ~/miniconda3/envs/euclid/lib/python3.12/site-packages/astropy/table/_column_mixins.pyx:89, in astropy.table._column_mixins._MaskedColumnGetitemShim.__getitem__()
     87 cdef class _MaskedColumnGetitemShim(_ColumnGetitemShim):
     88     def __getitem__(self, item):
---> 89         return base_getitem(self, item, masked_column_getitem)

File ~/miniconda3/envs/euclid/lib/python3.12/site-packages/astropy/table/_column_mixins.pyx:59, in astropy.table._column_mixins.base_getitem()
     57     return self.data[item]
     58 
---> 59 value = getitem(self, item)
     60 
     61 try:

File ~/miniconda3/envs/euclid/lib/python3.12/site-packages/astropy/table/_column_mixins.pyx:83, in astropy.table._column_mixins.masked_column_getitem()
     81 
     82 cdef inline object masked_column_getitem(object self, object item):
---> 83     value = MaskedArray.__getitem__(self, item)
     84     return self._copy_attrs_slice(value)
     85 

File ~/miniconda3/envs/euclid/lib/python3.12/site-packages/numpy/ma/core.py:3288, in MaskedArray.__getitem__(self, indx)
   3278 """
   3279 x.__getitem__(y) <==> x[y]
   3280 
   3281 Return the item described by i, as a masked array.
   3282 
   3283 """
   3284 # We could directly use ndarray.__getitem__ on self.
   3285 # But then we would have to modify __array_finalize__ to prevent the
   3286 # mask of being reshaped if it hasn't been set up properly yet
   3287 # So it's easier to stick to the current version
-> 3288 dout = self.data[indx]
   3289 _mask = self._mask
   3291 def _is_scalar(m):

File ~/miniconda3/envs/euclid/lib/python3.12/site-packages/astropy/table/column.py:1734, in MaskedColumn.data(self)
   1731 @property
   1732 def data(self):
   1733     """The plain MaskedArray data held by this column."""
-> 1734     out = self.view(np.ma.MaskedArray)
   1735     # By default, a MaskedArray view will set the _baseclass to be the
   1736     # same as that of our own class, i.e., BaseColumn.  Since we want
   1737     # to return a plain MaskedArray, we reset the baseclass accordingly.
   1738     out._baseclass = np.ndarray

File ~/miniconda3/envs/euclid/lib/python3.12/site-packages/numpy/ma/core.py:3264, in MaskedArray.view(self, dtype, type, fill_value)
   3261     type = dtype
   3262     dtype = None
-> 3264 output = super().view(*[a for a in (dtype, type) if a is not None])
   3266 # Make sure to reset the _fill_value if needed
   3267 if getattr(output, '_fill_value', None) is not None:

File ~/miniconda3/envs/euclid/lib/python3.12/site-packages/numpy/ma/core.py:3138, in MaskedArray.__array_finalize__(self, obj)
   3136 # Finalize the fill_value
   3137 if self._fill_value is not None:
-> 3138     self._fill_value = _check_fill_value(self._fill_value, self.dtype)
   3139 elif self.dtype.names is not None:
   3140     # Finalize the default fill_value for structured arrays
   3141     self._fill_value = _check_fill_value(None, self.dtype)

File ~/miniconda3/envs/euclid/lib/python3.12/site-packages/numpy/ma/core.py:516, in _check_fill_value(fill_value, ndtype)
    511     except (OverflowError, ValueError) as e:
    512         # Raise TypeError instead of OverflowError or ValueError.
    513         # OverflowError is seldom used, and the real problem here is
    514         # that the passed fill_value is not compatible with the ndtype.
    515         err_msg = "Cannot convert fill_value %s to dtype %s"
--> 516         raise TypeError(err_msg % (fill_value, ndtype)) from e
    517 return np.array(fill_value)

TypeError: Cannot convert fill_value N/A to dtype int64

It does work if I access the .data property of the MaskedColumn, which of course points to the underlying Numpy MaskedArray:

In [34]: np.strings.find(col.data, "foo")
Out[34]: 
masked_array(data=[ 0, -1, -1],
             mask=False,
       fill_value=np.str_('N/A'))

which implies that it has something to do with the wrapping of MaskedArray by MaskedColumn. The fact that it goes through the __repr__ is a bit suspicious?

Versions

import astropy
astropy.system_info()
platform
--------
platform.platform() = 'macOS-26.6.1-arm64-arm-64bit'
platform.version() = 'Darwin Kernel Version 25.6.0: Sat Jul 11 15:25:26 PDT 2026; root:xnu-12377.161.13~4/RELEASE_ARM64_T6030'
platform.python_version() = '3.12.3'

packages
--------
astropy              8.0.1
numpy                2.5.1
scipy                1.18.0
matplotlib           3.11.1
pandas               --
pyerfa               2.0.1.5

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions