Description
Using np.strings.find or np.char.find on a table MaskedColumn that has a string dtype raises an error. It used to work in astropy 6. It seems to be broken in astropy 7 and 8.
Expected behavior
If there is a column of strings, I expect to be able to find the string in the column. This is what worked in past versions of astropy, the latest being version 6.1.7:
In [2]: from astropy.table.column import MaskedColumn
...: import numpy as np
...:
...:
...: data = ["foo", "bar", "baz"]
...: col = MaskedColumn(data=data, mask=False, fill_value="N/A", dtype=np.dtype("<U8"))
...: np.char.find(col, "foo")
Out[2]: array([ 0, -1, -1])
And this was with:
In [3]: astropy.__version__
Out[3]: '6.1.7'
In [4]: np.__version__
Out[4]: '1.26.4'
This may seem like an obscure workflow, but when using astroquery to pull JWST data from MAST, the dataURI column in the returned astropy table is a MaskedColumn, and if one doesn't want to download every file that the API returns, one often wants to filter out some files, i.e. not download the *.jpg files or the *_rateints.fits files.
How to Reproduce
from astropy.table.column import MaskedColumn
import numpy as np
data = ["foo", "bar", "baz"]
col = MaskedColumn(data=data, mask=False, fill_value="N/A", dtype=np.dtype("<U8"))
np.strings.find(col, "foo")
And the traceback is:
In [33]: np.strings.find(col, "foo")
Out[33]: ---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
File ~/miniconda3/envs/euclid/lib/python3.12/site-packages/numpy/ma/core.py:510, in _check_fill_value(fill_value, ndtype)
509 try:
--> 510 fill_value = np.asarray(fill_value, dtype=ndtype)
511 except (OverflowError, ValueError) as e:
512 # Raise TypeError instead of OverflowError or ValueError.
513 # OverflowError is seldom used, and the real problem here is
514 # that the passed fill_value is not compatible with the ndtype.
ValueError: invalid literal for int() with base 10: np.str_('N/A')
The above exception was the direct cause of the following exception:
TypeError Traceback (most recent call last)
File ~/miniconda3/envs/euclid/lib/python3.12/site-packages/astropy/table/column.py:1338, in Column.__repr__(self)
1337 def __repr__(self):
-> 1338 return self._base_repr_(html=False)
File ~/miniconda3/envs/euclid/lib/python3.12/site-packages/astropy/table/column.py:1326, in Column._base_repr_(self, html)
1322 from astropy.utils.xml.writer import xml_escape
1324 descr = xml_escape(descr)
-> 1326 data_lines, outs = self._formatter._pformat_col(
1327 self, show_name=False, show_unit=False, show_length=False, html=html
1328 )
1330 out = descr + "\n".join(data_lines)
1332 return out
File ~/miniconda3/envs/euclid/lib/python3.12/site-packages/astropy/table/pprint.py:295, in TableFormatter._pformat_col(self, col, max_lines, show_name, show_unit, show_dtype, show_length, html, align)
283 col_strs_iter = self._pformat_col_iter(
284 col,
285 max_lines,
(...) 290 outs=outs,
291 )
293 # Replace tab and newline with text representations so they display nicely.
294 # Newline in particular is a problem in a multicolumn table.
--> 295 col_strs = [
296 val.replace("\t", "\\t").replace("\n", "\\n") for val in col_strs_iter
297 ]
298 if len(col_strs) > 0:
299 col_width = max(len(x) for x in col_strs)
File ~/miniconda3/envs/euclid/lib/python3.12/site-packages/astropy/table/pprint.py:562, in TableFormatter._pformat_col_iter(self, col, max_lines, show_name, show_unit, outs, show_dtype, show_length)
560 else:
561 try:
--> 562 yield format_col_str(idx)
563 except ValueError:
564 raise ValueError(
565 f'Unable to parse format string "{col_format}" for '
566 f'entry "{col[idx]}" in column "{col.info.name}" '
(...) 569 "for possible format specifications."
570 )
File ~/miniconda3/envs/euclid/lib/python3.12/site-packages/astropy/table/pprint.py:522, in TableFormatter._pformat_col_iter.<locals>.format_col_str(idx)
520 def format_col_str(idx):
521 if not multidims:
--> 522 return format_func(col_format, col if is_scalar else col[idx])
524 # Prevents columns like Column(data=[[(1,)],[(2,)]], name='a')
525 # with shape (n,1,...,1) from being printed as if there was
526 # more than one element in a row
527 if multidims_all_ones:
File ~/miniconda3/envs/euclid/lib/python3.12/site-packages/astropy/table/_column_mixins.pyx:89, in astropy.table._column_mixins._MaskedColumnGetitemShim.__getitem__()
87 cdef class _MaskedColumnGetitemShim(_ColumnGetitemShim):
88 def __getitem__(self, item):
---> 89 return base_getitem(self, item, masked_column_getitem)
File ~/miniconda3/envs/euclid/lib/python3.12/site-packages/astropy/table/_column_mixins.pyx:59, in astropy.table._column_mixins.base_getitem()
57 return self.data[item]
58
---> 59 value = getitem(self, item)
60
61 try:
File ~/miniconda3/envs/euclid/lib/python3.12/site-packages/astropy/table/_column_mixins.pyx:83, in astropy.table._column_mixins.masked_column_getitem()
81
82 cdef inline object masked_column_getitem(object self, object item):
---> 83 value = MaskedArray.__getitem__(self, item)
84 return self._copy_attrs_slice(value)
85
File ~/miniconda3/envs/euclid/lib/python3.12/site-packages/numpy/ma/core.py:3288, in MaskedArray.__getitem__(self, indx)
3278 """
3279 x.__getitem__(y) <==> x[y]
3280
3281 Return the item described by i, as a masked array.
3282
3283 """
3284 # We could directly use ndarray.__getitem__ on self.
3285 # But then we would have to modify __array_finalize__ to prevent the
3286 # mask of being reshaped if it hasn't been set up properly yet
3287 # So it's easier to stick to the current version
-> 3288 dout = self.data[indx]
3289 _mask = self._mask
3291 def _is_scalar(m):
File ~/miniconda3/envs/euclid/lib/python3.12/site-packages/astropy/table/column.py:1734, in MaskedColumn.data(self)
1731 @property
1732 def data(self):
1733 """The plain MaskedArray data held by this column."""
-> 1734 out = self.view(np.ma.MaskedArray)
1735 # By default, a MaskedArray view will set the _baseclass to be the
1736 # same as that of our own class, i.e., BaseColumn. Since we want
1737 # to return a plain MaskedArray, we reset the baseclass accordingly.
1738 out._baseclass = np.ndarray
File ~/miniconda3/envs/euclid/lib/python3.12/site-packages/numpy/ma/core.py:3264, in MaskedArray.view(self, dtype, type, fill_value)
3261 type = dtype
3262 dtype = None
-> 3264 output = super().view(*[a for a in (dtype, type) if a is not None])
3266 # Make sure to reset the _fill_value if needed
3267 if getattr(output, '_fill_value', None) is not None:
File ~/miniconda3/envs/euclid/lib/python3.12/site-packages/numpy/ma/core.py:3138, in MaskedArray.__array_finalize__(self, obj)
3136 # Finalize the fill_value
3137 if self._fill_value is not None:
-> 3138 self._fill_value = _check_fill_value(self._fill_value, self.dtype)
3139 elif self.dtype.names is not None:
3140 # Finalize the default fill_value for structured arrays
3141 self._fill_value = _check_fill_value(None, self.dtype)
File ~/miniconda3/envs/euclid/lib/python3.12/site-packages/numpy/ma/core.py:516, in _check_fill_value(fill_value, ndtype)
511 except (OverflowError, ValueError) as e:
512 # Raise TypeError instead of OverflowError or ValueError.
513 # OverflowError is seldom used, and the real problem here is
514 # that the passed fill_value is not compatible with the ndtype.
515 err_msg = "Cannot convert fill_value %s to dtype %s"
--> 516 raise TypeError(err_msg % (fill_value, ndtype)) from e
517 return np.array(fill_value)
TypeError: Cannot convert fill_value N/A to dtype int64
It does work if I access the .data property of the MaskedColumn, which of course points to the underlying Numpy MaskedArray:
In [34]: np.strings.find(col.data, "foo")
Out[34]:
masked_array(data=[ 0, -1, -1],
mask=False,
fill_value=np.str_('N/A'))
which implies that it has something to do with the wrapping of MaskedArray by MaskedColumn. The fact that it goes through the __repr__ is a bit suspicious?
Versions
import astropy
astropy.system_info()
platform
--------
platform.platform() = 'macOS-26.6.1-arm64-arm-64bit'
platform.version() = 'Darwin Kernel Version 25.6.0: Sat Jul 11 15:25:26 PDT 2026; root:xnu-12377.161.13~4/RELEASE_ARM64_T6030'
platform.python_version() = '3.12.3'
packages
--------
astropy 8.0.1
numpy 2.5.1
scipy 1.18.0
matplotlib 3.11.1
pandas --
pyerfa 2.0.1.5
Description
Using
np.strings.findornp.char.findon a tableMaskedColumnthat has a string dtype raises an error. It used to work in astropy 6. It seems to be broken in astropy 7 and 8.Expected behavior
If there is a column of strings, I expect to be able to find the string in the column. This is what worked in past versions of astropy, the latest being version 6.1.7:
And this was with:
This may seem like an obscure workflow, but when using
astroqueryto pull JWST data from MAST, thedataURIcolumn in the returned astropy table is aMaskedColumn, and if one doesn't want to download every file that the API returns, one often wants to filter out some files, i.e. not download the*.jpgfiles or the*_rateints.fitsfiles.How to Reproduce
And the traceback is:
It does work if I access the
.dataproperty of theMaskedColumn, which of course points to the underlying NumpyMaskedArray:which implies that it has something to do with the wrapping of
MaskedArraybyMaskedColumn. The fact that it goes through the__repr__is a bit suspicious?Versions