Fix problems in TimeSeries primary key handling - #20361
Conversation
|
Thank you for your contribution to Astropy! 🌌 This checklist is meant to remind the package maintainers who will review this pull request of some common things to look for.
|
| data=[[1, 2, 3, 4, 5]], | ||
| names=["a"], | ||
| ) | ||
| ts_sub = ts["time", "a"] |
There was a problem hiding this comment.
For column subset scenario, should the other path, of creating a new TimeSeries object explicitly, be included in the test as well?
ts_sub = TimeSeries(time=ts["time"], data={"a": ts["a"]})There was a problem hiding this comment.
It seems sensible to add this, once you've done it can you ping me @taldcroft and I can do a final review of this PR? (it seems fine to me otherwise)
There was a problem hiding this comment.
@astrofrog @taldcroft I don't think I can add to the PR. The following is the modified test that includes the case of creating a new TimeSeries object.
def test_downsample_subset_columns():
# Regression test for #20297: aggregate_downsample on a TimeSeries sliced
# by column names failed because the slice had an index but no primary key.
ts = TimeSeries(
time=Time(np.arange(2450000, 2450005), format="jd"),
data=[[1, 2, 3, 4, 5]],
names=["a"],
)
def do_test(ts_sub, label):
assert ts_sub.primary_key == ("time",), label
binned = aggregate_downsample(ts_sub, n_bins=2)
assert len(binned) == 2, label
assert_equal(binned["a"], aggregate_downsample(ts, n_bins=2)["a"], label)
do_test(ts["time", "a"], "subset by slicing")
do_test(TimeSeries(time=ts["time"], data={"a": ts["a"]}), "with new TimeSeries obj")There was a problem hiding this comment.
Sorry my grammar was confusing - @taldcroft once you've added this, ping me and I can do a final review!
There was a problem hiding this comment.
Thanks @orionlee. I updated the test like above and tests pass locally.
astrofrog
left a comment
There was a problem hiding this comment.
Looks good assuming CI passes
|
Oh hey, I cannot override protection to merge anymore. |
|
Well, that is annoying. @taldcroft , can you please rebase and squash the commits and remove my skip CI directive? Sorry for the trouble. @astrofrog , I need to be able to bypass the CI checks manually. Sometimes we have trivial change that skip CI. |
|
Can we allowlist you in the rules? |
2180060 to
24804ea
Compare
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
24804ea to
af291e0
Compare
af291e0 to
ea4a1f7
Compare
|
OK, tests are running now. 🤞 |
Summary
aggregate_downsample(),.ilocand.locfail with an unhelpfulTypeErroron aTimeSeriesthat was sliced by column names, or built from thetimecolumn of anotherTimeSeries. Selecting a few columns before downsampling is a natural thing to do with a wide light curve table, and the failure is far from the code that caused it.The cause is that a
timecolumn copied from an indexed table carries its index with it.TimeSeries.__init__andTimeSeries.add_columnonly calladd_index("time")when the table has no indices at all, so with the carried-in index present the call is skipped andprimary_keystaysNone, which is whatTableLocthen trips over. This PR makes theadd_columncheck look for an index ontimerather than for no indices, and always setstimeas the primary key.BinnedTimeSerieshad the same pattern and additionally calledadd_index("time_bin_start")unconditionally, so a column-slicedBinnedTimeSeriesended up with two identical indices; that is fixed the same way.Fixes #20297. Fixes #11704. Supersedes #20354.
Close #20299
AI disclosure
This PR includes AI-generated content using Claude Fable 5.1. I have carefully reviewed the content and fully understand the changes in
astropy/timeseries/core.py,sampled.py,binned.pyand the tests.Details
Click to expand
Always give a
TimeSeriesa primary index ontime, whatever indices its input columns carry1. Why the primary key ends up
NoneTable.primary_keyis a plain attribute that onlyTable.add_index()sets, and only when the table has no indices yet. Column copies made bycol_copy(..., copy_indices=True)deep-copy the column'sSlicedIndexobjects, so a table built from such columns has indices but no primary key. Three paths inastropy.timeserieshit this:ts["time", "a"]goes throughTable.__getitem__, which builds a newTimeSeriesfrom the copied columns. InTimeSeries.__init__thetimecolumn is then removed and re-added, but it is the same column object, still carrying its copied index.add_columnseeslen(self.indices) == 1, skipsadd_index("time"), andprimary_keyis never set.TimeSeries(time=ts.time)copies the indexedTimecolumn inadd_column, with the same outcome. This is TimeSeries.iloc failed when time comes from another TimeSeries #11704.BinnedTimeSeries.__init__has the same remove/re-add pattern fortime_bin_startand then callsadd_index("time_bin_start")unconditionally, so the sliced series has two identical indices, neither of them primary.TableLoc._get_index_id_and_itemthen returnsself.table.primary_key, i.e.None, andTableIndices.__getitem__(None)falls through tolist.__getitem__, giving theTypeErrorin the report.A related symptom on
main:ts.fold()on a series with a second index (sayts.add_index("a")) removes and re-addstime, and thelen(self.indices) == 0guard then leaves the folded series with a primary key of("time",)but no index ontime.2. Implementation
astropy/timeseries/core.py, newBaseTimeSeries._add_primary_index(colname):It adds an index on the column only if there is none, and sets the primary key explicitly because
add_indexdoes that only for the first index of a table.astropy/timeseries/sampled.py,add_column/add_columns: thelen(self.indices) == 0 and "time" in self.colnamesguard becomesif "time" in self.colnames: self._add_primary_index("time"). The constructor's remove/re-add oftimegoes throughadd_column, so this single change covers the column-slice case, #11704, andfold().astropy/timeseries/binned.py,BinnedTimeSeries.__init__:add_index("time_bin_start")becomes_add_primary_index("time_bin_start"). That removes the duplicate index for both the column-slice case andBinnedTimeSeries(time_bin_start=other.time_bin_start, ...).Nothing was removed from the public API. The approach in #20354 (an
elif self.primary_key is None: self.primary_key = ("time",)inadd_column) is not used because it assigns a key without checking that an index ontimeexists; withqt.add_index("a"); TimeSeries([qt["time"], qt["a"]])it producedIndexError: No index found for ['time']instead.3. Backwards compatibility
No public API changes. A
TimeSerieswhosetimecolumn has no carried-in index behaves exactly as before:add_index("time")is called once and becomes the primary key. What changes is only the previously broken states: column-sliced or column-constructed series now have a usable primary time index,BinnedTimeSeriesno longer accumulates duplicatetime_bin_startindices, andfold()on a series with a second index keeps its time index. A series that carries a second index (e.g. viats.add_index("a")) always hastimeas the primary key after anyadd_column; before, the primary key was whichever index was added first, which for aTimeSerieswas alreadytime.Changelog fragment:
docs/changes/timeseries/20361.bugfix.rst.🤖 Generated with Claude Code