Skip to content

Fix problems in TimeSeries primary key handling - #20361

Merged
taldcroft merged 4 commits into
astropy:mainfrom
taldcroft:fix-timeseries-primary-key
Sep 16, 2026
Merged

taldcroft merged 4 commits into
astropy:mainfrom
taldcroft:fix-timeseries-primary-key

Conversation

@taldcroft

@taldcroft taldcroft commented Sep 10, 2026

Copy link
Copy Markdown
Member

Summary

aggregate_downsample(), .iloc and .loc fail with an unhelpful TypeError on a TimeSeries that was sliced by column names, or built from the time column of another TimeSeries. Selecting a few columns before downsampling is a natural thing to do with a wide light curve table, and the failure is far from the code that caused it.

>>> ts = TimeSeries(time=Time(np.arange(2450000, 2450005), format="jd"), data=[[1, 2, 3, 4, 5]], names=["a"])
>>> aggregate_downsample(ts["time", "a"], n_bins=2)
TypeError: list indices must be integers or slices, not NoneType   # main
<BinnedTimeSeries length=2>                                        # this branch
>>> TimeSeries(time=ts.time).iloc[:]
TypeError: list indices must be integers or slices, not NoneType   # main (#11704)
<TimeSeries length=5>                                              # this branch

The cause is that a time column copied from an indexed table carries its index with it. TimeSeries.__init__ and TimeSeries.add_column only call add_index("time") when the table has no indices at all, so with the carried-in index present the call is skipped and primary_key stays None, which is what TableLoc then trips over. This PR makes the add_column check look for an index on time rather than for no indices, and always sets time as the primary key. BinnedTimeSeries had the same pattern and additionally called add_index("time_bin_start") unconditionally, so a column-sliced BinnedTimeSeries ended up with two identical indices; that is fixed the same way.

Fixes #20297. Fixes #11704. Supersedes #20354.

Close #20299

AI disclosure

This PR includes AI-generated content using Claude Fable 5.1. I have carefully reviewed the content and fully understand the changes in astropy/timeseries/core.py, sampled.py, binned.py and the tests.

  • I certify that I am human and take responsibility for the code and interactions with reviewers.

Details

Click to expand

Always give a TimeSeries a primary index on time, whatever indices its input columns carry

1. Why the primary key ends up None

Table.primary_key is a plain attribute that only Table.add_index() sets, and only when the table has no indices yet. Column copies made by col_copy(..., copy_indices=True) deep-copy the column's SlicedIndex objects, so a table built from such columns has indices but no primary key. Three paths in astropy.timeseries hit this:

  • ts["time", "a"] goes through Table.__getitem__, which builds a new TimeSeries from the copied columns. In TimeSeries.__init__ the time column is then removed and re-added, but it is the same column object, still carrying its copied index. add_column sees len(self.indices) == 1, skips add_index("time"), and primary_key is never set.
  • TimeSeries(time=ts.time) copies the indexed Time column in add_column, with the same outcome. This is TimeSeries.iloc failed when time comes from another TimeSeries #11704.
  • BinnedTimeSeries.__init__ has the same remove/re-add pattern for time_bin_start and then calls add_index("time_bin_start") unconditionally, so the sliced series has two identical indices, neither of them primary.

TableLoc._get_index_id_and_item then returns self.table.primary_key, i.e. None, and TableIndices.__getitem__(None) falls through to list.__getitem__, giving the TypeError in the report.

A related symptom on main: ts.fold() on a series with a second index (say ts.add_index("a")) removes and re-adds time, and the len(self.indices) == 0 guard then leaves the folded series with a primary key of ("time",) but no index on time.

2. Implementation

astropy/timeseries/core.py, new BaseTimeSeries._add_primary_index(colname):

if (colname,) not in [index.id for index in self.indices]:
    self.add_index(colname)
self.primary_key = (colname,)

It adds an index on the column only if there is none, and sets the primary key explicitly because add_index does that only for the first index of a table.

astropy/timeseries/sampled.py, add_column / add_columns: the len(self.indices) == 0 and "time" in self.colnames guard becomes if "time" in self.colnames: self._add_primary_index("time"). The constructor's remove/re-add of time goes through add_column, so this single change covers the column-slice case, #11704, and fold().

astropy/timeseries/binned.py, BinnedTimeSeries.__init__: add_index("time_bin_start") becomes _add_primary_index("time_bin_start"). That removes the duplicate index for both the column-slice case and BinnedTimeSeries(time_bin_start=other.time_bin_start, ...).

Nothing was removed from the public API. The approach in #20354 (an elif self.primary_key is None: self.primary_key = ("time",) in add_column) is not used because it assigns a key without checking that an index on time exists; with qt.add_index("a"); TimeSeries([qt["time"], qt["a"]]) it produced IndexError: No index found for ['time'] instead.

3. Backwards compatibility

No public API changes. A TimeSeries whose time column has no carried-in index behaves exactly as before: add_index("time") is called once and becomes the primary key. What changes is only the previously broken states: column-sliced or column-constructed series now have a usable primary time index, BinnedTimeSeries no longer accumulates duplicate time_bin_start indices, and fold() on a series with a second index keeps its time index. A series that carries a second index (e.g. via ts.add_index("a")) always has time as the primary key after any add_column; before, the primary key was whichever index was added first, which for a TimeSeries was already time.

Changelog fragment: docs/changes/timeseries/20361.bugfix.rst.

🤖 Generated with Claude Code

@github-actions

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Astropy! 🌌 This checklist is meant to remind the package maintainers who will review this pull request of some common things to look for.

  • Do the proposed changes actually accomplish desired goals?
  • Do the proposed changes follow the Astropy coding guidelines?
  • Are tests added/updated as required? If so, do they follow the Astropy testing guidelines?
  • Are docs added/updated as required? If so, do they follow the Astropy documentation guidelines?
  • Is rebase and/or squash necessary? If so, please provide the author with appropriate instructions. Also see instructions for rebase and squash.
  • Did the CI pass? If no, are the failures related? If you need to run daily and weekly cron jobs as part of the PR, please apply the "Extra CI" label. Codestyle issues can be fixed by the bot.
  • Is a change log needed? If yes, did the change log check pass? If no, add the "no-changelog-entry-needed" label. If this is a manual backport, use the "skip-changelog-checks" label unless special changelog handling is necessary.
  • Is this a big PR that makes a "What's new?" entry worthwhile and if so, is (1) a "what's new" entry included in this PR and (2) the "whatsnew-needed" label applied?
  • At the time of adding the milestone, if the milestone set requires a backport to release branch(es), apply the appropriate "backport-X.Y.x" label(s) before merge.

data=[[1, 2, 3, 4, 5]],
names=["a"],
)
ts_sub = ts["time", "a"]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For column subset scenario, should the other path, of creating a new TimeSeries object explicitly, be included in the test as well?

ts_sub = TimeSeries(time=ts["time"], data={"a": ts["a"]})

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems sensible to add this, once you've done it can you ping me @taldcroft and I can do a final review of this PR? (it seems fine to me otherwise)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@astrofrog @taldcroft I don't think I can add to the PR. The following is the modified test that includes the case of creating a new TimeSeries object.

def test_downsample_subset_columns():
    # Regression test for #20297: aggregate_downsample on a TimeSeries sliced
    # by column names failed because the slice had an index but no primary key.
    ts = TimeSeries(
        time=Time(np.arange(2450000, 2450005), format="jd"),
        data=[[1, 2, 3, 4, 5]],
        names=["a"],
    )

    def do_test(ts_sub, label):
        assert ts_sub.primary_key == ("time",), label
        binned = aggregate_downsample(ts_sub, n_bins=2)
        assert len(binned) == 2, label
        assert_equal(binned["a"], aggregate_downsample(ts, n_bins=2)["a"], label)

    do_test(ts["time", "a"], "subset by slicing")
    do_test(TimeSeries(time=ts["time"], data={"a": ts["a"]}), "with new TimeSeries obj")

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry my grammar was confusing - @taldcroft once you've added this, ping me and I can do a final review!

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @orionlee. I updated the test like above and tests pass locally.

@astrofrog astrofrog left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good assuming CI passes

Comment thread docs/changes/timeseries/20361.bugfix.rst Outdated
@pllim

pllim commented Sep 16, 2026

Copy link
Copy Markdown
Member

Oh hey, I cannot override protection to merge anymore.

@pllim

pllim commented Sep 16, 2026

Copy link
Copy Markdown
Member

Well, that is annoying. @taldcroft , can you please rebase and squash the commits and remove my skip CI directive? Sorry for the trouble.

@astrofrog , I need to be able to bypass the CI checks manually. Sometimes we have trivial change that skip CI.

@astrofrog

Copy link
Copy Markdown
Member

Can we allowlist you in the rules?

@taldcroft
taldcroft force-pushed the fix-timeseries-primary-key branch from 2180060 to 24804ea Compare September 16, 2026 15:55
@taldcroft
taldcroft force-pushed the fix-timeseries-primary-key branch from 24804ea to af291e0 Compare September 16, 2026 16:00
@taldcroft
taldcroft force-pushed the fix-timeseries-primary-key branch from af291e0 to ea4a1f7 Compare September 16, 2026 16:04
@taldcroft
taldcroft enabled auto-merge (squash) September 16, 2026 16:05
@taldcroft

Copy link
Copy Markdown
Member Author

OK, tests are running now. 🤞

@taldcroft
taldcroft merged commit 11a4f3f into astropy:main Sep 16, 2026
35 of 38 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

4 participants