Tracking issue for migration to Pandas 3.0
Pandas released version 3.0 in Jan 2026 which introduces breaking changes for our codebase. It would be useful to address the places where it currently breaks, so we can remove the upperbound version we introduced in #3186. Even if we eventually drop pandas altogether this is probably a worthwhile effort.
Migration is prepared in separate PRs and will require pre-emptive regression testing before final removal of the upperbound.
Some examples of the required changes are listed here. For a full overview see the Pandas migration guide.
Copy on write (CoW)
This is arguably one of the most important ones as it removes the possibility of mutating parent dataframes via assignments in their subset views (e.g. chained .loc[..] assignment), for more predictable behavior. The worst-case to avoid is that data manipulations in our codebase might silently take no effect on the intended parent DataFrame.
String extension
Pandas 3.0 introduces a new default str datatype that is more efficient than object. Unfortunately this default datatype does not work well with our multi-index HDFs: by default the index/column levels become the new str extension dtype instead of object and PyTables (to_hdf) cannot serialize a MultiIndex whose levels use extension dtypes:
NotImplementedError: Saving a MultiIndex with an extension dtype is not supported.
This would require us to change all dataframes back to object before saving.
PRs
Tracking issue for migration to Pandas 3.0
Pandas released version 3.0 in Jan 2026 which introduces breaking changes for our codebase. It would be useful to address the places where it currently breaks, so we can remove the upperbound version we introduced in #3186. Even if we eventually drop pandas altogether this is probably a worthwhile effort.
Migration is prepared in separate PRs and will require pre-emptive regression testing before final removal of the upperbound.
Some examples of the required changes are listed here. For a full overview see the Pandas migration guide.
Copy on write (CoW)
This is arguably one of the most important ones as it removes the possibility of mutating parent dataframes via assignments in their subset views (e.g. chained
.loc[..]assignment), for more predictable behavior. The worst-case to avoid is that data manipulations in our codebase might silently take no effect on the intended parent DataFrame.String extension
Pandas 3.0 introduces a new default
strdatatype that is more efficient thanobject. Unfortunately this default datatype does not work well with our multi-index HDFs: by default the index/column levels become the newstrextension dtype instead ofobjectand PyTables (to_hdf) cannot serialize a MultiIndex whose levels use extension dtypes:This would require us to change all dataframes back to
objectbefore saving.PRs
keyargument for pd.DataFrame.to_hdf() #3185