remove dependency upper bounds for numpy and matplotlib - #3499

@deruyter92 Some additional findings

numpy lower bound and pickle

Once the <2 cap is gone, np-1 and np-2 might see some exchange of project files.
np2 can load pickle files created with np 1.26, and vice versa. For 1.25 and earlier loading np2 pickles will throw an exception.
See NumPy 2.0 migration guide.

We may want numpy<2 only for tf-cu11

From PyPI metadata for each pinned TF version:

CI changes

Every full-suite job installs the [tf] extra so after merge, CI resolves numpy 1 everywhere. We may have to add more jobs.
The downloaded fixtures (tests/data/{dets,montblanc_*,trimouse_*}.pickle, montblanc_tracks.h5) were written under numpy 1 and are readable by numpy 2, so they work.

Removing both caps together is required

matplotlib 3.8.4 declares only numpy>=1.21, with no upper bound. So dropping numpy<2 while leaving matplotlib<3.9 would try to resolve matplotlib 3.8.4 + np2. Same risk for any compiled transitive dep that declares no numpy cap (pycocotools, tables, h5py, scikit-image), so an import smoke test under numpy 2 may be worth it.

numba

numba 0.54 requires numpy<1.21, which no DLC-supported numpy version satisfies. Per numba's support matrix, 0.60.0 is the first release supporting numpy 2.0 and the latest (0.67) supports 2.0 <= numpy < 2.6.
Should still play nice with TF opt deps I think.

TF specific (low priority)

np.lib.pad is still used at visualize.py:93 and vis_dataset.py:70. Unlikely to be a problem as we're not migrating TF to np2.

We may want numpy<2 only for tf-cu11

@C-Achard agreed, just arrived at the same conclusion

Every full-suite job installs the [tf] extra so after merge, CI resolves numpy 1 everywhere. We may have to add more jobs.

Instead, we should just remove the upperbound from tf right?

Edit: TF defines an upper bound themselves, which means that installing the tf extra naturally resolves to numpy 1.x for tensorflow versions up to 2.17

Let's keep an eye on the canary lane CI. Errors should surface here (softly).

Fixed numba + errors surfaced by the canary lane in #3508

We should really decide whether this gets included now or later; I would include #3508 in 3.0.2 anyway but I think either:

• We ship 3.0.2 ASAP with no pins release
• Or we spend a bit more time on it while [Sync] merge to main for 3.0.2 release - bug fixes; knowledge indexing; future compatibility & dependencies  #3495 is being reviewed, and make sure everything is viable

but I don't think merging this as is right now is advisable.

Missing:

• napari release with no numpy pin
• Follow-up fixes for numpy2 compatibility #3508
• Tighten strict handling for canary CI #3509 / Raise lower bounds to NumPy 2 capable releases #3510

Note: fmpose3d also still pins numpy. See AdaptiveMotorControlLab/FMPose3D#36

I don't think merging this as is right now is advisable
Of course, agreed. We won't merge this until CI is completely green. Assuming that our test-coverage is enough, I would say let's merge after our tests indicate we are ready. Any remaining compat issues from untested code can be easily addressed when they surface.

I think either:

• We ship 3.0.2 ASAP with no pins release
• Or we spend a bit more time on it while [Sync] merge to main for 3.0.2 release - bug fixes; knowledge indexing; future compatibility & dependencies  #3495 is being reviewed

Agreed. I would say we ship 3.0.2 ASAP in any case. If these PRs are fully resolved before the review + release is finished, we can include it. Otherwise we will ship in the next release.

Review comments below. @C-Achard agreed? Shall I address these myself?