NEP 57: NumPy platform support#30748
Conversation
[skip ci]
|
FWIW there is some pressure to provide windows ARM wheels for statsmodels. So far we haven't although recent pandas may make it more likely. The big issues for us is that no one has a local windows ARM computer so it is really tedious to try and figure out any windows ARM specific issues using only CI. |
If anyone has an ARM macbook, you can run ARM Windows in a VM pretty easily. Sadly Microsoft hasn't yet published free images for this purpose so you need a Windows license or an unlicensed install. |
|
@bashtage a few thoughts:
|
|
Thank you for providing leadership on this @rgommers. It would be good to point to something when we get these questions. I think downstream packages will also use this NEP. |
|
Should we also include dropping support in this NEP? Current examples would be PyPy, Mac on X86_64, and 32 bit Windows. |
There are several comments on decision making. I'm not sure if we can make it much more precise than:
Everything else, and especially for "dropping" is going to be quite case-specific I'd think. Am I missing something? Happy to be convinced to add more detail, I'm just not sure what to write that can be useful and generally applicable. |
|
Maybe just changing this:
to this:
It amounts to the same thing but more clearly states that dropping support is a thing and that there isn't a hard-and-fast rule. |
I included that now, thanks. |
|
I wonder why docs CI is not running here, are we excluding NEPs from any CI? NEP0 states
so I think we should merge this whenever the author feels it is can be taken out of PR-draft (as opposed to NEP-draft) status |
Just because I added
I'll do one more round to get it there. |
|
One question I had was names of one or more maintainers to put behind the Tier 2 platforms, now that it seems everyone is fine with that principle. I also didn't want to make assumptions without asking, so let me do that here:
|
|
Oh sorry, should have asked for Tier 3 as well
|
|
Out of the Tier 1 platforms, only Windows arm64 seems a bit questionable - it might be early; |
| being shipped, for NumPy or for a comparable project, then download data | ||
| from PyPI may be obtained through BigQuery. For new platforms, sources | ||
| like the* | ||
| `Steam Hardware & Software Survey <https://store.steampowered.com/hwsurvey/?platform=combined>`__ |
There was a problem hiding this comment.
This website reports only PC counts and does not include the server market, which makes it an unreliable way to gauge this.
There was a problem hiding this comment.
"sources like" indicates that. If data for server platforms is needed for a specific proposal, then more suitable data sources are very welcome. We've just never done that exercise before, so I have nothing to point to - suggestions are very welcome.
I'd prefer not to be added as a maintainer. My expertise here is limited to the build infrastructure, which doesn't cover if a build doesn't work due to e.g. compiler failures, or if there are test fails. |
|
No worries at all, thanks for considering it @andyfaff |
|
Sure, I'm happy to be the contact person for free-threaded support. |
Sure, I'm happy to be the contact person for ppc64le. |
I'm happy to be a point of contact for Pyodide. Thanks for asking, @rgommers! I'd also ask @hoodmane and @ryanking13 to weigh in, since they can help reduce the bus factor – they also did a lot of the work back when pyodide-build did not support Meson/meson-python, when I was not yet involved. |
There was a problem hiding this comment.
I like this proposal! thanks for writing it, Ralf! I left a few comments inline; some on language clarifications, and a few other questions. Besides these, I had a few other (hopefully small) suggestions:
- non-blocking: perhaps we should have a sentence at the top to describe what "platform" means here? This is understandable to everyone involved here, obviously, but perhaps it would be useful to note that "platform" for the purposes of this document is not an OS/architecture combination in the traditional sense, but also encompasses interpreter variants, such as free-threaded CPython.
- non-blocking: there's no reference to PyPI storage limits at the moment. I am aware that this is a NEP for platform support, but since you mentioned the Stable ABI, I think it warrants a brief acknowledgement, since there is a practical limit that will keep getting approached with time.
- is there (or could there be) a grace period and/or call for new maintainers before demotung a platform? I assume it will all be discussed on the mailing list, but it doesn't describe what happens if/after a maintainer for a platform becomes unavailable.
since all wheels are released via the numpy-release repo, would it make sense for points of contact to be given triage/non-write permissions to it or be added as CODEOWNERS so that they can receive relevant pings? We do this for cibuildwheel and I've felt that it helps to have someone looking at what is being changed for a platform.edit: turns out this is not possible right now, and is a long-standing feature request for GitHub (of course it is 🤷🏻): https://github.com/orgs/community/discussions/23042, https://github.com/orgs/community/discussions/29982
P.S. while going through many of the historical discussions while reviewing this, I came across #20089 (comment), it's finally nice that this NEP will be landing soon :D
| - Linux x86-64 (manylinux) | ||
| - Linux aarch64 (manylinux) |
There was a problem hiding this comment.
Could you please explain (mostly for my curiosity) why musl wheels are Tier 2 rather than Tier 1? They are built on GitHub CI, released on PyPI, and from https://github.com/numpy/numpy/issues?q=%22musl%22, I think there have been fewer issues than usual post 2022/2023 after uploading PyPI wheels was finalised (though I don't know how much effort it takes for all maintainers to keep them working).
There was a problem hiding this comment.
They're much more niche than manylinux, and we had a discussion about it that could have turned out differently. They're just not a must-have-so-we're-all-responsible.
Sure, we care a lot to make all the numpy features work in Pyodide, and happy to be a point of contact. |
[skip actions] [skipp cirrus] [skip azp]
Thanks for the feedback everyone. Updated and taken out of draft now. I resolved all the straightforward and accepted comments, left the ones I replied to open. I think this is okay to merge as is now, probably after giving the reviewers with open comments at least a day or two to either resolve them or reply again. |
agriyakhetarpal
left a comment
There was a problem hiding this comment.
Thanks, Ralf! Looking at your latest commit 6f49427 and the other threads above where you responded, my comments have been resolved. The PyPI storage limit suggestion I had will get encompassed by this sentence, since the discussion about storage limits will ultimately happen there:
Moving a platform to a higher support tier, if that higher tier includes releasing wheels on PyPI for that platform, must be discussed on the mailing list.
And I noticed that my comment about what "platform" is covered by the abstract, which I think I ended up missing because there was a comment in the GitHub UI over it.
Looking forward to this NEP; thanks for adding Gyeongjae and me to the list!
[skip actions] [skip cirrus] [skip azp]
|
Added Kumar, and changed the bullet lists of platforms to reST tables with more details and links out to tracking issues. I'm happy with how this looks now. |
| | Linux ppc64le | Runs on IBM-provided self-hosted | Sandeep Gupta | | ||
| | | runners, see gh-22318_ | | | ||
| +--------------------+----------------------------------------+----------------------------------+ | ||
| | Emscripten/Pyodide | We provide nightly wheels, used for | Agriya Khetarpal, Gyeongjae Choi | |
There was a problem hiding this comment.
Let's not commit to permanent support?
| | Emscripten/Pyodide | We provide nightly wheels, used for | Agriya Khetarpal, Gyeongjae Choi | | |
| | Emscripten/Pyodide | We currently provide nightly wheels, used for | Agriya Khetarpal, Gyeongjae Choi | |
There was a problem hiding this comment.
Accepted the comment; I didn't read it as permanent support either way though. Nothing except wheels on PyPI carries long-term guarantees.
| +------------------------------------+--------------------------------+ | ||
| | Platform | Notes | | ||
| +====================================+================================+ | ||
| | PyPy | See gh-30416_ | |
There was a problem hiding this comment.
Maybe remove PyPy since things are in a state of flux
There was a problem hiding this comment.
I'd like to keep it since it was a supported platform until very recently. I edited the note for clarity: "Was Tier 2 until the 2.4.x releases, see gh-30416"
[skip actions] [skip cirrus] [skip azp]
It is still in 2.4.x, we dropped it along with Python 3.11 in 2.5. |
Yeah I know, with "until" I meant |
|
Ah, Fortran indexing :) Until has the implication "when it changed", <= would be clearer. |
[skip actions] [skip cirrus] [skip azp]
|
Thanks @rgommers and all the reviewers. Let’s put this in |
This is a rough draft, with the purpose of agreeing on the principles first on the mailing list before polishing the text. Please keep the big picture discussion on the mailing list. Edits for correctness/completeness are of course welcome on this PR already.
Relevant issues and PRs for one specific platform: