pip test infrastructure planning
23 January 2020
Participants: Pradyun, Sumana, Ernest
Contents
Agenda
Today: mostly exploratory/explanatory about what is running now
Then: look into what the long-term goals are
So:
- Pipelines
- Pain points
- Goals
Pipelines
There are numerous different pipelines
Documented already: https://pip.pypa.io/en/latest/development/ci/
Pipelines:
the docs at pip.pypa.io and implied in NEWS provide context & info
Pradyun: We use Azure, GitHub, and Travis as our 3 CI providers -- see matrix at https://pip.pypa.io/en/latest/development/ci/
Pain points
Wait time
Pain point # 1: reduce time between a pull request submission and GitHub reporting the success:
- currently: 1 hour the tests on a single job might take 20 minutes these providers only give us so many free job executors so, going through the entire execution cycle is ~1hr
Ernest: pretty much every project with a big enough matrix runs into that
Inconsistent test environment
Pain point # 2:
- we don't have a consistent env of processes that ends up happening across multiple CI providers
Sumana: other projects have surely resolved this somehow?
Pradyun: Django has their own CI on Jenkins with a pretty big matrix
Sumana: is it ridiculous for PSF to run our own Jenkins?
Ernest: Linux, Mac, and Windows targeting -- PSF Infra doesn't have any that is on Mac and Windows.
So this is why most project end up relying on Travis/Azure/GitHub Actions
Similar to the wheel-building block.
We don't have the capacity to run the farm necessary
- TODO - sumana: find stats request and send to Ernest (now done)
Thus, PAIN POINT #2 is outside the scope of this project -- future fundraising opportunity
Automation snags and permissions
Pain point #3:
- Automation and pip issue tracker - Pradyun has wanted to improve this for a while get access to relevant Heroku instances and what not would need access/credentials from Donald
Ernest: PyPA bot and BrownTruck bot on Heroku
the 2 projects have GitHub users that Donald has pw for
- TODO: Sumana to bug Donald on this
- TODO: Sumana: to invite Donald to a future call
Goals
A bunch of Warehouse improvements come to mind, such as:
2-step release process -- https://github.com/pypa/warehouse/issues/726
yanking -- https://github.com/pypa/warehouse/pull/5838 (open PR)
Automated test of installability - Warehouse issue #194 https://github.com/pypa/warehouse/issues/194
(later, for resolver, Warehouse metadata API would help -- https://github.com/pypa/warehouse/issues/474 I believe )
Do any of these Warehouse improvements seem to be on the horizon?
Ernest: Yanking is closest to implementable. There's a PEP & a draft PR from Donald.
2-step is next closest.
Automated test of installability -- much further away (complexity)
How do we hit point #1?
Pradyun's idea: Speeding up the test suite https://github.com/pypa/pip/issues/4497
(Pradyun is considering putting this up as a GSoC project -- test suite stuff -- working with PSF GSoC Admins)
Sumana: Could we ask Azure and Travis for more slots?
Ernest: Azure more likely. Travis: hard to get in contact with, especially since acquisition
Ernest: we can ASK.
What the current parallel jobs count is for Azure pipeline:
PyPA has a single pool for a bunch of projects.
- TODO: Sumana to seek out Travis contact to find way to increase or at least maintain pool
(They're a sponsor via in-kind donation)
Conclusions
CONCLUSIONS
- TODO: Sumana to share in community channels that speeding up test suite is our current approach re: these problems, and Pradyun (with colleagues) to work on that
- TODO: group to reassess in ~6 weeks, see if that's still an approach we want to pursue
- TODO: Sumana to bug Donald on Heroku/bot permissions
- TODO: Sumana: to invite Donald to a future call
- TODO: Sumana to seek out Travis contact to find way to increase or at least maintain pool
TODO: Sumana to note these meeting notes in https://github.com/pypa/warehouse/issues/5837 re yanking prioritization