= pip test infrastructure planning = 23 January 2020 Participants: Pradyun, Sumana, Ernest <> == Agenda == Today: mostly exploratory/explanatory about what is running now Then: look into what the long-term goals are So: 1. Pipelines 2. Pain points 3. Goals == Pipelines == There are numerous different pipelines Documented already: https://pip.pypa.io/en/latest/development/ci/ Pipelines: the docs at pip.pypa.io and implied in NEWS provide context & info Pradyun: We use Azure, !GitHub, and Travis as our 3 CI providers -- see matrix at https://pip.pypa.io/en/latest/development/ci/ == Pain points == === Wait time === Pain point # 1: reduce time between a pull request submission and GitHub reporting the success: currently: 1 hour the tests on a single job might take 20 minutes these providers only give us so many free job executors so, going through the entire execution cycle is ~1hr Ernest: pretty much every project with a big enough matrix runs into that === Inconsistent test environment === Pain point # 2: we don't have a consistent env of processes that ends up happening across multiple CI providers Sumana: other projects have surely resolved this somehow? Pradyun: Django has their own CI on Jenkins with a pretty big matrix Sumana: is it ridiculous for PSF to run our own Jenkins? Ernest: Linux, Mac, and Windows targeting -- PSF Infra doesn't have any that is on Mac and Windows. So this is why most project end up relying on Travis/Azure/GitHub Actions Similar to the wheel-building block. We don't have the capacity to run the farm necessary * TODO - sumana: find stats request and send to Ernest (now done) Thus, PAIN POINT #2 is outside the scope of this project -- future fundraising opportunity === Automation snags and permissions === Pain point #3: Automation and pip issue tracker - Pradyun has wanted to improve this for a while get access to relevant Heroku instances and what not would need access/credentials from Donald Ernest: PyPA bot and !BrownTruck bot on Heroku the 2 projects have !GitHub users that Donald has pw for * https://github.com/orgs/pypa/people/BrownTruck * https://github.com/orgs/pypa/people/pypa-bot * TODO: Sumana to bug Donald on this * TODO: Sumana: to invite Donald to a future call == Goals == A bunch of Warehouse improvements come to mind, such as: * 2-step release process -- https://github.com/pypa/warehouse/issues/726 * yanking -- https://github.com/pypa/warehouse/pull/5838 (open PR) * Automated test of installability - Warehouse issue #194 https://github.com/pypa/warehouse/issues/194 * (later, for resolver, Warehouse metadata API would help -- https://github.com/pypa/warehouse/issues/474 I believe ) Do any of these Warehouse improvements seem to be on the horizon? Ernest: Yanking is closest to implementable. There's a PEP & a draft PR from Donald. 2-step is next closest. Automated test of installability -- much further away (complexity) How do we hit point #1? Pradyun's idea: Speeding up the test suite https://github.com/pypa/pip/issues/4497 (Pradyun is considering putting this up as a GSoC project -- test suite stuff -- working with PSF GSoC Admins) Sumana: Could we ask Azure and Travis for more slots? Ernest: Azure more likely. Travis: hard to get in contact with, especially since acquisition Ernest: we can ASK. What the current parallel jobs count is for Azure pipeline: PyPA has a single pool for a bunch of projects. * TODO: Sumana to seek out Travis contact to find way to increase or at least maintain pool (They're a sponsor via in-kind donation) == Conclusions == CONCLUSIONS * TODO: Sumana to share in community channels that speeding up test suite is our current approach re: these problems, and Pradyun (with colleagues) to work on that * TODO: group to reassess in ~6 weeks, see if that's still an approach we want to pursue * TODO: Sumana to bug Donald on Heroku/bot permissions * TODO: Sumana: to invite Donald to a future call * TODO: Sumana to seek out Travis contact to find way to increase or at least maintain pool * TODO: Sumana to note these meeting notes in https://github.com/pypa/warehouse/issues/5837 re yanking prioritization