Revision 2 as of 2020-01-27 17:19:58

Clear message

pip test infrastructure planning

23 January 2020

Participants: Pradyun, Sumana, Ernest


Today: mostly exploratory/explanatory about what is running now

Then: look into what the long-term goals are


  1. Pipelines
  2. Pain points
  3. Goals


There are numerous different pipelines

Documented already:


Pradyun: We use Azure, GitHub, and Travis as our 3 CI providers -- see matrix at

Pain points

Wait time

Pain point # 1: reduce time between a pull request submission and GitHub reporting the success:

Ernest: pretty much every project with a big enough matrix runs into that

Inconsistent test environment

Pain point # 2:

Sumana: other projects have surely resolved this somehow?

Pradyun: Django has their own CI on Jenkins with a pretty big matrix

Sumana: is it ridiculous for PSF to run our own Jenkins?

Ernest: Linux, Mac, and Windows targeting -- PSF Infra doesn't have any that is on Mac and Windows.

So this is why most project end up relying on Travis/Azure/GitHub Actions

Similar to the wheel-building block.

We don't have the capacity to run the farm necessary

Thus, PAIN POINT #2 is outside the scope of this project -- future fundraising opportunity

Automation snags and permissions

Pain point #3:

Ernest: PyPA bot and BrownTruck bot on Heroku

the 2 projects have GitHub users that Donald has pw for


A bunch of Warehouse improvements come to mind, such as:

Do any of these Warehouse improvements seem to be on the horizon?

Ernest: Yanking is closest to implementable. There's a PEP & a draft PR from Donald.

2-step is next closest.

Automated test of installability -- much further away (complexity)

How do we hit point #1?

Pradyun's idea: Speeding up the test suite

(Pradyun is considering putting this up as a GSoC project -- test suite stuff -- working with PSF GSoC Admins)

Sumana: Could we ask Azure and Travis for more slots?

Ernest: Azure more likely. Travis: hard to get in contact with, especially since acquisition

Ernest: we can ASK.

What the current parallel jobs count is for Azure pipeline:

PyPA has a single pool for a bunch of projects.

(They're a sponsor via in-kind donation)



Unable to view page? See the FrontPage for instructions.