Revision 8 as of 2009-04-22 05:06:03

Clear message

Introduction

One of the big challenges for Python going forward is providing a testing infrastructure for Python Packages.

There are now over 6,000 packages listed on PyPi - and this number can only get bigger.

Then, there are the three major operating systems:

To complicate the problem, there are now many versions of each operating system.

Multiply those two combinations by all of the versions of Python that already exist (not to mention the ones coming) and we start to that we are heading into complexity. If there are 4 major windows revisions and 4 major Linuxes, and two major Mac platforms, we end up with perphaps (6,000 x (4 + 4 + 2)) 60,000 delivery possibilities.

That number then needs to be multiplied by the number of python versions, which possibly include 2.1, 2.2, 2.3, 2.4, 2.5, 2.6 and coming up.. the 3 series...

So that could be (60,000 x 7 (python versions)) 420,000 variations of known python packages.

To date... the testing has been done... we have to assume... manually with some automation.

But we can't expect package maintainers to be forever testing their own code on platforms that they simply don't have access to.

A more reasonable and cost effective option is to have this testing done on a server farm virtual environment building infrastructure.

In simple terms, we need to build all the packages that exist for Python on a daily basis on all of the environments and report any issues back to the registered maintainers.

Fortunately, Google and Amazon web services to name just two, offer commercial virtual server farms that could be employed to do the above build process of all the python packages.

Python scripts could then be developed utilising the "test" frameworks to supervise the build on each and every platform.

With this basic structure, a daily building/testing infrastructure working across the different versions of python and operating systems, could easily become a reality.

Server Farm Virtual Environments

My experience with cloud computing environments is limited to Amazon Web Services. I understand that Google and others may offer a comparable service.

At present they offer virtual environments for both Windows and Linux. These can be seen on these links:

One challenge will be to find an equivilent service that offers Mac Virtual Machines in a server farm.

Test Scripts

A test script will be developed that will cycle through all the packages on pypi, download the package and build it on all available platforms.

The results of the build can then be made available via some sort of web delivery system. Describing on which platforms the builds were successful and not.

In the past, it has been difficult for developers to test on all platforms.

These facilities are bound to improve overal code quality across the python universe.

Scope of Testing

It's important to define what and can be and what cannot be tested.

The scope of the framework will be:

Unable to edit the page? See the FrontPage for instructions.