Revision 14 as of 2011-05-25 17:09:09

Clear message

Pypi Testing Infrastructure

Create a server that tests uploaded packages on PyPI.

Gsoc 2011

General designs

Terminology

The entire project has been divided into two parts - Environment part and Execution part . Each part has been detailed below.

Environment part

Architecture

Raw Data API

Command API

Slave

The slave performs the following tasks

Master

Execution part

Detailed work

During the preparation's discussions of the project, we have cut the project into two different parts which are depending on each other. This proposal is one of them: the execution part. The other part the establishment of clean environment (inside a VM instance): Environment part.

Work can be divided into three parts, which I will detail below:

Create an execution manager. Write the essentials tasks. Write the common tasks.

Global Architecture

Even if this proposal concern the execution part of PYTI, choices made during preparation's discussions have an influence on the work, i will detail theses choices below and their influences:

Tests will be executed inside a Virtual Machine instance, so we need to execute source codes from untrusted source. We chose to cut off access to the network in order to avoid sending mail, deny of service... This choice can be problematic if tests need access to the network, but good practices recommend to mock them. If several distributions cannot mock network access, we must create a method to control network access in a more accurately way.

As vm instance will not have access to network, the program which will start the instance must prepare all that the instance will need, distribution archive, of course, but also distribution's dependencies archives. The dependencies computing must be done before starting the vm instance, this part is detailed below in the "Dependency setup" part.

The last point is about Raw Data API, in fact, tasks will generate raw data (see below for terminology) and they should be sent to the slave. This API is part of the complementary part of the project, but as the execution part will use it, it should be designed with the participation of all.

Execution manager

Running the tests on package content may be split into different independent tasks. These tasks can be written independently, but they must all be executed anyway. They cannot be performed in any order, as one specific task may depends on an another one and if a task fail, the system should not performed other tasks that depend on that which has failed.

This is the role of the execution manager, it will manage all the different tasks and execute them in the right order, ensuring that tasks success. Execution manager must manage these scenario:

Task 'A' depends on 'B' and 'B' one depends on 'C'. Execution order must be: 'C', 'B' and finally 'A'. If task 'B' fail, execution manager must mark 'A' as skipped due to error in one of it's dependencies. Execution manager must also provide a way for tasks to exchange data, and so we can imagine that tasks will depends on data instead of traditional dependencies. Moreover, as tasks should generate raw data, it makes more sense to manage raw data dependencies than tasks dependencies. For example:

Task 'A' need 'D' data. Task 'B' produce 'D' data during his execution. Execution order must be: 'B' and 'A'.

Essential tasks

Some of the tasks are essential and must be implemented as soon as possible.

The second one is the installation. It's one of the first task that will be executed. It's the most important task as it's one of the part which has motivated the entire project. Indeed, old packaging libraries has designed Setup script as traditional python module and it causes some problem because packagers can add valid python code in their script and nothing prevent them to add a "os.sytem('rm -Rf /')" statement. The idea is to authorize it and trace all the access to file system, network and system calls. The detection of all this comportment can be done with using a tracing library such as Strace or SystemTap. Utilization of such tools will produce raw data that we must to process to make them readable for human and useable for harmful comportment detection.

The third one is build, indeed some of python distributions may include non-python source code. These non-python source code must be compiled before the installation of the distribution, but it could fail. Currently, the packaging libraries take care of that, but it could be difficult to have detailed raw data. So we need to study how these libraries work and how we can have raw data from them, and if it's not possible, we should improve the building part of theses libraries.

Common tasks

Moreover, some tasks may be considered as standard, such as the following:

Test execution task, result of unittest, doctest is a very common task and should be also included in the execution. Test execution may also include the code coverage measure. Test execution causes some problems, as there is more than one library for testing. Work must include analysis of existent testing libraries and make tasks for each of them. This task may depends on external dependencies (specific databases, non easy-installable libraries, ...) and we need to choose which external dependencies we will install on our VM instances.

Another common task is quality check. Quality check is a vast subject, so we have thought about pylint and pychecker in first time. Another quality tool can be added case by case depending on features added of these tools.

A final task that can be added if we have time is a performance task. The idea is measuring the time took by the others tasks. This measure is not very important but could be implemented if we have time and if community really want it.

copy content of proposal <here>

existing solutions

Master Slave Architecture

What we need:

Existing solutions (just the master slave architecture):

Buildbot

Bitten

Condor

Findings

Task Manager

What we need:

Additional things to take into account:

Existing solutions:

report your finding <here>

weekly meeting report

2012-05-12

Blog

IRC

Mailing List

Source

Python Version

Wiki

Weekly Meeting

For the next week(Boris): Code : Finish some little things on PyTasks (need it for a project) and compare it to another task manager included in tools like waff and narval (not sure about pony-build) Design : Will read some of my bookmarks about distributed computing communication : Write a summary about task manager and about which protocol we can use for API

For the next week(Yeswanth):

Code : Not much this week( Will work on reading about python coding conventions) Other tools/design : Will go through condor or other equivalents and see how it fits in this project Communication : Report on condor (http://www.cs.wisc.edu/condor/) or other equivalents

Unable to edit the page? See the FrontPage for instructions.