Pypi Testing Infrastructure

The Python Package Index (namely PyPI) allows anyone to upload projects. This testing infrastructure wants to provide a way to analyse the distributions available at pypi using metrics such as test coverage, test results, PEP8 etc. as well as feedback on the installation (does it went well? are some weird files modified? etc.)

This wiki page defines the features that will be part of this testing infrastructure, as well as the metrics that will be used.

Gsoc 2011

Two students (Boris and Yeswanth) are currently working on this project. You can follow the overall progress on their respective blogs:

General designs

Terminology

The entire project has been divided into two parts - Environment part and Execution part . Each part has been detailed below.

Environment part

Architecture

Raw Data API

Command API

Slave

The slave performs the following tasks

Master

Execution part

Detailed work

During the preparation's discussions of the project, we have cut the project into two different parts which are depending on each other. This proposal is one of them: the execution part. The other part the establishment of clean environment (inside a VM instance): Environment part.

Work can be divided into three parts, which I will detail below:

Create an execution manager. Write the essentials tasks. Write the common tasks.

Global Architecture

Even if this proposal concern the execution part of PYTI, choices made during preparation's discussions have an influence on the work, i will detail theses choices below and their influences:

Tests will be executed inside a Virtual Machine instance, so we need to execute source codes from untrusted source. We chose to cut off access to the network in order to avoid sending mail, deny of service... This choice can be problematic if tests need access to the network, but good practices recommend to mock them. If several distributions cannot mock network access, we must create a method to control network access in a more accurately way.

As vm instance will not have access to network, the program which will start the instance must prepare all that the instance will need, distribution archive, of course, but also distribution's dependencies archives. The dependencies computing must be done before starting the vm instance, this part is detailed below in the "Dependency setup" part.

The last point is about Raw Data API, in fact, tasks will generate raw data (see below for terminology) and they should be sent to the slave. This API is part of the complementary part of the project, but as the execution part will use it, it should be designed with the participation of all.

Execution manager

Running the tests on package content may be split into different independent tasks. These tasks can be written independently, but they must all be executed anyway. They cannot be performed in any order, as one specific task may depends on an another one and if a task fail, the system should not performed other tasks that depend on that which has failed.

This is the role of the execution manager, it will manage all the different tasks and execute them in the right order, ensuring that tasks success. Execution manager must manage these scenario:

Task 'A' depends on 'B' and 'B' one depends on 'C'. Execution order must be: 'C', 'B' and finally 'A'. If task 'B' fail, execution manager must mark 'A' as skipped due to error in one of it's dependencies. Execution manager must also provide a way for tasks to exchange data, and so we can imagine that tasks will depends on data instead of traditional dependencies. Moreover, as tasks should generate raw data, it makes more sense to manage raw data dependencies than tasks dependencies. For example:

Task 'A' need 'D' data. Task 'B' produce 'D' data during his execution. Execution order must be: 'B' and 'A'.

Essential tasks

Some of the tasks are essential and must be implemented as soon as possible.

The second one is the installation. It's one of the first task that will be executed. It's the most important task as it's one of the part which has motivated the entire project. Indeed, old packaging libraries has designed Setup script as traditional python module and it causes some problem because packagers can add valid python code in their script and nothing prevent them to add a "os.sytem('rm -Rf /')" statement. The idea is to authorize it and trace all the access to file system, network and system calls. The detection of all this comportment can be done with using a tracing library such as Strace or SystemTap. Utilization of such tools will produce raw data that we must to process to make them readable for human and useable for harmful comportment detection.

The third one is build, indeed some of python distributions may include non-python source code. These non-python source code must be compiled before the installation of the distribution, but it could fail. Currently, the packaging libraries take care of that, but it could be difficult to have detailed raw data. So we need to study how these libraries work and how we can have raw data from them, and if it's not possible, we should improve the building part of theses libraries.

Common tasks

Moreover, some tasks may be considered as standard, such as the following:

Test execution task, result of unittest, doctest is a very common task and should be also included in the execution. Test execution may also include the code coverage measure. Test execution causes some problems, as there is more than one library for testing. Work must include analysis of existent testing libraries and make tasks for each of them. This task may depends on external dependencies (specific databases, non easy-installable libraries, ...) and we need to choose which external dependencies we will install on our VM instances.

Another common task is quality check. Quality check is a vast subject, so we have thought about pylint and pychecker in first time. Another quality tool can be added case by case depending on features added of these tools.

A final task that can be added if we have time is a performance task. The idea is measuring the time took by the others tasks. This measure is not very important but could be implemented if we have time and if community really want it.

copy content of proposal <here>

existing solutions

Master Slave Architecture

What we need:

Existing solutions (just the master slave architecture):

Buildbot

Bitten

Condor

Findings

Task Manager

What we need:

Additional things to take into account:

Existing solutions:

Concepts

Pony-build

Pony-build is the continuous integration tools used last year for PYTI. Characteristics:

Score

Total: 4

Apycot/narval

Narval is a task-manager library written by logilab and is used by apycot, a continuous integration tool written by logilab. Characteristics:

Score

Total: 11

Buildbot

Buildbot is the reference of continuous integration tool in python world. Characteristics:

Score

Total: 10

PythonTasks

PythonTasks is one of boris's project, it's still in beta and no doc is written for the moment. Characteristics:

Score

Total: 11

Dependent part

Setup API

Raw Data API

Tracing library

Comparison of VMs

Criteria

Xen

Qemu+KQemu

KVM

VirtualBox

Host OS

Linux,Solaris,BSD

Most

Linux

Most

Guest OS

Most(Kernel need to be modified)

Most

Most

Most

Speed

Performance loss on disk intensive operations

Qemu is usually slow. But Kqemu has good performance

Good performance

Good performance

Virtual Hard Disk Image Format

VMDK,VHD

VMDK,VHD,QCOW2,RAW

VMDK,VHD,QCOW2,RAW

VDI,VMDK,VHD

Community Support

Community not very active in the fast few years

Support dropped for KQemu

RedHat is currently supporting it

Oracle is currently working on it

Clearly the competition drops down to Vbox,KVM Reasons for choosing Vbox over KVM 1)Vbox support for multiple platforms 2)KVM’s native disk image QCOW2 is very slow. Ofcourse I could use other disk images but that would require external tools. 3)KVM’s dependency on linux kernel

So VirtualBox to be used.

VM Handling

Different modules in the VM :

1)Starting the VM(simple version implemented, unit tests remaining) 2)Stopping the VM(simple version implemented, unit tests remaining) 3)XML Handler (simple version implemented, unit tests remaining) 4)Rollback 5)Virtual Hard Disk Handler

weekly meeting report

2012-05-12

Blog

IRC

Mailing List

Source

Python Version

Wiki

Weekly Meeting

For the next week(Boris): Code : Finish some little things on PyTasks (need it for a project) and compare it to another task manager included in tools like waff and narval (not sure about pony-build) Design : Will read some of my bookmarks about distributed computing communication : Write a summary about task manager and about which protocol we can use for API

For the next week(Yeswanth):

Code : Not much this week( Will work on reading about python coding conventions) Other tools/design : Will go through condor or other equivalents and see how it fits in this project Communication : Report on condor (http://www.cs.wisc.edu/condor/) or other equivalents

2012-05-31

Interaction Between Slave and VM
  1. Interaction between slave and VM happens through a shared folder.
  2. Slave downloads the distributions
  3. Slave creates a VM image with a specific os(that is to be tested on ) also mentioning a folder(i.e shared folder) where downloaded distributions are placed(along with the configuration file).
  4. The VM performs the tasks as mentioned in the configuration file
  5. VM copies the results and generates a raw data file (consists of all the raw data after tests are conducted) which is placed in the shared folder).
  6. Slave sends the report file to "Result Storage".

Overall Architecture
  1. Master handles "general task scheduling"
  2. Slave handles "task execution" (into vm)
  3. Result Storage handles "storing and viewing results"

Focus for the near future

For the next week (Yeswanth)

For the next week (Boris)

PyPITestingInfrastructure (last edited 2011-06-14 10:08:49 by 115)

Unable to edit the page? See the FrontPage for instructions.