Differences between revisions 57 and 58
Revision 57 as of 2011-10-20 02:58:49
Size: 5421
Editor: c-174-52-131-129
Comment:
Revision 58 as of 2012-03-26 08:32:38
Size: 5887
Editor: nat-pool-bne-t1
Comment: Specific notes on required resources
Deletions are marked like this. Additions are marked like this.
Line 61: Line 61:
Build Slaves and Security
-------------------------
== Required Resources ==

Based on [2]_, the recommended resource allocations for a Python buildbot are at least:

* 2 CPUs
* 512 MB RAM
* 30 GB free disk space

Note: the bigmem tests won't run in this configuration, since they require substantially more memory. However, these resources should be sufficient to ensure Python compiles correctly on the platform and can run the rest of test suite.

.. [2]: http://mail.python.org/pipermail/python-dev/2012-March/117978.html

== Build Slaves and Security ==

This page contains general information about the Python BuildBot setup and issues.

Here are the current BuildBot results. For command-line querying of buidbot status (instead of the aforementioned Web page), you can use the bbreport tool.

We need to add more platforms (and compilers!) to the BuildBot farm. If you have one of these machines you are willing to run BuildBot on, please contact python-dev@python.org:

  • Windows (and see BuildbotOnWindows)

  • AIX
  • HP-UX
  • FreeBSD, NetBSD, OpenBSD
  • Solaris / OpenSolaris / OpenIndiana / Illumos

It might also be beneficial to run on older versions of operating systems or others not mentioned above. Feel free to contact us if you would like to offer another type of system.

Installing a buildslave

You need to install a recent version of Buildbot. If your OS has a packaging system (e.g. under Linux), it is probably available from the standard repositories. Otherwise, you'll have to build it from source (in which case you'll have to install Python and Twisted first).

A buildslave also needs the subversion client, and it needs to be able to build Python and some of its extension modules. It means you should have the standard development tools installed (compiler, linker), and the development headers for a couple of third-party libraries (such as zlib and OpenSSL).

Once all this is done, create a new user "buildbot" if it doesn't exist (your package manager might have done it for you). Then:

 % su - buildbot
 % mkdir buildarea
 % buildslave create-slave `pwd`/buildarea dinsdale.python.org:9020 slavename slavepasswd

If you have an older version of buildbot installed, the command will be "buildbot" instead of "buildslave". If this is the case, you should upgrade to a newer version of buildbot. If you can't find either, you should check to see that /usr/local/bin (the default installed location) is on your path.

You'll need to get someone to create the slavename/slavepasswd on dinsdale.python.org before doing this. Ask on python-dev@python.org once you're ready.

Then edit buildarea/info/admin and buildarea/info/host to set them appropriately.

Finally, start buildbot (still under the 'buildbot' user) with:

 % buildslave start ~/buildarea

Once the buildbot is running, don't forget to monitor the build results and solve any setup issues causing test failures.

Required ports

In order for the buildbot to operate properly, it needs access to various TCP ports on the internet. Below is a list of known ports and hosts; this list is not exhaustive and may evolve as new tests get written.

Port

Host

Description

20, 21

ftp.kernel.org, ftp.mirror.nl

test_urllib2net

53

your DNS server

test_socket (and others implicitly)

80

python.org

(several tests)

119

news.gmane.org

test_nntplib

443

(various)

test_ssl

465

smtp.gmail.com

test_smtpnet

9020

python.org

Buildbot connection

Many tests will also create local TCP sockets and connect to them (usually through "localhost" or "127.0.0.1"), so make sure this capability hasn't been disabled.

The following is a summary of a discussion that happened on python-dev regarding Buildbot security. No consensus came about, but the information is still helpful as a point of reference.

Required Resources

Based on [2]_, the recommended resource allocations for a Python buildbot are at least:

* 2 CPUs * 512 MB RAM * 30 GB free disk space

Note: the bigmem tests won't run in this configuration, since they require substantially more memory. However, these resources should be sufficient to ensure Python compiles correctly on the platform and can run the rest of test suite.

.. [2]: http://mail.python.org/pipermail/python-dev/2012-March/117978.html

Build Slaves and Security

Here are some important points of which you should be aware when running a build slave [1]_:

1. Anyone setting up a build slave should take care to invoke the build

  • in an environment where an out-of-control build slave, potentially executing arbitrarily horrible and/or malicious code, should not damage anything. Builders should always be isolated from valuable resources, although the specific mechanism of isolation may differ. A virtual machine is a good default, but may not be sufficient; other tools for cutting of the builder from the outside world would be chroot jails, solaris zones, etc.

2. Code runs differently as privileged vs. unprivileged users.

  • Therefore builders should be set up in both configurations, running the full test suite, to ensure that all code runs as expected in both configurations. Some tests, as the start of `this thread

    <http://mail.python.org/pipermail/python-dev/2011-October/113935.html>`_ indicates, must have some special logic to make sure they do or do not run, or run differently, in privileged vs. unprivileged configurations, but generally speaking most things should work in both places.

3. Access to root may provide access to slightly surprising resources,

  • even within a VM (such as the ability to send spoofed IP packets, change the MAC address of even virtual ethernet cards, etc), and administrators should be aware that this is the case when configuring the host environment for a run-as-root builder. You don't want to end up with a compromised test VM that can snoop on your network.

.. [1] See the related python-dev thread <http://mail.python.org/pipermail/python-dev/2011-October/113935.html>_.

BuildBot (last edited 2018-05-05 00:04:24 by GregoryPSmith)

Unable to edit the page? See the FrontPage for instructions.