Notes on setting up a buildbot slave on Windows for core Python testing.

Because the Windows version of Python includes several external software packages, building and testing Python on Windows is elaborate. The first thing to do is to learn how to do everything needed to build and run the Python tests from a regular checkout. See PCbuild/readme.txt for all that, and come back here in a week

Setup: Installing stuff

Running a Windows buildbot slave for core Python testing requires installing more stuff:

• However it is you installed SVN, make sure the svn.exe executable is on your PATH.

• Install a recent Python.

• Install a matching version of pywin32. (If you installed ActivePython, then you have pywin32 already.)

• Install Twisted.
• Install buildbot.

The last one is a setup.py-style install, and you should run setup.py with the version of Python you want to install it under.

I used the following versions:

• Subversion 1.4.5
• Python 2.5.1

• PyWin32 build 210

• Twisted 2.5.0
• Buildbot 0.7.6

I performed the buildbot install using 'setup.py' and a couple unexpected things happened. First, the buildbot.bat script it created in my /Python25/Scripts/ directory contained hard-coded references to /Python23. It also assumed I had python.exe on my PATH, which I don't. I edited buildbot.bat by hand to contain this line instead:

    @\Python25\python \Python25\Scripts\buildbot %*

In addition, the file: Python25\Tools\buildbot\buildbot-0.7.6\build\lib\buildbot\scripts\startup.py contained a couple incorrect twisted lines that may be easily corrected by replacing them with:

if platformType == "win32":
    from twisted.scripts import _twistw
    run = _twistw.run

That's all I changed. I'll continue using \Python25 below as Python's installation directory; substitute as needed.

Setup: Configuring your buildbot slave

To make more progress, you have to get an admin for the buildbot master to teach the master about your machine, and negotiate a buildbot name and password with them.

When you have all that, run this from a command prompt:

    cd \Python25\Scripts\
    buildbot create-slave DIR HOST:PORT NAME PASSWORD

This is a one-time setup step, creating a directory for the buildbot slave to run in, and teaching it what it needs to connect to the buildbot master:

• DIR You have a free choice about this. It's an absolute path to a directory, which probably shouldn't already exist. The buildbot slave does all its work under this directory.

• HOST:PORT The master's admin will tell you the hostname and port to use for the buildbot master.

• NAME The name of your buildbot slave. This shows up in the buildbot web page, and as part of directory names under DIR. Mine happened to be peters-windows. The master's admin will tell you this.

• PASSWORD The master's admin will also tell you this.

Running that creates directory DIR. If you look at file DIR\buildbot.tac, you'll see that it's Python source containing the info you entered on the command line. You shouldn't need to change it.

You should edit files DIR\info\admin and DIR\info\host. It would be nicer if they had .txt extensions, but you'll figure it out. admin just contains your name and email address, while host should describe your testing machine briefly (like OS, processor, maybe amount of RAM). This info shows up in buildbot web pages.

Running the buildbot slave

Setup is complete now for the Python trunk (see below for the 2.4 branch). To start your buildbot slave running, open a DOS box, cd to \Python24\Scripts\, and enter:

    buildbot start DIR

where DIR is the buildbot slave directory you picked. There's no output and the command never finishes. You're done, BTW! It's impossible that anything could go wrong

Note that DIR\twistd.log will grow over time. Useful info about Twisted's communication with the master appears here. Oddly enough, the easiest way to find out what's happening on your machine is to view the buildbot status page on python.org.

2.4 branch: Getting Python to compile

After you do all the above, the buildbot compile step will fail every time on the 2.4 branch. This is because compiling a full Python on Windows requires accessing header (and other) files from third-party software.

As of 21 March 2006, the trunk buildbot slave is smart enough to fetch and compile the external projects it needs all by itself. Setup for the release24-maint branch requires more work from you. Note that your buildbot slave will be asked to run tests for the trunk and the 2.4 branch, so you can't skip this.

Because you started at the start, and learned how to build Python from a checkout first, you already downloaded this stuff and fought your way through compiling it. Now you have to copy some of those directories so that the branch's buildbot compile step can find them too.

The general rule is that the VC project files expect all such directories to be siblings of PCbuild's parent. So, e.g., the buildbot software checked out the 2.4 Python into my directory DIR\2.4.NAME\build\. PCbuild is a subdirectory of that, build is PCbuild's parent, and so I copied these directories under DIR\2.4.NAME\, to make them siblings of build:

    db-4.2.52
    openssl-0.9.7d
    tcltk
    zlib-1.2.3

Ignorance is Bliss

The above is everything I did. In particular, I didn't bother doing dozens of things various pieces of docs suggested I do. I didn't see any point to them, and Linux-heads are forever suggesting needless complications on Windows. Trust me

Common Problems

• The buildbot compile step fails with

LINK : fatal error LNK1104: cannot open file './python25_d.dll'

during generate buildinfo. This never fixes itself. Sometimes when the buildbot software tries to kill the Python process (this isn't "normal", but it does happen from time to time for various reasons), it fails, and leaves the python_d.exe process behind. Find that process and kill it yourself.

• You get

exceptions.TypeError: cannot concatenate 'str' and 'NoneType' objects

right at the start, coming from the buildbot software. A common cause (there may be others -- unsure) is doing a "force build" from the web interface on the trunk, but neglecting to type trunk in the "Branch to build" box. Although it's not obvious, you can't leave that box empty (well, not if you want to make progress ...).

Configuring for buildmsi.bat

To make buildmsi.bat work, the following prerequisites must be met:

• Cygwin must be installed, in the path, and must contain make, python 2.5.x (x>=1), and binutils

• HTML Help Workshop must be installed in its default location (Program Files\HTML Help Workshop)
• The Win32 extensions need to be installed (but they should be, according to the instructions above, anyway)
• the slave must run at least buildbot 0.7.5.