Differences between revisions 23 and 24
Revision 23 as of 2017-06-18 21:28:14
Size: 7343
Editor: SteveHolden
Comment:
Revision 24 as of 2017-11-10 17:49:26
Size: 0
Editor: MarkMangoba
Comment: per https://mail.python.org/pipermail/python-buildbots/2017-October/000161.html
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
## page was renamed from BuilbotOnWindows
= Notes on setting up a buildbot slave on Windows for core Python testing. =

__NOTE: THIS INFORMATION IS OUTDATED AND LIKELY TO BE MISLEADING__ instead see the information on https://buildbot.net/

Because the Windows version of Python includes several external software packages, building and testing Python on Windows is elaborate. The first thing to do is to learn how to do everything needed to build and run the Python tests from a regular checkout. See `PCbuild/readme.txt` for all that, and come back here in a week ;-)

== Setup: Installing stuff ==

Running a Windows buildbot slave for core Python testing requires installing more stuff:

 * However it is you installed SVN, make sure the `svn.exe` executable is on your `PATH`.
 * Install a recent Python.
 * Install a matching version of pywin32. (If you installed ActivePython, then you have pywin32 already.)
 * Install Twisted.
 * Install buildbot.

The last one is a `setup.py`-style install, and you should run `setup.py` with the version of Python you want to install it under.

I used the following versions:
 * Subversion 1.4.5
 * Python 2.5.1
 * PyWin32 build 210
 * Twisted 2.5.0
 * Buildbot 0.7.6

I performed the buildbot install using 'setup.py' and a couple unexpected things happened. First, the `buildbot.bat` script it created in my `/Python25/Scripts/` directory contained hard-coded references to `/Python23`. It also assumed I had `python.exe` on my `PATH`, which I don't. I edited `buildbot.bat` by hand to contain this line instead:

{{{
    @\Python25\python \Python25\Scripts\buildbot %*
}}}

In addition, the file: Python25\Tools\buildbot\buildbot-0.7.6\build\lib\buildbot\scripts\startup.py contained a couple incorrect twisted lines that may be easily corrected by replacing them with:

{{{
if platformType == "win32":
    from twisted.scripts import _twistw
    run = _twistw.run
}}}

That's all I changed. I'll continue using `\Python25` below as Python's installation directory; substitute as needed.

== Setup: Configuring your buildbot slave ==

To make more progress, you have to get an admin for the buildbot master to teach the master about your machine, and negotiate a buildbot name and password with them.

When you have all that, run this from a command prompt:

{{{
    cd \Python25\Scripts\
    buildbot create-slave DIR HOST:PORT NAME PASSWORD
}}}

This is a one-time setup step, creating a directory for the buildbot slave to run in, and teaching it what it needs to connect to the buildbot master:

 * `DIR` You have a free choice about this. It's an absolute path to a directory, which probably shouldn't already exist. The buildbot slave does all its work under this directory.
 * `HOST:PORT` The master's admin will tell you the hostname and port to use for the buildbot master.
 * `NAME` The name of your buildbot slave. This shows up in the buildbot web page, and as part of directory names under `DIR`. Mine happened to be `peters-windows`. The master's admin will tell you this.
 * `PASSWORD` The master's admin will also tell you this.

Running that creates directory `DIR`. If you look at file `DIR\buildbot.tac`, you'll see that it's Python source containing the info you entered on the command line. You shouldn't need to change it.

You '''should''' edit files `DIR\info\admin` and `DIR\info\host`. It would be nicer if they had `.txt` extensions, but you'll figure it out. `admin` just contains your name and email address, while `host` should describe your testing machine briefly (like OS, processor, maybe amount of RAM). This info shows up in buildbot web pages.

== Running the buildbot slave ==

Setup is complete now for the Python trunk (see below for the 2.4 branch). To start your buildbot slave running, open a DOS box, cd to `\Python24\Scripts`\, and enter:

{{{
    buildbot start DIR
}}}

where `DIR` is the buildbot slave directory you picked. There's no output and the command never finishes. You're done, BTW! It's impossible that anything could go wrong ;-)

Note that `DIR\twistd.log` will grow over time. Useful info about Twisted's communication with the master appears here. Oddly enough, the easiest way to find out what's happening on your machine is to view the buildbot status page on `python.org`.

== 2.4 branch: Getting Python to compile ==

After you do all the above, the buildbot compile step will fail every time on the 2.4 branch. This is because compiling a full Python on Windows requires accessing header (and other) files from third-party software.

As of 21 March 2006, the trunk buildbot slave is smart enough to fetch and compile the external projects it needs all by itself. Setup for the `release24-maint` branch requires more work from you. Note that your buildbot slave will be asked to run tests for the trunk and the 2.4 branch, so you can't skip this.

Because you started at the start, and learned how to build Python from a checkout first, you already downloaded this stuff and fought your way through compiling it. Now you have to copy some of those directories so that the branch's buildbot compile step can find them too.

The general rule is that the VC project files expect all such directories to be siblings of `PCbuild`'s parent. So, e.g., the buildbot software checked out the 2.4 Python into my directory `DIR\2.4.NAME\build\`. `PCbuild` is a subdirectory of that, `build` is `PCbuild`'s parent, and so I copied these directories under `DIR\2.4.NAME\`, to make them siblings of `build`:

{{{
    db-4.2.52
    openssl-0.9.7d
    tcltk
    zlib-1.2.3
}}}


== Ignorance is Bliss ==

The above is ''everything'' I did. In particular, I didn't bother doing dozens of things various pieces of docs suggested I do. I didn't see any point to them, and Linux-heads are forever suggesting needless complications on Windows. Trust me :-)

== Common Problems ==

 * The buildbot compile step fails with
{{{
LINK : fatal error LNK1104: cannot open file './python25_d.dll'
}}}

during `generate buildinfo`. This never fixes itself. Sometimes when the buildbot software tries to kill the Python process (this isn't "normal", but it does happen from time to time for various reasons), it fails, and leaves the `python_d.exe` process behind. Find that process and kill it yourself.

 * You get

{{{
exceptions.TypeError: cannot concatenate 'str' and 'NoneType' objects
}}}

right at the start, coming from the buildbot software. A common cause (there may be others -- unsure) is doing a "force build" from the web interface on the trunk, but neglecting to type `trunk` in the "Branch to build" box. Although it's not obvious, you can't leave that box empty (well, not if you want to make progress ...).

== Configuring for buildmsi.bat ==

To make buildmsi.bat work, the following prerequisites must be met:

  * Cygwin must be installed, in the path, and must contain make, python 2.5.x (x>=1), and binutils

  * HTML Help Workshop must be installed in its default location (Program Files\HTML Help Workshop)

  * The Win32 extensions need to be installed (but they should be, according to the instructions above, anyway)

  * the slave must run at least buildbot 0.7.5.

Unable to edit the page? See the FrontPage for instructions.