- Getting a copy of b.p.o tracker configuration
- Resources for Tracker Development
- Getting Your Own Jython Tracker
Getting a copy of b.p.o tracker configuration
bugs.python.org is powered by Roundup tracker software released under MIT License. To be honest, it uses a heavily patched Roundup stored in this repository:
http://hg.python.org/tracker/roundup (make sure to use bugs.python.org branch)
The specific configuration (called 'tracker home' or 'instance') that is running on http://bugs.python.org/ is named python-dev. The source is stored in Mercurial repository and can be downloaded from:
- PostgreSQL (as a backend database)
- psycopg (bindings for the above)
copy of python-dev configuration
pydotorg has several official trackers, which configuration is stored in Mercurial repositories accessible from the http://hg.python.org/tracker/ address:
roundup: a clone of the original Roundup with our modifications;
python-dev: the instance used on bugs.python.org;
meta: the instance used for the meta tracker;
jython: the instance used for the jython tracker;
setuptools: the instance used for the setuptools tracker;
rietveld: a clone of Rietveld used to do reviews on the bug tracker;
django-gae2django: a clone of django-gae2django necessary for rietveld;
The roundup repo is a clone of http://roundup.hg.sourceforge.net/hgweb/roundup/roundup that contains an additional branch named "bugs.python.org" that contains some modification we made to Roundup.
To get a clone of roundup use:
- Read-only clone
hg clone http://hg.python.org/tracker/roundup
- Read/Write clone (if you have write access to the repository)
hg clone ssh://email@example.com/tracker/roundup
Before installing roundup, remember to change branch, otherwise the instances won't work:
cd roundup hg up bugs.python.org
To install roundup use:
sudo python setup.py install --prefix /opt/tracker-roundup/
Install and configure PostgreSQL
For the python-dev tracker instance we use PostgreSQL as backend database. Roundup has support for many different databases for backend (anydbm, metakit, MySQL, PostgreSQL, SQLite...), but for the python-dev instance, PostgreSQL was selected as it has the best performance for large installations. Also, for performance reasons, there is at least one place in the python-dev instance that has a hardcoded dependency on the backend being an SQL-based one, so you do need PostgreSQL even for development work (another SQL-based backend might work, but this has not been tested). I recommend that you run postgreSQL to make your development environment behave exactly like the environment the real tracker runs in.
Details on how to install PostgreSQL is outside the scope of this Wiki Page (sudo apt-get install postgresql should be enough). Both version 7.4 and 8.1 of PostgreSQL has been tested without any known problems. It has been reported that 8.2, 8.3, 8.4, and 9.1 work well.
Access configuration of postgresql can be tricky. You need to make sure that the user you're running roundup as (your unix/windows user) has access to the postgresql database you intend to use for roundup, as the postgresql user you've configured in roundup's configuration.
For a development environment with no demands on security, the easiest way to configure this is probably to allow all users to connect as any database user as long as the request originates from the local host. This can be accomplished editing pg_hba.conf (usually found in /etc/postgresql/8.x/main/ in Linux or /Library/PostgreSQL/9.x/data/ in Mac OS X -- you might need privileges to edit it). The end of the file should look like this (the four changed lines are preceded by ##):
# Database administrative login by UNIX sockets ##local all postgres ident local all postgres trust # TYPE DATABASE USER CIDR-ADDRESS METHOD # "local" is for Unix domain socket connections only ##local all all ident local all all trust # IPv4 local connections: ##host all all 127.0.0.1/32 md5 host all all 127.0.0.1/32 trust # IPv6 local connections: ##host all all ::1/128 md5 host all all ::1/128 trust
PLEASE NOTE: This is not a secure configuration on a multi-user machine.
After the change in pg_hba.conf, reload your postgreSQL database to make sure it knows about your new access configuration. Something similar to
sudo /etc/init.d/postgresql reload
should do the trick on Unix-like platforms.
On Mac OS X, do sudo su - postgres. Then use pg_ctl to restart the server:
/Library/PostgreSQL/9.x/bin/pg_ctl restart -D /Library/PostgreSQL/9.x/data
Create a database user that is allowed to create databases. This user will be used when roundup connects to the database.
$ psql -U postgres postgres=# create user roundup; postgres=# alter user roundup with createdb;
Upgrading PostgreSQL from 8.4 to 9.1
If you have a tracker instance running on PostgreSQL 8.4 and you want to update it to 9.1, follow these steps:
1. Install postgresql-9.1;
2. Remove the new default cluster using sudo pg_dropcluster --stop 9.1 main;
3. Update the old clusted with sudo pg_upgradecluster 8.4 main;
4. If you get the error:
The PostgreSQL server failed to start. Please check the log output: FATAL: could not create shared memory segment: Invalid argument DETAIL: Failed system call was shmget(key=5432001, size=34922496, 03600). HINT: This error usually means that PostgreSQL's request for a shared memory segment exceeded your kernel's SHMMAX parameter. You can either reduce the request size or reconfigure the kernel with larger SHMMAX. To reduce the request size (currently 34922496 bytes), reduce PostgreSQL's shared memory usage, perhaps by reducing shared_buffers or max_connections. If the request size is already small, it's possible that it is less than your kernel's SHMMIN parameter, in which case raising the request size or reconfiguring SHMMIN is called for.
you check the value of shmmax with sysctl -a | grep -i shm. You can change the value permanently adding kernel.shmmax = 34922496 in /etc/sysctl.d/30-postgresql-shm.conf and running sudo sysctl -p to apply the changes, or temporarily with sudo sysctl kernel.shmmax=34922496.
5. If everything works with 9.1 you can then remove 8.4;
Installing psycopg or psycopg2
You need to install psycopg or psycopg2:
sudo apt-get install python-psycopg2
In case that doesn't work, you can download the latest .tar.gz from http://www.initd.org, extract it and install it:
tar xzf psycopg2-2.3.2.tar.gz cd psycopg2-2.3.2/ python setup.py build sudo python setup.py install
If during the python setup.py build you get:
"fatal error: Python.h: No such file or directory", you need to sudo apt-get install python-dev;
"fatal error: libpq-fe.h: No such file or directory", you need to sudo apt-get install libpq-dev;
If you are having other problems or installing a different version check psycopg2-2.3.2/INSTALL.
Checkout the python-dev instance
To get a clone of the python-dev tracker instance use:
- Read-only clone
hg clone http://hg.python.org/tracker/python-dev
- Read/Write clone (if you have write access to the repository)
hg clone ssh://firstname.lastname@example.org/tracker/python-dev
Checkout the Rietveld integration
Running the Rietveld code review tool is optional. If you want to work on it, check out the django-gae2django and rietveld clones in the python-dev/ directory:
- Read-only clone
cd python-dev hg clone http://hg.python.org/tracker/django-gae2django/ hg clone http://hg.python.org/tracker/rietveld/
- Read/Write clone (if you have write access to the repository)
cd python-dev hg clone ssh://email@example.com/tracker/django-gae2django/ hg clone ssh://firstname.lastname@example.org/tracker/rietveld/
After getting the clones remember to update the branches to bugs.python.org:
cd django-gae2django hg up bugs.python.org cd .. cd rietveld hg up bugs.python.org
In addition, Django needs to be installed; e.g. the Django 1.2 or 1.3 Debian packages work fine.
Configure your Development Roundup Instance
Now enter the python-dev directory, and create the 'db' directory, as well as the db/backend_name file which decides which backend (i.e. database type) to use:
cd instances/python-dev mkdir db echo postgresql > db/backend_name
Note: the db dir contains all the messages and files attached to issues, and it's removed and recreated every time you run roundup-admin init.
Copy python-dev/config.ini.template into config.ini, and modify it according to your site. Pay special attention to settings marked with NO DEFAULT. However, if you've configure your PostgreSQL as above, most settings should be OK.
If you need to do development on the busybody or tellteam detectors, copy detectors/config.ini.template into detectors/config.ini, and adjust the e-mail addresses accordingly.
If you don't want to configure roundup to send emails, you can set
debug = debugmail.txt
at the end of the [mail] section. The mail will just be saved on the file you specify.
If you want to see the tracebacks in the browser instead of having them sent by email, at the end of the [web] section set
debug = on
Initialize your roundup instance:
/opt/tracker-roundup/bin/roundup-admin -i <your python-dev directory> init
Provide a password for the 'admin' user when asked for.
Note: you have to provide the full path to your python-dev directory, otherwise the command will fail.
If you are getting the error:
"roundup.hyperdb.DatabaseError: FATAL: password authentication failed for user "roundup"" check again pg_hba.conf;
"roundup.configuration.OptionUnsetError: MAIL_DOMAIN is not set and has no default" set the option in python-dev/config.ini, [mail] section, domain;
Start your roundup instance:
/opt/tracker-roundup/bin/roundup-server -p 9999 python-dev=<your python-dev directory>
You will have to use this command every time you (re)start roundup, so it's easier to create an alias for it with:
alias start-roundup='/opt/tracker-roundup/bin/roundup-server -p 9999 python-dev=<your python-dev directory>'
It might be a good idea to create another alias for roundup admin too:
alias admin-roundup='/opt/tracker-roundup/bin/roundup-admin -i <your python-dev directory>'
then add them to your .bashrc (or ~/.profile in Mac OS X)
You should now be able to browse http://localhost:9999/python-dev/ and get a roundup instance that looks just like http://bugs.python.org, except for some missing new values for fields like Stages and Keywords. It's possible to replace initial_data.py with an updated version so that your fields will have values that match those currently present in the Python Tracker.
The IDs for values in a given field might be different from those in the Python Tracker. This should only be a problem if you try to import CSV files from one tracker into the other.
Create an user from User->register in the left sidebar. You can then use roundup-admin to set developer privileges for it:
/opt/tracker-roundup/bin/roundup-admin -i <your python-dev directory> Roundup 1.4.10 ready for input. Type "help" for help. roundup> display user8 ... roundup> set user8 roles=Developer roundup> commit roundup>
You can find your user id clicking on "Your Details" in the left sidebar and then looking at the address bar (it should show something similar to http://localhost:9999/python-dev/user8).
The Roundup and Rietveld data are stored in the same Postgres database. After the Roundup tables have been created, add to python-dev/config.ini:
[django] secret_key = <the_secret_key>
To generate the secret key you can use an online django secret key generator, or just put an arbitrary string.
Before continuing make sure to check with "hg branch" that the branch is "bugs.python.org" on both the rietveld and the django-gae2django clones.
cd rietveld python manage.py syncdb cd .. PYTHONPATH=detectors/:/opt/tracker-roundup/lib/python2.7/site-packages/ python scripts/initrietveld
If you are upgrading from an older installation, review the revision history of rietveld/migration.
To run the Rietveld code, do
python manage.py runserver
This will run Rietveld by default on port 8000. Alternatively to the development server, you can also configure your web server to run rietveld.wsgi under the /review URL.
Rietveld patchset creation is an offline activity. If you have new patches in your tracker instance, run
If you get errors like:
django.db.utils.DatabaseError: column codereview_repository.guid does not exist LINE 1: ...itory"."url", "codereview_repository"."owner_id", "coderevie...
you can try to remove the table(s) from the database and re-run syncdb and initrietveld:
$ psql -U roundup psql (9.1.4) Type "help" for help. roundup=> \d codereview_repository Table "public.codereview_repository" ... check here if some column is missing and if it is drop the table ... roundup=> drop table codereview_repository cascade; NOTICE: drop cascades to constraint codereview_branch_repo_id_fkey on table codereview_branch DROP TABLE roundup=> \q $ cd rietveld $ python manage.py syncdb Creating tables ... Creating table codereview_repository Installing custom SQL ... Installing indexes ... No fixtures found. $ cd .. $ PYTHONPATH=detectors/:/opt/tracker-roundup/lib/python2.7/site-packages/ python scripts/initrietveld
irkerd is a daemon that sends notification on IRC whenever a message is added or an issue is created, and replaces the CIA.vc service.
You will need:
* irker: http://www.catb.org/esr/irker/
* the irc package: http://pypi.python.org/pypi/irc/
To install them you can use:
wget http://pypi.python.org/packages/source/i/irc/irc-5.0.1.zip unzip irc-5.0.1.zip cd irc-5.0.1/ sudo python setup.py install cd .. wget http://www.catb.org/~esr/irker/irker-1.17.tar.gz tar -zxvf irker-1.17.tar.gz
If you get the error ImportError: No module named hgtools.plugins during the sudo python setup.py install, you can try the command again and it should automagically work.
Make sure that you have listed the address of the channels that should receive the messages in detectors/config.ini, e.g.:
[irker] channels = irc://chat.freenode.net/python-dev
To start irker use:
cd irker-1.17/ ./irkerd &
You can also use ./irkerd -d3 to see debugging messages.
Resources for Tracker Development
Subscribe to and ask your question on the tracker-discuss mailing list. See http://mail.python.org/mailman/listinfo/tracker-discuss
Since the Python Tracker is a slightly modified version of Roundup, both Roundup's Documentation and issue tracker contain relevant information about how the Python Tracker works and problems one might find working with its code.
To start roundup-admin interactively use:
/opt/tracker-roundup/bin/roundup-admin -i <your python-dev directory>
To get the list of users, issues, msgs, files, etc:
roundup> list user
To see info about a specific user, issue, msg, file, etc:
roundup> display userXXX
To change the role of an user:
roundup> set userXXX roles=User,Developer roundup> commit
Users have limited privileges and can edit just some of the fields in the issue page. Developers can edit all the fields and remove msgs and files. Coordinators can also mark/unmark messages as spam and change some of the settings of other users (e.g. if they submitted the contributor agreement).
To add back a message deleted accidentally:
roundup> set issueXXX messages=+YYY roundup> commit
The message will be added back in the right position. (This should be equivalent to b.p.o/issueXXX?@action=edit&@add@messages=YYY, but this doesn't seem to work.)
The Meta Tracker
Setting Up an Instance in a VirtualEnv
The Test Tracker
Public test instance of the Python tracker: http://bot.bio.br/python-dev/ (currently offine)
Instance that attempts to match the code used in http://bugs.python.org to allow testing, reproducing and verifying fixes for tracker bugs. Can also be used as a sandbox by users interested in learning about Developer, Coordinator or Admin tasks and features. Testing new content (e.g. Components or Statuses) is OK, but new features show be tested in the Experimental Tracker instead.
The Experimental Tracker
Modified version of the Python tracker: http://bot.bio.br/python-dev-exp/ (currently offline)
It's an instance to showcase and test new features.
New features (2009-04-18):
There is also a list of additional DesiredTrackerFeatures for which no patches yet exist.
Getting Your Own Jython Tracker
To set up a local Jython tracker, please follow instructions for python-dev tracker with following exceptions:
There is no CIA.vc detector
- The repository for Jython tracker can be cloned with:
hg clone http://hg.python.org/tracker/jython/