Size: 16944
Comment:
|
← Revision 78 as of 2017-03-01 19:24:10 ⇥
Size: 18702
Comment: Replace outdated vagrant section with docker.
|
Deletions are marked like this. | Additions are marked like this. |
Line 1: | Line 1: |
This page is dedicated to development of Python bug tracker instance at http://bugs.python.org/ <<BR>> We have a [[http://psf.upfronthosting.co.za/roundup/meta/|meta tracker]] for current issues and [[DesiredTrackerFeatures|DesiredTrackerFeatures]] page for recording ideas. See also [[TrackerDevelopmentPlanning|TrackerDevelopmentPlanning]]. |
|
Line 3: | Line 7: |
= Getting Your Own python-dev Tracker = This page describes how to setup a roundup tracker instance that works exactly as the tracker for python-dev on http://bugs.python.org. The intended audience is people that want to help out with the tracker, developing new features etc. == Requirements == * Roundup, from the vendor branch used for python-dev (see below). * PostgreSQL for use as backend database. * Installation of psycopg. * !BeautifulSoup and !M2Crypto * A checkout of the spambayes_integration (instructions are given below). * A checkout of the python-dev instance from python's subversion repository (instructions are given below). == Installing Roundup == For python-dev, we use a slightly modified version of Roundup which has some features that are not yet in the core Roundup. To keep track of the changes, we keep a vendor branch of roundup. Check out [[http://svn.python.org/view/tracker|/tracker]] from the python subversion repository as documented on http://python.org/dev/tools/. For example, here's how to do read-only (if you don't have commit privileges) and read/write (if you have them) checkouts: {{{ svn co http://svn.python.org/projects/tracker svn co svn+ssh://pythondev@svn.python.org/tracker }}} The tracker directory contains the roundup source and the instances (e.g. python-dev, spambayes, jython, meta tracker). cd in the tracker/roundup-src dir and install Roundup using setup.py (note: you might need privileges to run this command): {{{ cd roundup-src sudo python setup.py install --prefix /opt/tracker-roundup/ }}} |
= Docker = The simplest way to work with the tracker is by using Docker. Clone [[https://github.com/python/docker-bpo|the docker-bpo repo]] and follow the README. If you want to install the tracker manually follow the instructions below. = Requirements = This configuration is deployed on production server, so it is good to stick it for development as well: * Roundup 1.4.20+ (patched version) * PostgreSQL (as a backend database) * psycopg2 (bindings for the above) * pyoidc (OpenID Connect library from https://github.com/rohe/pyoidc) http://bugs.python.org/ (`python-dev` tracker instance or configuration) is running PostgreSQL on backed, because it has the best performance for large installations. Besides, there is at least one place in the `python-dev` config with hardcoded dependency on the SQL-based backed. == Getting the source == Get patched version of [[http://www.roundup-tracker.org/|Roundup tracker]] from here: * http://hg.python.org/tracker/roundup/ (the changes are in the `bugs.python.org` branch) Tracker configuration for http://bugs.python.org/ (called 'tracker home', 'instance' or `environment) is here: * Official: http://hg.python.org/tracker/python-dev/ * Mirror: https://bitbucket.org/rirror/bugs.python.org == Other tracker configurations == There are several Roundup trackers used for Python development. Configuration for them is maintained in http://hg.python.org/tracker/ repositories: * '''roundup''': a clone of the original Roundup with our modifications; * '''python-dev''': the instance used on bugs.python.org; * '''meta''': the instance used for the meta tracker; * '''jython''': the instance used for the jython tracker; * '''setuptools''': the instance used for the setuptools tracker; * '''rietveld''': a clone of Rietveld used to do reviews on the bug tracker; * '''django-gae2django''': a clone of django-gae2django necessary for rietveld; = Setup bugs.python.org instance = == Install patched Roundup == Clone code and switch to `bugs.python.org` branch before installing. Here it is installed to hardcoded system-wide location at `/opt/tracker-roundup/`. * Read-only clone {{{ hg clone http://hg.python.org/tracker/roundup }}} * Read/Write clone (if you have write access to the repository) {{{ hg clone ssh://hg@hg.python.org/tracker/roundup }}} {{{ cd roundup hg up bugs.python.org sudo python setup.py install --prefix /opt/tracker-roundup/ }}} == Checkout the python-dev instance == * Read-only clone {{{ hg clone http://hg.python.org/tracker/python-dev }}} * Read/Write clone (if you have write access to the repository) {{{ hg clone ssh://hg@hg.python.org/tracker/python-dev }}} == Checkout the Rietveld integration (optional) == If you want to work on Rietveld code review tool, check out the django-gae2django and rietveld clones in the python-dev/ directory: * Read-only clone {{{ cd python-dev hg clone http://hg.python.org/tracker/django-gae2django/ hg clone http://hg.python.org/tracker/rietveld/ }}} * Read/Write clone (if you have write access to the repository) {{{ cd python-dev hg clone ssh://hg@hg.python.org/tracker/django-gae2django/ hg clone ssh://hg@hg.python.org/tracker/rietveld/ }}} After getting the clones '''remember to update the branches to bugs.python.org''': {{{ cd django-gae2django hg up bugs.python.org cd .. cd rietveld hg up bugs.python.org }}} In addition, Django needs to be installed; e.g. the Django 1.2 or 1.3 Debian packages work fine. == Install psycopg2 (PostgreSQL access library for Python) == For Debian/Ubuntu: {{{ sudo apt-get install python-psycopg2 }}} For other systems, follow instructions from [[http://initd.org/psycopg/install/|psycopg2 site]]. |
Line 50: | Line 140: |
For the python-dev tracker instance we use PostgreSQL as backend database. Roundup has support for many different databases for backend (anydbm, metakit, MySQL, PostgreSQL, SQLite...), but for the python-dev instance, PostgreSQL was selected as it has the best performance for large installations. Also, for performance reasons, there is at least one place in the python-dev instance that has a hardcoded dependency on the backend being an SQL-based one, so you do need PostgreSQL even for development work (another SQL-based backend might work, but this has not been tested). I recommend that you run postgreSQL to make your development environment behave exactly like the environment the real tracker runs in. Details on how to install PostgreSQL is outside the scope of this Wiki Page ({{{sudo apt-get install postgresql}}} should be enough). Both version 7.4 and 8.1 of PostgreSQL has been tested without any known problems. It has been reported that 8.2 and 8.3 work well. Access configuration of postgresql can be tricky. You need to make sure that the user you're running roundup as (your unix/windows user) has access to the postgresql database you intend to use for roundup, as the postgresql user you've configured in roundup's configuration. For a development environment with no demands on security, the easiest way to configure this is probably to allow all users to connect as any database user as long as the request originates from the local host. This can be accomplished editing pg_hba.conf (usually found in /etc/postgresql/8.x/main/ -- you might need privileges). The end of the file should look like this (the four changed lines are preceded by ##): |
Other backends may work, but it is recommended to keep development environment close to production. It has been reported that 8.2, 8.3, 8.4, and 9.1 work well. See [[UpgradingPostgreSQL]] if you need to upgrade version. Access control in PostgreSQL is tricky. You need to know Unix user, under which you run Roundup, setup access for him to PostgreSQL database, and add PostgreSQL user that you've configured in Roundup's configuration. For a development environment the easiest way is to allow any user from localhost to connect as any database user. This can be accomplished by editing pg_hba.conf (usually found in /etc/postgresql/9.x/main/ in Linux or /Library/PostgreSQL/9.x/data/ in Mac OS X -- you might need privileges to edit it). The end of the file should look like this (the four changed lines are preceded by ##): |
Line 97: | Line 167: |
'''PLEASE NOTE''': This is not a secure configuration on a multi-user machine. After the change in pg_hba.conf, reload your postgreSQL database to make sure it knows about your new access configuration. Something similar to |
'''PLEASE NOTE''': This is not a secure configuration on a multi-user machine. After the change in pg_hba.conf, reload your postgreSQL database to make sure it knows about your new access configuration. Something similar to |
Line 106: | Line 177: |
Line 108: | Line 181: |
Create a database user that is allowed to create databases. This user will be used when roundup connects to the database. |
On Mac OS X, do {{{sudo su - postgres}}}. Then use pg_ctl to restart the server: {{{ /Library/PostgreSQL/9.x/bin/pg_ctl restart -D /Library/PostgreSQL/9.x/data }}} Create a database user that is allowed to create databases. This user will be used when roundup connects to the database. |
Line 117: | Line 197: |
== Installing psycopg or psycopg2 == You need to install psycopg or psycopg2. Download it from http://www.initd.org, as pre-packaged versions have been reported to cause problems (note: this warning has been here for a while and might be obsolete). If you downloaded the latest .tar.gz extract it and install it: {{{ tar xzf psycopg2-2.3.2.tar.gz cd psycopg2-2.3.2/ python setup.py build sudo python setup.py install }}} If during the ''python setup.py build'' you get: * "fatal error: Python.h: No such file or directory", you need to {{{sudo apt-get install python-dev}}}; * "fatal error: libpq-fe.h: No such file or directory", you need to {{{sudo apt-get install libpq-dev}}}; If you are having other problems or installing a different version check psycopg2-2.3.2/INSTALL. == Checkout spambayes_integration == '''Note''': If you followed [[#Installing_Roundup|the instructions]] and checked out {{{http://svn.python.org/projects/tracker}}} you can '''skip this step'''. The !SpamBayes detector is '''optional''', but you must still check out the {{{spambayes_integration}}} instance. You can get a checkout of {{{spambayes_integration}}} with: {{{ svn co http://svn.python.org/projects/tracker/instances/spambayes_integration }}} If you do intend to use it, you will need to also [[#SpamBayesSetup | install SpamBayes proper and configure it appropriately]]. Otherwise, a [[#SpamBayesdummyconfiguration |dummy configuration]] should work. == Checkout the python-dev instance == '''Note''': If you followed [[#Installing_Roundup|the instructions]] and checked out {{{http://svn.python.org/projects/tracker}}} you can '''skip this step'''. The [[http://svn.python.org/view/tracker/instances/python-dev/ |python-dev]] tracker instance is version-controlled with subversion, and resides in the python subversion repository. {{{ svn co http://svn.python.org/projects/tracker/instances/python-dev }}} Keep in mind that the python-dev instance has relative symlinks to {{{../../spambayes_integration/[...]}}}, so if you did separate checkouts and your directory layout is different you have to update these links: {{{ extensions/spambayes.py -> ../../spambayes_integration/extensions/spambayes.py detectors/spambayes.py -> ../../spambayes_integration/detectors/spambayes.py }}} |
|
Line 171: | Line 201: |
Now enter the python-dev directory, and create the 'db' directory (not in svn as it's just to much fuzz to keep it there - every time you run ''roundup-admin init'' it is removed and recreated, confusing svn who wants to keep its .svn directory intact), as well as the db/backend_name file which decides which backend (i.e. database type) to use: {{{ cd instances/python-dev |
Now enter the python-dev directory, and create the 'db' directory, as well as the db/backend_name file which decides which backend (i.e. database type) to use: {{{ cd python-dev |
Line 184: | Line 210: |
Copy python-dev/config.ini.template into config.ini, and modify it according to your site. Pay special attention to settings marked with ''NO DEFAULT''. However, if you've configure your PostgreSQL as above, most settings should be OK. If you need to do development on the '''busybody''' or '''tellteam''' detectors, copy detectors/config.ini.template into detectors/config.ini, and adjust the e-mail addresses accordingly. <<Anchor(ciavc)>> The '''CIA.vc''' detector sends emails about issue changes and is on by default. Disable it by adding a {{{return}}} to its init() function (detectors/cia.py): {{{#!python def init(db): return # <-- db.issue.react('create', sendcia) db.issue.react('set', sendcia) }}} |
Note: the db dir contains all the messages and files attached to issues, and it's removed and recreated every time you run {{{roundup-admin init}}}. Copy {{{python-dev/config.ini.template}}} into {{{config.ini}}} and {{{detectors/config.ini.template}}} into {{{detectors/config.ini}}}: {{{ cp config.ini.template config.ini cp detectors/config.ini.template detectors/config.ini }}} The former will let you configure your roundup instance; the latter, detectors such as '''irker''', '''busybody''', or '''tellteam'''. In config.ini pay special attention to settings marked with ''NO DEFAULT''. {{{config.ini.template}}} already contains some useful default values for a development enviroment, for example {{{ debug = debugmail.txt }}} at the end of the {{{[mail]}}} section saves outgoing mails on a local file; and {{{ debug = on }}} at the end of the {{{[web]}}} section shows tracebacks in the browser instead of having them sent by email. |
Line 207: | Line 248: |
Line 209: | Line 251: |
Line 211: | Line 254: |
Line 212: | Line 256: |
* "{{{roundup.hyperdb.DatabaseError: FATAL: password authentication failed for user "roundup"}}}" check again pg_hba.conf; * "{{{roundup.configuration.OptionUnsetError: MAIL_DOMAIN is not set and has no default}}}" set the option in python-dev/config.ini, [mail] section, domain; '''Note''': it is possible to set the [mail] domain to "NO DEFAULT" to make the {{{init}}} work. However Roundup needs a mail server to send mails so you'll need to configure at the least the admin mail later on. It might be possible to have Roundup work with no mails at all, but the right configuration is still unknown (feel free to experiment with it!). |
* "{{{roundup.hyperdb.DatabaseError: FATAL: password authentication failed for user "roundup"}}}" check again pg_hba.conf; * "{{{roundup.configuration.OptionUnsetError: MAIL_DOMAIN is not set and has no default}}}" set the option in python-dev/config.ini, [mail] section, domain; |
Line 225: | Line 268: |
Line 226: | Line 270: |
Line 229: | Line 275: |
and add it to your {{{.bashrc}}}. You should now be able to browse http://localhost:9999/python-dev/ and get a roundup instance that looks just like http://bugs.python.org, except for some missing new values for fields like Stages and Keywords. It's possible to replace '''initial_data.py''' with an [[attachment:initial_data_updated.py | updated version]] so that your fields will have values that match those currently present in the Python Tracker. |
It might be a good idea to create another alias for roundup admin too: {{{ alias admin-roundup='/opt/tracker-roundup/bin/roundup-admin -i <your python-dev directory>' }}} then add them to your {{{.bashrc}}} (or {{{~/.profile}}} in Mac OS X) You should now be able to browse http://localhost:9999/python-dev/ and get a roundup instance that looks just like http://bugs.python.org, except for some missing new values for fields like Stages and Keywords. It's possible to replace '''initial_data.py''' with an [[TrackerDevelopment?action=AttachFile&do=view&target=initial_data_updated.py|updated version]] so that your fields will have values that match those currently present in the Python Tracker. |
Line 236: | Line 293: |
Create an user from User->register in the left sidebar. You can then use roundup-admin to set developer privileges for it: |
Create an user from User->register in the left sidebar. Note that if you have set {{{debug = debugmail.txt}}} as suggested above, you will have to open the file and copy the URL in the browser (make sure to replace {{{=3D}}} with {{{=}}} in the URL and to remove the trailing {{{=}}} where the url is broken at the end of the line). You can then use roundup-admin to set developer privileges for it: |
Line 240: | Line 299: |
Roundup 1.4.10 ready for input. | Roundup 1.5.0 ready for input. |
Line 242: | Line 301: |
roundup> display user8 | roundup> display user3 |
Line 244: | Line 303: |
roundup> set user8 roles=Developer | roundup> set user3 roles=User,Developer,Coordinator |
Line 249: | Line 308: |
You can find your user id clicking on "Your Details" in the left sidebar and then looking at the address bar (it should show something similar to http://localhost:9999/python-dev/user8). == SpamBayes Setup == If you intend to use the !SpamBayes detector to weed out spam from your tracker you will need to install it and start an instance of its {{{core_server}}} application. Briefly: * Download and install at least version 1.1a4 of the [[http://sourceforge.net/projects/spambayes|SpamBayes source]] from !SourceForge. If you are running on Windows '''do not''' use the binary installer! * Establish a basic configuration, e.g.: Devote a directory to it, e.g.: {{{ SBDIR=/usr/local/spambayes/core_server # or whatever... mkdir -p $SBDIR }}} Create an INI file: {{{ cd $SBDIR cat > bayescustomize.ini <<EOF [globals] verbose:False [Headers] include_evidence:True include_score:True [Tokenizer] record_header_absence:True summarize_email_prefixes:True summarize_email_suffixes:True mine_received_headers:True x-pick_apart_urls:True x-fancy_url_recognition:False x-lookup_ip:True lookup_ip_cache:$SBDIR/dnscache.pck max_image_size:100000 crack_image_cache:$SBDIR/imagecache.pck crack_images:True image_size:True ocr_engine:gocr [Classifier] use_bigrams:False [Categorization] ham_cutoff:0.2 spam_cutoff:0.85 [Storage] persistent_storage_file:$SBDIR/hammie.db persistent_use_database:pickle messageinfo_storage_file:$SBDIR/messageinfo.fs [html_ui] display_score:True EOF }}} Finally, run it: {{{ BAYESCUSTOMIZE=$SBDIR/bayescustomize.ini core_server.py -m XMLRPCPlugin }}} * If you are seeing image spam (this seems not to be a problem with wiki spam - yet), download and install [[http://jocr.sourceforge.net/|GOCR]]. == Common Problems == === SpamBayes dummy configuration === Incomplete/Missing !SpamBayes setup:: In your detectors/config.ini file set {{{spambayes_uri}}} to a syntactically valid, but nonexistent URI, e.g.: {{{ spambayes_uri = "http://localhost:9999/nobodyhere" }}} Check [[http://mail.python.org/pipermail/tracker-discuss/2008-February/001549.html |other available workarounds]]. === Misbehaving detectors === Problematic detectors may be disabled by [[#ciavc |returning from their init() function]], as was the case of detectors/audit2to3.py (no longer present). Note that this would be a temporary solution before a graceful fix became available. Reporting issues with local instances is the best way help solving the problem. |
You can find your user id clicking on "Your Details" in the left sidebar and then looking at the address bar (it should show something similar to http://localhost:9999/python-dev/user3). == Rietveld Setup == The Roundup and Rietveld data are stored in the same Postgres database. After the Roundup tables have been created, add to python-dev/config.ini: {{{ [django] secret_key = <the_secret_key> }}} To generate the secret key you can use an online django secret key generator, or just put an arbitrary string. Before continuing make sure to check with "hg branch" that the branch is "bugs.python.org" on both the rietveld and the django-gae2django clones. {{{ cd rietveld python manage.py syncdb cd .. PYTHONPATH=detectors/:/opt/tracker-roundup/lib/python2.7/site-packages/ python scripts/initrietveld }}} If you are upgrading from an older installation, review the revision history of rietveld/migration. To run the Rietveld code, do {{{ python manage.py runserver }}} This will run Rietveld by default on port 8000. Alternatively to the development server, you can also configure your web server to run rietveld.wsgi under the /review URL. Rietveld patchset creation is an offline activity. If you have new patches in your tracker instance, run {{{ scripts/addpatchsets }}} If you get errors like: {{{ django.db.utils.DatabaseError: column codereview_repository.guid does not exist LINE 1: ...itory"."url", "codereview_repository"."owner_id", "coderevie... }}} you can try to remove the table(s) from the database and re-run syncdb and initrietveld: {{{ $ psql -U roundup psql (9.1.4) Type "help" for help. roundup=> \d codereview_repository Table "public.codereview_repository" ... check here if some column is missing and if it is drop the table ... roundup=> drop table codereview_repository cascade; NOTICE: drop cascades to constraint codereview_branch_repo_id_fkey on table codereview_branch DROP TABLE roundup=> \q $ cd rietveld $ python manage.py syncdb Creating tables ... Creating table codereview_repository Installing custom SQL ... Installing indexes ... No fixtures found. $ cd .. $ PYTHONPATH=detectors/:/opt/tracker-roundup/lib/python2.7/site-packages/ python scripts/initrietveld }}} == irker setup == irkerd is a daemon that sends notification on IRC whenever a message is added or an issue is created, and replaces the CIA.vc service. You will need: * irker: http://www.catb.org/esr/irker/ * the irc package: http://pypi.python.org/pypi/irc/ To install them you can use: {{{ wget http://pypi.python.org/packages/source/i/irc/irc-5.0.1.zip unzip irc-5.0.1.zip cd irc-5.0.1/ sudo python setup.py install cd .. wget http://www.catb.org/~esr/irker/irker-1.17.tar.gz tar -zxvf irker-1.17.tar.gz }}} If you get the error {{{ImportError: No module named hgtools.plugins}}} during the {{{sudo python setup.py install}}}, you can try the command again and it should automagically work. Make sure that you have listed the address of the channels that should receive the messages in {{{detectors/config.ini}}}, e.g.: {{{ [irker] channels = irc://chat.freenode.net/python-dev }}} To start irker use: {{{ cd irker-1.17/ ./irkerd & }}} You can also use {{{./irkerd -d3}}} to see debugging messages. |
Line 327: | Line 458: |
Line 329: | Line 461: |
Line 331: | Line 464: |
Since the Python Tracker is a slightly modified version of [[http://www.roundup-tracker.org |Roundup]], both Roundup's [[http://www.roundup-tracker.org/docs.html |Documentation]] and [[http://issues.roundup-tracker.org/ |issue tracker]] contain relevant information about how the Python Tracker works and problems one might find working with its code. | Since the Python Tracker is a slightly modified version of [[http://www.roundup-tracker.org|Roundup]], both Roundup's [[http://www.roundup-tracker.org/docs.html|Documentation]] and [[http://issues.roundup-tracker.org/|issue tracker]] contain relevant information about how the Python Tracker works and problems one might find working with its code. == Using roundup-admin == To start roundup-admin interactively use: {{{ /opt/tracker-roundup/bin/roundup-admin -i <your python-dev directory> }}} To get the list of users, issues, msgs, files, etc: {{{ roundup> list user }}} To see info about a specific user, issue, msg, file, etc: {{{ roundup> display userXXX }}} To change the role of an user: {{{ roundup> set userXXX roles=User,Developer roundup> commit }}} Users have limited privileges and can edit just some of the fields in the issue page. Developers can edit all the fields and remove msgs and files. Coordinators can also mark/unmark messages as spam and change some of the settings of other users (e.g. if they submitted the contributor agreement). To add back a message deleted accidentally: {{{ roundup> set issueXXX messages=+YYY roundup> commit }}} The message will be added back in the right position. (This should be equivalent to b.p.o/issueXXX?@action=edit&@add@messages=YYY, but this doesn't seem to work.) |
Line 335: | Line 521: |
A [[http://psf.upfronthosting.co.za/roundup/meta/ |Meta Tracker]] is available for handling bugs and features requests for the [[http://bugs.python.org/ |Python Tracker]]. == Learn more About the Importer(s) == http://svn.python.org/projects/tracker/importer/README.rst has some information about the importer(s) written to convert sourceforge data into roundup data. |
A [[http://psf.upfronthosting.co.za/roundup/meta/|Meta Tracker]] is available for handling bugs and features requests for the [[http://bugs.python.org/|Python Tracker]]. == Setting Up an Instance in a VirtualEnv == See [[TrackerDevelopmentVirtualEnv|TrackerDevelopmentVirtualEnv]]. |
Line 343: | Line 533: |
Public test instance of the Python tracker: http://bot.bio.br/python-dev/ (currently offine) | Public test instance of the Python tracker: http://bot.bio.br/python-dev/ (currently offine) |
Line 347: | Line 539: |
Currently the email system is disabled (redirected to file), so people interested in having an account there to test new features should email [[http://mail.python.org/mailman/listinfo/tracker-discuss|tracker-discuss]] to get one. Maintained by DanielDiniz. | Currently the email system is disabled (redirected to file), so people interested in having an account there to test new features should email [[http://mail.python.org/mailman/listinfo/tracker-discuss|tracker-discuss]] to get one. Maintained by [[DanielDiniz|DanielDiniz]]. |
Line 352: | Line 545: |
Line 354: | Line 548: |
Line 356: | Line 551: |
Line 357: | Line 553: |
Line 368: | Line 566: |
Currently the email system is disabled (redirected to file), so people interested in having an account there to test new features should email [[http://mail.python.org/mailman/listinfo/tracker-discuss|tracker-discuss]] to get one. Maintained by DanielDiniz. There is also a list of additional DesiredTrackerFeatures for which no patches yet exist. = Getting Your Own jython Tracker = To set up a local jython tracker, please follow instructions for python-dev tracker with following exceptions: * There is no need for !BeautifulSoup and !M2Crypto |
Currently the email system is disabled (redirected to file), so people interested in having an account there to test new features should email [[http://mail.python.org/mailman/listinfo/tracker-discuss|tracker-discuss]] to get one. Maintained by [[DanielDiniz|DanielDiniz]]. There is also a list of additional [[DesiredTrackerFeatures|DesiredTrackerFeatures]] for which no patches yet exist. = Getting Your Own Jython Tracker = To set up a local Jython tracker, please follow instructions for python-dev tracker with following exceptions: |
Line 380: | Line 581: |
* The code for jython tracker can be checked out using the command: {{{ svn co http://svn.python.org/projects/tracker/instances/jython }}} |
* The repository for Jython tracker can be cloned with: {{{ hg clone http://hg.python.org/tracker/jython/ }}} |
Line 387: | Line 593: |
CategoryTracker CategoryDevelopmentProcess | [[CategoryTracker|CategoryTracker]] [[CategoryDevelopmentProcess|CategoryDevelopmentProcess]] |
This page is dedicated to development of Python bug tracker instance at http://bugs.python.org/
We have a meta tracker for current issues and DesiredTrackerFeatures page for recording ideas. See also TrackerDevelopmentPlanning.
Contents
Docker
The simplest way to work with the tracker is by using Docker.
Clone the docker-bpo repo and follow the README.
If you want to install the tracker manually follow the instructions below.
Requirements
This configuration is deployed on production server, so it is good to stick it for development as well:
- Roundup 1.4.20+ (patched version)
- PostgreSQL (as a backend database)
- psycopg2 (bindings for the above)
pyoidc (OpenID Connect library from https://github.com/rohe/pyoidc)
http://bugs.python.org/ (python-dev tracker instance or configuration) is running PostgreSQL on backed, because it has the best performance for large installations. Besides, there is at least one place in the python-dev config with hardcoded dependency on the SQL-based backed.
Getting the source
Get patched version of Roundup tracker from here:
http://hg.python.org/tracker/roundup/ (the changes are in the bugs.python.org branch)
Tracker configuration for http://bugs.python.org/ (called 'tracker home', 'instance' or `environment) is here:
Official: http://hg.python.org/tracker/python-dev/
Other tracker configurations
There are several Roundup trackers used for Python development. Configuration for them is maintained in http://hg.python.org/tracker/ repositories:
roundup: a clone of the original Roundup with our modifications;
python-dev: the instance used on bugs.python.org;
meta: the instance used for the meta tracker;
jython: the instance used for the jython tracker;
setuptools: the instance used for the setuptools tracker;
rietveld: a clone of Rietveld used to do reviews on the bug tracker;
django-gae2django: a clone of django-gae2django necessary for rietveld;
Setup bugs.python.org instance
Install patched Roundup
Clone code and switch to bugs.python.org branch before installing. Here it is installed to hardcoded system-wide location at /opt/tracker-roundup/.
- Read-only clone
hg clone http://hg.python.org/tracker/roundup
- Read/Write clone (if you have write access to the repository)
hg clone ssh://hg@hg.python.org/tracker/roundup
cd roundup hg up bugs.python.org sudo python setup.py install --prefix /opt/tracker-roundup/
Checkout the python-dev instance
- Read-only clone
hg clone http://hg.python.org/tracker/python-dev
- Read/Write clone (if you have write access to the repository)
hg clone ssh://hg@hg.python.org/tracker/python-dev
Checkout the Rietveld integration (optional)
If you want to work on Rietveld code review tool, check out the django-gae2django and rietveld clones in the python-dev/ directory:
- Read-only clone
cd python-dev hg clone http://hg.python.org/tracker/django-gae2django/ hg clone http://hg.python.org/tracker/rietveld/
- Read/Write clone (if you have write access to the repository)
cd python-dev hg clone ssh://hg@hg.python.org/tracker/django-gae2django/ hg clone ssh://hg@hg.python.org/tracker/rietveld/
After getting the clones remember to update the branches to bugs.python.org:
cd django-gae2django hg up bugs.python.org cd .. cd rietveld hg up bugs.python.org
In addition, Django needs to be installed; e.g. the Django 1.2 or 1.3 Debian packages work fine.
Install psycopg2 (PostgreSQL access library for Python)
For Debian/Ubuntu:
sudo apt-get install python-psycopg2
For other systems, follow instructions from psycopg2 site.
Install and configure PostgreSQL
Other backends may work, but it is recommended to keep development environment close to production.
It has been reported that 8.2, 8.3, 8.4, and 9.1 work well. See UpgradingPostgreSQL if you need to upgrade version.
Access control in PostgreSQL is tricky. You need to know Unix user, under which you run Roundup, setup access for him to PostgreSQL database, and add PostgreSQL user that you've configured in Roundup's configuration.
For a development environment the easiest way is to allow any user from localhost to connect as any database user. This can be accomplished by editing pg_hba.conf (usually found in /etc/postgresql/9.x/main/ in Linux or /Library/PostgreSQL/9.x/data/ in Mac OS X -- you might need privileges to edit it). The end of the file should look like this (the four changed lines are preceded by ##):
# Database administrative login by UNIX sockets ##local all postgres ident local all postgres trust # TYPE DATABASE USER CIDR-ADDRESS METHOD # "local" is for Unix domain socket connections only ##local all all ident local all all trust # IPv4 local connections: ##host all all 127.0.0.1/32 md5 host all all 127.0.0.1/32 trust # IPv6 local connections: ##host all all ::1/128 md5 host all all ::1/128 trust
PLEASE NOTE: This is not a secure configuration on a multi-user machine.
After the change in pg_hba.conf, reload your postgreSQL database to make sure it knows about your new access configuration. Something similar to
sudo /etc/init.d/postgresql reload
should do the trick on Unix-like platforms.
On Mac OS X, do sudo su - postgres. Then use pg_ctl to restart the server:
/Library/PostgreSQL/9.x/bin/pg_ctl restart -D /Library/PostgreSQL/9.x/data
Create a database user that is allowed to create databases. This user will be used when roundup connects to the database.
$ psql -U postgres postgres=# create user roundup; postgres=# alter user roundup with createdb;
Configure your Development Roundup Instance
Now enter the python-dev directory, and create the 'db' directory, as well as the db/backend_name file which decides which backend (i.e. database type) to use:
cd python-dev mkdir db echo postgresql > db/backend_name
Note: the db dir contains all the messages and files attached to issues, and it's removed and recreated every time you run roundup-admin init.
Copy python-dev/config.ini.template into config.ini and detectors/config.ini.template into detectors/config.ini:
cp config.ini.template config.ini cp detectors/config.ini.template detectors/config.ini
The former will let you configure your roundup instance; the latter, detectors such as irker, busybody, or tellteam.
In config.ini pay special attention to settings marked with NO DEFAULT.
config.ini.template already contains some useful default values for a development enviroment, for example
debug = debugmail.txt
at the end of the [mail] section saves outgoing mails on a local file; and
debug = on
at the end of the [web] section shows tracebacks in the browser instead of having them sent by email.
Initialize your roundup instance:
/opt/tracker-roundup/bin/roundup-admin -i <your python-dev directory> init
Provide a password for the 'admin' user when asked for.
Note: you have to provide the full path to your python-dev directory, otherwise the command will fail.
If you are getting the error:
"roundup.hyperdb.DatabaseError: FATAL: password authentication failed for user "roundup"" check again pg_hba.conf;
"roundup.configuration.OptionUnsetError: MAIL_DOMAIN is not set and has no default" set the option in python-dev/config.ini, [mail] section, domain;
Start your roundup instance:
/opt/tracker-roundup/bin/roundup-server -p 9999 python-dev=<your python-dev directory>
You will have to use this command every time you (re)start roundup, so it's easier to create an alias for it with:
alias start-roundup='/opt/tracker-roundup/bin/roundup-server -p 9999 python-dev=<your python-dev directory>'
It might be a good idea to create another alias for roundup admin too:
alias admin-roundup='/opt/tracker-roundup/bin/roundup-admin -i <your python-dev directory>'
then add them to your .bashrc (or ~/.profile in Mac OS X)
You should now be able to browse http://localhost:9999/python-dev/ and get a roundup instance that looks just like http://bugs.python.org, except for some missing new values for fields like Stages and Keywords. It's possible to replace initial_data.py with an updated version so that your fields will have values that match those currently present in the Python Tracker.
The IDs for values in a given field might be different from those in the Python Tracker. This should only be a problem if you try to import CSV files from one tracker into the other.
Create an user from User->register in the left sidebar. Note that if you have set debug = debugmail.txt as suggested above, you will have to open the file and copy the URL in the browser (make sure to replace =3D with = in the URL and to remove the trailing = where the url is broken at the end of the line). You can then use roundup-admin to set developer privileges for it:
/opt/tracker-roundup/bin/roundup-admin -i <your python-dev directory> Roundup 1.5.0 ready for input. Type "help" for help. roundup> display user3 ... roundup> set user3 roles=User,Developer,Coordinator roundup> commit roundup>
You can find your user id clicking on "Your Details" in the left sidebar and then looking at the address bar (it should show something similar to http://localhost:9999/python-dev/user3).
Rietveld Setup
The Roundup and Rietveld data are stored in the same Postgres database. After the Roundup tables have been created, add to python-dev/config.ini:
[django] secret_key = <the_secret_key>
To generate the secret key you can use an online django secret key generator, or just put an arbitrary string.
Before continuing make sure to check with "hg branch" that the branch is "bugs.python.org" on both the rietveld and the django-gae2django clones.
cd rietveld python manage.py syncdb cd .. PYTHONPATH=detectors/:/opt/tracker-roundup/lib/python2.7/site-packages/ python scripts/initrietveld
If you are upgrading from an older installation, review the revision history of rietveld/migration.
To run the Rietveld code, do
python manage.py runserver
This will run Rietveld by default on port 8000. Alternatively to the development server, you can also configure your web server to run rietveld.wsgi under the /review URL.
Rietveld patchset creation is an offline activity. If you have new patches in your tracker instance, run
scripts/addpatchsets
If you get errors like:
django.db.utils.DatabaseError: column codereview_repository.guid does not exist LINE 1: ...itory"."url", "codereview_repository"."owner_id", "coderevie...
you can try to remove the table(s) from the database and re-run syncdb and initrietveld:
$ psql -U roundup psql (9.1.4) Type "help" for help. roundup=> \d codereview_repository Table "public.codereview_repository" ... check here if some column is missing and if it is drop the table ... roundup=> drop table codereview_repository cascade; NOTICE: drop cascades to constraint codereview_branch_repo_id_fkey on table codereview_branch DROP TABLE roundup=> \q $ cd rietveld $ python manage.py syncdb Creating tables ... Creating table codereview_repository Installing custom SQL ... Installing indexes ... No fixtures found. $ cd .. $ PYTHONPATH=detectors/:/opt/tracker-roundup/lib/python2.7/site-packages/ python scripts/initrietveld
irker setup
irkerd is a daemon that sends notification on IRC whenever a message is added or an issue is created, and replaces the CIA.vc service.
You will need:
* irker: http://www.catb.org/esr/irker/
* the irc package: http://pypi.python.org/pypi/irc/
To install them you can use:
wget http://pypi.python.org/packages/source/i/irc/irc-5.0.1.zip unzip irc-5.0.1.zip cd irc-5.0.1/ sudo python setup.py install cd .. wget http://www.catb.org/~esr/irker/irker-1.17.tar.gz tar -zxvf irker-1.17.tar.gz
If you get the error ImportError: No module named hgtools.plugins during the sudo python setup.py install, you can try the command again and it should automagically work.
Make sure that you have listed the address of the channels that should receive the messages in detectors/config.ini, e.g.:
[irker] channels = irc://chat.freenode.net/python-dev
To start irker use:
cd irker-1.17/ ./irkerd &
You can also use ./irkerd -d3 to see debugging messages.
Resources for Tracker Development
Getting Help
Subscribe to and ask your question on the tracker-discuss mailing list. See http://mail.python.org/mailman/listinfo/tracker-discuss
Since the Python Tracker is a slightly modified version of Roundup, both Roundup's Documentation and issue tracker contain relevant information about how the Python Tracker works and problems one might find working with its code.
Using roundup-admin
To start roundup-admin interactively use:
/opt/tracker-roundup/bin/roundup-admin -i <your python-dev directory>
To get the list of users, issues, msgs, files, etc:
roundup> list user
To see info about a specific user, issue, msg, file, etc:
roundup> display userXXX
To change the role of an user:
roundup> set userXXX roles=User,Developer roundup> commit
Users have limited privileges and can edit just some of the fields in the issue page. Developers can edit all the fields and remove msgs and files. Coordinators can also mark/unmark messages as spam and change some of the settings of other users (e.g. if they submitted the contributor agreement).
To add back a message deleted accidentally:
roundup> set issueXXX messages=+YYY roundup> commit
The message will be added back in the right position. (This should be equivalent to b.p.o/issueXXX?@action=edit&@add@messages=YYY, but this doesn't seem to work.)
The Meta Tracker
A Meta Tracker is available for handling bugs and features requests for the Python Tracker.
Setting Up an Instance in a VirtualEnv
See TrackerDevelopmentVirtualEnv.
The Test Tracker
Public test instance of the Python tracker: http://bot.bio.br/python-dev/ (currently offine)
Instance that attempts to match the code used in http://bugs.python.org to allow testing, reproducing and verifying fixes for tracker bugs. Can also be used as a sandbox by users interested in learning about Developer, Coordinator or Admin tasks and features. Testing new content (e.g. Components or Statuses) is OK, but new features show be tested in the Experimental Tracker instead.
Currently the email system is disabled (redirected to file), so people interested in having an account there to test new features should email tracker-discuss to get one. Maintained by DanielDiniz.
The Experimental Tracker
Modified version of the Python tracker: http://bot.bio.br/python-dev-exp/ (currently offline)
It's an instance to showcase and test new features.
New features (2009-04-18):
Claim ('assign to self') and add/remove self as nosy buttons
"Email me" buttons for messages and issues, "Reply by email"
Currently the email system is disabled (redirected to file), so people interested in having an account there to test new features should email tracker-discuss to get one. Maintained by DanielDiniz.
There is also a list of additional DesiredTrackerFeatures for which no patches yet exist.
Getting Your Own Jython Tracker
To set up a local Jython tracker, please follow instructions for python-dev tracker with following exceptions:
There is no CIA.vc detector
- The repository for Jython tracker can be cloned with:
hg clone http://hg.python.org/tracker/jython/