19421
Comment: Add a section that explains how to update to PostgreSQL 9.1
|
20602
Explain how to disable spambayes on the local instance.
|
Deletions are marked like this. | Additions are marked like this. |
Line 168: | Line 168: |
If you do intend to use it, you will need to also [[#SpamBayesSetup | install SpamBayes proper and configure it appropriately]]. Otherwise, a [[#SpamBayesdummyconfiguration |dummy configuration]] should work. | If you do intend to use it, you will need to also [[#SpamBayesSetup | install SpamBayes proper and configure it appropriately]]. Otherwise, a [[#SpamBayes_dummy_configuration |dummy configuration]] should work. |
Line 342: | Line 342: |
Another workaround is to: 1. disable the {{{sb_is_spam}}} function in extensions/spambayes.py adding a {{{return False}}}: {{{ def sb_is_spam(obj): return False # <- add this cutoff_score = float(obj._db.config.detectors['SPAMBAYES_SPAM_CUTOFF']) }}} 2. disable the {{{may_view_spam.__call__}}} method in schema.py adding a {{{return True}}}: {{{ def __call__(self, db, userid, itemid): return True # <- add this cutoff_score = float(db.config.detectors['SPAMBAYES_SPAM_CUTOFF']) }}} 3. disable the {{{is_spam}}} function in detectors/sendmail.py adding a {{{return False}}}: {{{ def is_spam(db, msgid): """Return true if message has a spambayes score above db.config.detectors['SPAMBAYES_SPAM_CUTOFF']. Also return true if msgid is None, which happens when there are no messages (i.e., a property-only change)""" return False # <- add this if not msgid: }}} '''Note:''' this is just a workaround, and these changes will be visible with {{{svn diff}}}, so pay attention to not commit them accidentally. A better fix would be to add a config option to enable/disable spambayes. |
This page describes how to enhance Python bug tracker instance at http://bugs.python.org/ To find some stuff to work on, look at the issues in meta tracker and DesiredTrackerFeatures page.
Contents
- Getting Your Own python-dev Tracker
- Resources for Tracker Development
- Getting Your Own jython Tracker
Getting Your Own python-dev Tracker
bugs.python.org is a Roundup tracker instance named python-dev. Below is a HowTo to get the similar setup for development on your machine.
Requirements
- Roundup, from the vendor branch used for python-dev (see below).
- PostgreSQL for use as backend database.
- Installation of psycopg.
BeautifulSoup and M2Crypto
- A checkout of the spambayes_integration (instructions are given below).
- A checkout of the python-dev instance from python's subversion repository (instructions are given below).
Installing Roundup
For python-dev, we use a slightly modified version of Roundup which has some features that are not yet in the core Roundup. To keep track of the changes, we keep a vendor branch of roundup.
Check out /tracker from the python subversion repository as documented on http://python.org/dev/tools/. For example, here's how to do read-only (if you don't have commit privileges) and read/write (if you have them) checkouts:
svn co http://svn.python.org/projects/tracker svn co svn+ssh://pythondev@svn.python.org/tracker
The tracker directory contains the roundup source and the instances (e.g. python-dev, spambayes, jython, meta tracker).
cd in the tracker/roundup-src dir and install Roundup using setup.py (note: you might need privileges to run this command):
cd roundup-src sudo python setup.py install --prefix /opt/tracker-roundup/
Install and configure PostgreSQL
For the python-dev tracker instance we use PostgreSQL as backend database. Roundup has support for many different databases for backend (anydbm, metakit, MySQL, PostgreSQL, SQLite...), but for the python-dev instance, PostgreSQL was selected as it has the best performance for large installations. Also, for performance reasons, there is at least one place in the python-dev instance that has a hardcoded dependency on the backend being an SQL-based one, so you do need PostgreSQL even for development work (another SQL-based backend might work, but this has not been tested). I recommend that you run postgreSQL to make your development environment behave exactly like the environment the real tracker runs in.
Details on how to install PostgreSQL is outside the scope of this Wiki Page (sudo apt-get install postgresql should be enough). Both version 7.4 and 8.1 of PostgreSQL has been tested without any known problems. It has been reported that 8.2, 8.3, 8.4, and 9.1 work well.
Access configuration of postgresql can be tricky. You need to make sure that the user you're running roundup as (your unix/windows user) has access to the postgresql database you intend to use for roundup, as the postgresql user you've configured in roundup's configuration.
For a development environment with no demands on security, the easiest way to configure this is probably to allow all users to connect as any database user as long as the request originates from the local host. This can be accomplished editing pg_hba.conf (usually found in /etc/postgresql/8.x/main/ -- you might need privileges). The end of the file should look like this (the four changed lines are preceded by ##):
# Database administrative login by UNIX sockets ##local all postgres ident local all postgres trust # TYPE DATABASE USER CIDR-ADDRESS METHOD # "local" is for Unix domain socket connections only ##local all all ident local all all trust # IPv4 local connections: ##host all all 127.0.0.1/32 md5 host all all 127.0.0.1/32 trust # IPv6 local connections: ##host all all ::1/128 md5 host all all ::1/128 trust
PLEASE NOTE: This is not a secure configuration on a multi-user machine.
After the change in pg_hba.conf, reload your postgreSQL database to make sure it knows about your new access configuration. Something similar to
sudo /etc/init.d/postgresql reload
should do the trick on Unix-like platforms.
Create a database user that is allowed to create databases. This user will be used when roundup connects to the database.
$ psql -U postgres postgres=# create user roundup; postgres=# alter user roundup with createdb;
Upgrading PostgreSQL from 8.4 to 9.1
If you have a tracker instance running on PostgreSQL 8.4 and you want to update it to 9.1, follow these steps:
1. Install postgresql-9.1;
2. Remove the new default cluster using sudo pg_dropcluster --stop 9.1 main;
3. Update the old clusted with sudo pg_upgradecluster 8.4 main;
4. If you get the error:
The PostgreSQL server failed to start. Please check the log output: FATAL: could not create shared memory segment: Invalid argument DETAIL: Failed system call was shmget(key=5432001, size=34922496, 03600). HINT: This error usually means that PostgreSQL's request for a shared memory segment exceeded your kernel's SHMMAX parameter. You can either reduce the request size or reconfigure the kernel with larger SHMMAX. To reduce the request size (currently 34922496 bytes), reduce PostgreSQL's shared memory usage, perhaps by reducing shared_buffers or max_connections. If the request size is already small, it's possible that it is less than your kernel's SHMMIN parameter, in which case raising the request size or reconfiguring SHMMIN is called for.
check the value of shmmax with sysctl -a | grep -i shm and update it with sudo sysctl kernel.shmmax=34922496
5. If everything works with 9.1 you can then remove 8.4;
Installing psycopg or psycopg2
You need to install psycopg or psycopg2:
sudo apt-get install python-psycopg2
In case that doesn't work, you can download the latest .tar.gz from http://www.initd.org, extract it and install it:
tar xzf psycopg2-2.3.2.tar.gz cd psycopg2-2.3.2/ python setup.py build sudo python setup.py install
If during the python setup.py build you get:
"fatal error: Python.h: No such file or directory", you need to sudo apt-get install python-dev;
"fatal error: libpq-fe.h: No such file or directory", you need to sudo apt-get install libpq-dev;
If you are having other problems or installing a different version check psycopg2-2.3.2/INSTALL.
Checkout spambayes_integration
Note: If you followed the instructions and checked out http://svn.python.org/projects/tracker you can skip this step.
The SpamBayes detector is optional, but you must still check out the spambayes_integration instance.
You can get a checkout of spambayes_integration with:
svn co http://svn.python.org/projects/tracker/instances/spambayes_integration
If you do intend to use it, you will need to also install SpamBayes proper and configure it appropriately. Otherwise, a dummy configuration should work.
Checkout the python-dev instance
Note: If you followed the instructions and checked out http://svn.python.org/projects/tracker you can skip this step.
The python-dev tracker instance is version-controlled with subversion, and resides in the python subversion repository.
svn co http://svn.python.org/projects/tracker/instances/python-dev
Keep in mind that the python-dev instance has relative symlinks to ../../spambayes_integration/[...], so if you did separate checkouts and your directory layout is different you have to update these links:
extensions/spambayes.py -> ../../spambayes_integration/extensions/spambayes.py detectors/spambayes.py -> ../../spambayes_integration/detectors/spambayes.py
Configure your Development Roundup Instance
Now enter the python-dev directory, and create the 'db' directory (not in svn as it's just to much fuzz to keep it there - every time you run roundup-admin init it is removed and recreated, confusing svn who wants to keep its .svn directory intact), as well as the db/backend_name file which decides which backend (i.e. database type) to use:
cd instances/python-dev mkdir db echo postgresql > db/backend_name
Copy python-dev/config.ini.template into config.ini, and modify it according to your site. Pay special attention to settings marked with NO DEFAULT. However, if you've configure your PostgreSQL as above, most settings should be OK.
If you need to do development on the busybody or tellteam detectors, copy detectors/config.ini.template into detectors/config.ini, and adjust the e-mail addresses accordingly. The CIA.vc detector sends emails about issue changes and is on by default. Disable it by adding a return to its init() function (detectors/cia.py):
Initialize your roundup instance:
/opt/tracker-roundup/bin/roundup-admin -i <your python-dev directory> init
Provide a password for the 'admin' user when asked for.
Note: you have to provide the full path to your python-dev directory, otherwise the command will fail.
If you are getting the error:
"roundup.hyperdb.DatabaseError: FATAL: password authentication failed for user "roundup"" check again pg_hba.conf;
"roundup.configuration.OptionUnsetError: MAIL_DOMAIN is not set and has no default" set the option in python-dev/config.ini, [mail] section, domain;
Note: it is possible to set the [mail] domain to "NO DEFAULT" to make the init work. However Roundup needs a mail server to send mails so you'll need to configure at the least the admin mail later on. It might be possible to have Roundup work with no mails at all, but the right configuration is still unknown (feel free to experiment with it!).
Start your roundup instance:
/opt/tracker-roundup/bin/roundup-server -p 9999 python-dev=<your python-dev directory>
You will have to use this command every time you (re)start roundup, so it's easier to create an alias for it with:
alias start-roundup='/opt/tracker-roundup/bin/roundup-server -p 9999 python-dev=<your python-dev directory>'
and add it to your .bashrc.
You should now be able to browse http://localhost:9999/python-dev/ and get a roundup instance that looks just like http://bugs.python.org, except for some missing new values for fields like Stages and Keywords. It's possible to replace initial_data.py with an updated version so that your fields will have values that match those currently present in the Python Tracker.
The IDs for values in a given field might be different from those in the Python Tracker. This should only be a problem if you try to import CSV files from one tracker into the other.
Create an user from User->register in the left sidebar. You can then use roundup-admin to set developer privileges for it:
/opt/tracker-roundup/bin/roundup-admin -i <your python-dev directory> Roundup 1.4.10 ready for input. Type "help" for help. roundup> display user8 ... roundup> set user8 roles=Developer roundup> commit roundup>
You can find your user id clicking on "Your Details" in the left sidebar and then looking at the address bar (it should show something similar to http://localhost:9999/python-dev/user8).
SpamBayes Setup
If you intend to use the SpamBayes detector to weed out spam from your tracker you will need to install it and start an instance of its core_server application. Briefly:
Download and install at least version 1.1a4 of the SpamBayes source from SourceForge. If you are running on Windows do not use the binary installer!
- Establish a basic configuration, e.g.:
Devote a directory to it, e.g.:
SBDIR=/usr/local/spambayes/core_server # or whatever... mkdir -p $SBDIR
Create an INI file:
cd $SBDIR cat > bayescustomize.ini <<EOF [globals] verbose:False [Headers] include_evidence:True include_score:True [Tokenizer] record_header_absence:True summarize_email_prefixes:True summarize_email_suffixes:True mine_received_headers:True x-pick_apart_urls:True x-fancy_url_recognition:False x-lookup_ip:True lookup_ip_cache:$SBDIR/dnscache.pck max_image_size:100000 crack_image_cache:$SBDIR/imagecache.pck crack_images:True image_size:True ocr_engine:gocr [Classifier] use_bigrams:False [Categorization] ham_cutoff:0.2 spam_cutoff:0.85 [Storage] persistent_storage_file:$SBDIR/hammie.db persistent_use_database:pickle messageinfo_storage_file:$SBDIR/messageinfo.fs [html_ui] display_score:True EOF
Finally, run it:
BAYESCUSTOMIZE=$SBDIR/bayescustomize.ini core_server.py -m XMLRPCPlugin
If you are seeing image spam (this seems not to be a problem with wiki spam - yet), download and install GOCR.
Common Problems
SpamBayes dummy configuration
Incomplete/Missing SpamBayes setup:: In your detectors/config.ini file set spambayes_uri to a syntactically valid, but nonexistent URI, e.g.:
spambayes_uri = "http://localhost:9999/nobodyhere"
Another workaround is to:
1. disable the sb_is_spam function in extensions/spambayes.py adding a return False:
def sb_is_spam(obj): return False # <- add this cutoff_score = float(obj._db.config.detectors['SPAMBAYES_SPAM_CUTOFF'])
2. disable the may_view_spam.__call__ method in schema.py adding a return True:
def __call__(self, db, userid, itemid): return True # <- add this cutoff_score = float(db.config.detectors['SPAMBAYES_SPAM_CUTOFF'])
3. disable the is_spam function in detectors/sendmail.py adding a return False:
def is_spam(db, msgid): """Return true if message has a spambayes score above db.config.detectors['SPAMBAYES_SPAM_CUTOFF']. Also return true if msgid is None, which happens when there are no messages (i.e., a property-only change)""" return False # <- add this if not msgid:
Note: this is just a workaround, and these changes will be visible with svn diff, so pay attention to not commit them accidentally. A better fix would be to add a config option to enable/disable spambayes.
Check other available workarounds.
Misbehaving detectors
Problematic detectors may be disabled by returning from their init() function, as was the case of detectors/audit2to3.py (no longer present). Note that this would be a temporary solution before a graceful fix became available. Reporting issues with local instances is the best way help solving the problem.
Resources for Tracker Development
Getting Help
Subscribe to and ask your question on the tracker-discuss mailing list. See http://mail.python.org/mailman/listinfo/tracker-discuss
Since the Python Tracker is a slightly modified version of Roundup, both Roundup's Documentation and issue tracker contain relevant information about how the Python Tracker works and problems one might find working with its code.
Using roundup-admin
To start roundup-admin interactively use:
/opt/tracker-roundup/bin/roundup-admin -i <your python-dev directory>
To get the list of users, issues, msgs, files, etc:
roundup> list user
To see info about a specific user, issue, msg, file, etc:
roundup> display userXXX
To change the role of an user:
roundup> set userXXX roles=User,Developer roundup> commit
Users have limited privileges and can edit just some of the fields in the issue page. Developers can edit all the fields and remove msgs and files. Coordinators can also mark/unmark messages as spam and change some of the settings of other users (e.g. if they submitted the contributor agreement).
To add back a message deleted accidentally:
roundup> set issueXXX messages=+YYY roundup> commit
The message will be added back in the right position. (This should be equivalent to b.p.o/issueXXX?@action=edit&@add@messages=YYY, but this doesn't seem to work.)
The Meta Tracker
A Meta Tracker is available for handling bugs and features requests for the Python Tracker.
Learn more About the Importer(s)
http://svn.python.org/projects/tracker/importer/README.rst has some information about the importer(s) written to convert sourceforge data into roundup data.
The Test Tracker
Public test instance of the Python tracker: http://bot.bio.br/python-dev/ (currently offine)
Instance that attempts to match the code used in http://bugs.python.org to allow testing, reproducing and verifying fixes for tracker bugs. Can also be used as a sandbox by users interested in learning about Developer, Coordinator or Admin tasks and features. Testing new content (e.g. Components or Statuses) is OK, but new features show be tested in the Experimental Tracker instead.
Currently the email system is disabled (redirected to file), so people interested in having an account there to test new features should email tracker-discuss to get one. Maintained by DanielDiniz.
The Experimental Tracker
Modified version of the Python tracker: http://bot.bio.br/python-dev-exp/ (currently offline)
It's an instance to showcase and test new features.
New features (2009-04-18):
Claim ('assign to self') and add/remove self as nosy buttons
"Email me" buttons for messages and issues, "Reply by email"
Currently the email system is disabled (redirected to file), so people interested in having an account there to test new features should email tracker-discuss to get one. Maintained by DanielDiniz.
There is also a list of additional DesiredTrackerFeatures for which no patches yet exist.
Getting Your Own jython Tracker
To set up a local jython tracker, please follow instructions for python-dev tracker with following exceptions:
There is no need for BeautifulSoup and M2Crypto
There is no CIA.vc detector
- The code for jython tracker can be checked out using the command:
svn co http://svn.python.org/projects/tracker/instances/jython