This page documents the development of the Python Package Index (formerly known as Cheeseshop):

http://pypi.python.org

More about PyPI

CheeseShop page has all information not directly related to PyPI development.

Requests for new Trove categories

Python frameworks with plugins or packages that target the framework can get their own category. The category should only be added *after* such packages exist. Complementary packages can link to each other from their descriptions, they do not need a category to link them together; only when packages are provided by different people does a category need to be created. To ask for a category email catalog-sig@python.org .

Developing the Package Index

The PyPI code resides in Subversion repository with anonymous and developer's access
- https://svn.python.org/packages/ (trunk,features,branches)
- svn+ssh://svn.python.org/data/repos/packages/trunk/pypi
Bug and patch tracker http://sourceforge.net/tracker/?group_id=66150&atid=513503
Discussions on Catalog SIG http://mail.python.org/mailman/listinfo/catalog-sig
API that is used by easy_install http://peak.telecommunity.com/DevCenter/EasyInstall#package-index-api

PyPI architecture and endpoints

PyPI is a WSGI application that can be executed standalone using python pypi.wsgi command if all requirements are met. pypi.wsgi contains usual WSGI wrapper code and delegates request processing to WebUI.run() method from webui.py. This method just opens DB and handles exceptions, actual request processing is done in WebUI.inner_run(). This method analyzes URL endpoint and executes appropriate handler. As of 2011-03, the rules to match endpoints to handlers are the following:

/simple	WebUI.run_simple()	dump all package names on single html page
/simple/(.+)/	WebUI.run_simple()	dump all links for a package in html list
/serversig/(.+)/	.run_simple_sign()	save as above, but signed by server

...

TO-DO list

PEP for metadata 1.2 -- not finished and needs more catalog-sig discussion)
documented procedures for "taking over" entries should the original owner of the entry go away (and any required system support)
tooltips for field labels
change notification emails
moderated user reviews and ratings (this would require quite a lot of support from volunteers though)
per-classifier "wiki" content to allow description and discussion around each classifier (perhaps what packages are available and how they relate to one another)
screenshot images (with thumbnailing and a "latest screenshot" on the front page?) - or perhaps icons instead of thumbnails for some packages?

Something that's been requested, but needs much more thought and analysis to see whether it causes any problems: the ability to treat project names and versions as case-insensitive, while removing extraneous characters (as in pkg_resources.safe_name()) for purposes both of searching and determining name uniqueness when registering.

Done

command-line tool to query pypi and fetch entries: yolk

Not Going TO-DO

Edit PEP 243 to reflect reality. The interface is implemented in the distutils register and upload commands. This code is good enough for documentation, especially because it's the only implementation necessary.

Proposals

EnhancedPyPI Enhance multiple package index servers support in Distutils.

Development Environment Hints

PyPI uses postgresql 8.3 as a database, with a roll it yourself web framework based on different python modules. It uses apache2 as the web server.

It can run using wsgi, cgi, fcgi and mod_python.

Before restoring database, "pypi" role must exists:

  createuser pypi

These are notes so we can remember how to dump / restore the packages database:

  pg_dump -Fc -b packages >packages-20060706.dump
  createdb packages
  pg_restore -O -d packages -Fc packages-20060706.dump

Ask RichardJones if you need a database dump.

PageTemplates are obtained from:

  svn co svn://svn.zope.org/repos/main/zope.pagetemplate/trunk zope.pagetemplate
  svn co svn://svn.zope.org/repos/main/zope.tal/trunk zope.tal
  svn co svn://svn.zope.org/repos/main/zope.interface/trunk zope.interface
  svn co svn://svn.zope.org/repos/main/zope.i18nmessageid/trunk zope.i18nmessageid
  svn co svn://svn.zope.org/repos/main/zope.tales/trunk zope.tales

We also need to install ez_setup, celementtree, docutils, psycopg2, and M2Crypto.

# For debian, this should get you most of the dependencies. apt-get install zope3 python-celementtree python-psycopg

Then go ahead and edit pypi.ini and this line in pypi.cgi:

  cfg = config.Config('/tmp/pypi.ini', 'webui')

if your config.ini isn't in /tmp/pypi.ini. You can leave it as 'config.ini' if it's in the same directory as pypi.py.

You will need to add cheesecake_password=yourpasshere into the config.ini in the webui section.

To integrate it with Apache, we recommend to use WSGI through mod_wsgi. Your configuration should look like this:

   WSGIDaemonProcess pypi display-name=wsgi-pypi python-path=/usr/lib/zope2.9/lib/python processes=4 threads=1
   WSGIProcessGroup pypi
   WSGIPassAuthorization On
   WSGIScriptAlias /pypi /data/pypi/src/pypi/pypi.wsgi
   WSGIScriptAlias /simple /data/pypi/src/pypi/pypi.wsgi

CategoryDevelopmentProcess

Page

User