Differences between revisions 20 and 82 (spanning 62 versions)
⇤ ← Revision 20 as of 2006-07-06 14:44:31 →
Size: 3882
Editor: w3cache
Comment:
← Revision 82 as of 2019-08-23 10:57:11 → ⇥
Size: 5122
Editor: SumanaHarihareswara
Comment: removing legacy PyPI content, updating links
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
This page documents the development of the Python Cheese Shop: This page is about development of the [[http://pypi.org|Python Package Index]] (formerly known as Cheeseshop):
Line 3: Line 3:
  http://cheeseshop.python.org  * More Info: CheeseShop
 * PyPI APIs: [[PyPIJSON|JSON]], [[PyPIXmlRpc|XMLRPC]]
Line 5: Line 6:
The PyPI (packages index, Cheese Shop web interface) code resides in SVN at: = Developing the Package Index =
Line 7: Line 8:
  https://svn.python.org/packages/ (trunk,features,branches) == PyPI.nextgen: ==
Line 9: Line 10:
Requests for new Trove categories: Currently, as of 2019-06-27, PyPI has undergone a complete rewrite from scratch, and as a result much of the information on this page is not up-to-date.
Line 11: Line 12:
  Python frameworks with plugins or packages that target the framework
  can get their own category. The category should only be added *after*
  such packages exist. Complimentary packages can link to each other
  from their descriptions, they do not need a category to link them
  together; only when packages are provided by different people does a
  category need to be created. To ask for a category email
  catalog-sig@python.org.		 The development moved from Mercurial (Python) to Git (C, shell). License changed from BSD-3 to Apache 2.0. Project code named 'warehouse' can be downloaded from:
Line 19: Line 14:
Patches and bug reports should be submitted to:  https://github.com/pypa/warehouse
Line 21: Line 16:
  http://sourceforge.net/tracker/?group_id=66150&atid=513503 == Testing Your Stuff Against PyPI ==
Line 23: Line 18:
Discussion about the Cheese Shop should be had on the Catalog SIG mailing list: If you need to test stuff against PyPI (registration, uploading, API activities) then please [[https://packaging.python.org/guides/using-testpypi/|use the alternative server, test.pypi.org]].
Line 25: Line 20:
  http://mail.python.org/mailman/listinfo/catalog-sig

== XML-RPC Interface ==

The Cheese Shop has an XML-RPC interface. See CheeseShopXmlRpc for details.
Line 33: Line 23:
 * [http://www.python.org/peps/pep-0345.html PEP for metadata 1.2] -- not finished and needs more catalog-sig discussion)
 * auto-generate download_urls for package uploads
 * command-line tool to query pypi and fetch entries
 * DOAP support		 Meta-to-do: file the issues below that are still unresolved in [[https://github.com/pypa/warehouse/issues|the GitHub issue tracker]], then remove this section. -- SumanaHarihareswara <<DateTime(2019-08-23T06:57:10-0400)>>

 * A dump of download counts.
 * A big structured dump of all package meta-data.
 * A link from package to RTFD.
 * [[http://www.python.org/peps/pep-0345.html|PEP for metadata 1.2]] -- not finished and needs more catalog-sig discussion)
Line 38: Line 30:
 * tooltips for field labels
Line 40: Line 31:
 * auto-compute [http://pycheesecake.org Cheesecake] scores
 * moderated user reviews and ratings
 * strip leading whitespace from description before ReST
 * "what's related" using classifiers and possibly user-supplied information		  * per-classifier "wiki" content to allow description and discussion around each classifier (perhaps what packages are available and how they relate to one another)
 * screenshot images (with thumbnailing and a "latest screenshot" on the front page?) - or perhaps icons instead of thumbnails for some packages?
Line 45: Line 34:
Something that's been requested, but needs much more thought and analysis to see whether it causes any problems: the ability to treat project names and versions as case-insensitive, while removing extraneous characters (as in pkg_resources.safe_name()) for
purposes both of searching and determining name uniqueness when registering.
Something that's been requested, but needs much more thought and analysis to see whether it causes any problems: the ability to treat project names and versions as case-insensitive, while removing extraneous characters (as in pkg_resources.safe_name()) for purposes both of searching and determining name uniqueness when registering.


=== Done ===


 * command-line tool to query pypi and fetch entries: [[http://pypi.python.org/pypi/yolk|yolk]]
Line 50: Line 46:
 * edit [http://www.python.org/peps/pep-0243.html PEP 243] to reflect reality
Line 52: Line 47:
== Development Environment Hints ==  * Edit [[http://www.python.org/peps/pep-0243.html|PEP 243]] to reflect reality. The interface is implemented in the distutils register and upload commands. This code is good enough for documentation, especially because it's the only implementation necessary.
 * moderated user reviews and ratings (this would require quite a lot of support from volunteers though)
Line 54: Line 50:
Before restoring database, "pypi" role must exists:
Line 56: Line 51:
  createuser pypi == Proposals ==
Line 58: Line 53:
These are notes so we can remember how to dump / restore the packages database:
Line 60: Line 54:
  pg_dump -Fc -b packages >packages-20060706.dump  * [[http://wiki.python.org/moin/EnhancedPyPI|EnhancedPyPI]] Enhance multiple package index servers support in Distutils.
Line 62: Line 56:
  createdb packages
Line 64: Line 57:
  pg_restore -O -d packages -Fc packages-20060706.dump == Previous PyPI version ==
Line 66: Line 59:
Ask RichardJones if you need a database dump. The legacy version of PyPI is the code that was running on http://pypi.python.org for many years, till mid-2018. [[https://lwn.net/Articles/751458/|This LWN article goes into the history.]] The information below should help you get around the code.
Line 68: Line 61:
PageTemplates are obtained from:  .
 The PyPI code was hosted under the Python Packaging Authority project: https://bitbucket.org/pypa/pypi and is now on GItHub: https://github.com/pypa/pypi-legacy
Line 70: Line 64:
  svn co svn://svn.zope.org/repos/main/zope.pagetemplate/trunk zope. pagetemplate  .
 Bug and patch tracker https://github.com/pypa/pypi-legacy/issues
Line 72: Line 67:
  svn co svn://svn.zope.org/repos/main/zope.tal/trunk zope.tal  .
 [[http://mail.python.org/mailman/listinfo/distutils-sig|Mailing List]] ([[http://dir.gmane.org/gmane.comp.python.distutils|Gmane]] web interface)
Line 74: Line 70:
  svn co svn://svn.zope.org/repos/main/zope.interface/trunk zope.interface  .
 API that is used by easy_install http://peak.telecommunity.com/DevCenter/EasyInstall#package-index-api
Line 76: Line 73:
  svn co svn://svn.zope.org/repos/main/zope.i18nmessageid/trunk zope.i18nmessageid  * [[PyPIOAuth]] - authentication library for Google and Launchpad logins
Line 78: Line 75:
  svn co svn://svn.zope.org/repos/main/zope.tales/trunk zope.tales
Line 80: Line 76:
We also need to install ez_setup and docutils. == Legacy PyPI architecture and endpoints ==
Line 82: Line 78:
Then go ahead and edit pypi.ini and this line in pypi.cgi:
Line 84: Line 79:
  cfg = config.Config('/tmp/pypi.ini', 'webui') PyPI is a WSGI application that can be executed standalone using `python pypi.wsgi` command if all requirements are met. `pypi.wsgi` contains usual WSGI wrapper code and delegates request processing to `WebUI.run()` method from `webui.py`. This method just opens DB and handles exceptions, actual request processing is done in `WebUI.inner_run()`. This method analyzes URL endpoint and executes appropriate handler. As of 2011-04, the rules to match endpoints to handlers are the following:
Line 86: Line 81:
if your config.ini isn't in /tmp/pypi.ini. You can leave it as 'config.ini' if it's in the same directory as pypi.py.
Line 88: Line 82:
If you wish to test functionality that requires login, then you will need to configure your apache to pass through basic auth. You do this by adding lines '''similar''' to the following to your apache2 conf file:
Line 90: Line 83:
{{{
  RewriteCond %{HTTP:Authorization} ^(.+)$		 ||/simple ||WebUI.run_simple() ||dump all package names on single html page ||
||/simple/(.+)/ ||WebUI.run_simple() ||dump all links for a package in html list ||
||/serversig/(.+)/ ||.run_simple_sign() ||save as above, but signed by server ||
||/mirrors ||.mirrors() ||display static page with a list of mirrors ||
||/daytime ||.daytime() ||display current server time ||
Line 93: Line 89:
  RewriteRule ^/cgi-bin/pypi.cgi(.*) /Library/WebServer/CGI-Executables/pypi.cgi$1 [e=HTTP_CGI_AUTHORIZATION:%1,t=application/x-httpd-cgi,l]
}}}
Line 96: Line 90:
If this doesn't work, then you can use these lines to debug:
Line 98: Line 91:
{{{
  RewriteLog /var/log/httpd/rewrite.log
Line 101: Line 92:
  RewriteLogLevel 9
}}}
...


XML-RPC requests are detected by CONTENT_TYPE=`text/xml` variable in CGI environment and processed by `rpc.RequestHandler().__call__()`. List of XML-RPC "endpoints" is available on [[PyPIXmlRpc]] page.



== Legacy PyPI Development Environment Hints ==

Removed (visible in [[https://wiki.python.org/moin/CheeseShopDev?action=recall&rev=81|page history]]) because developing and running legacy PyPI is deprecated. -- SumanaHarihareswara <<DateTime(2019-08-23T06:57:10-0400)>>

This page is about development of the Python Package Index (formerly known as Cheeseshop):

Developing the Package Index

PyPI.nextgen:

Currently, as of 2019-06-27, PyPI has undergone a complete rewrite from scratch, and as a result much of the information on this page is not up-to-date.

The development moved from Mercurial (Python) to Git (C, shell). License changed from BSD-3 to Apache 2.0. Project code named 'warehouse' can be downloaded from:

Testing Your Stuff Against PyPI

If you need to test stuff against PyPI (registration, uploading, API activities) then please use the alternative server, test.pypi.org.

TO-DO list

Meta-to-do: file the issues below that are still unresolved in the GitHub issue tracker, then remove this section. -- SumanaHarihareswara 2019-08-23 10:57:10

  • A dump of download counts.
  • A big structured dump of all package meta-data.
  • A link from package to RTFD.

  • PEP for metadata 1.2 -- not finished and needs more catalog-sig discussion)

  • documented procedures for "taking over" entries should the original owner of the entry go away (and any required system support)
  • change notification emails
  • per-classifier "wiki" content to allow description and discussion around each classifier (perhaps what packages are available and how they relate to one another)
  • screenshot images (with thumbnailing and a "latest screenshot" on the front page?) - or perhaps icons instead of thumbnails for some packages?

Something that's been requested, but needs much more thought and analysis to see whether it causes any problems: the ability to treat project names and versions as case-insensitive, while removing extraneous characters (as in pkg_resources.safe_name()) for purposes both of searching and determining name uniqueness when registering.

Done

  • command-line tool to query pypi and fetch entries: yolk

Not Going TO-DO

  • Edit PEP 243 to reflect reality. The interface is implemented in the distutils register and upload commands. This code is good enough for documentation, especially because it's the only implementation necessary.

  • moderated user reviews and ratings (this would require quite a lot of support from volunteers though)

Proposals

  • EnhancedPyPI Enhance multiple package index servers support in Distutils.

Previous PyPI version

The legacy version of PyPI is the code that was running on http://pypi.python.org for many years, till mid-2018. This LWN article goes into the history. The information below should help you get around the code.

Legacy PyPI architecture and endpoints

PyPI is a WSGI application that can be executed standalone using python pypi.wsgi command if all requirements are met. pypi.wsgi contains usual WSGI wrapper code and delegates request processing to WebUI.run() method from webui.py. This method just opens DB and handles exceptions, actual request processing is done in WebUI.inner_run(). This method analyzes URL endpoint and executes appropriate handler. As of 2011-04, the rules to match endpoints to handlers are the following:

/simple

WebUI.run_simple()

dump all package names on single html page

/simple/(.+)/

WebUI.run_simple()

dump all links for a package in html list

/serversig/(.+)/

.run_simple_sign()

save as above, but signed by server

/mirrors

.mirrors()

display static page with a list of mirrors

/daytime

.daytime()

display current server time

...

XML-RPC requests are detected by CONTENT_TYPE=text/xml variable in CGI environment and processed by rpc.RequestHandler().__call__(). List of XML-RPC "endpoints" is available on PyPIXmlRpc page.

Legacy PyPI Development Environment Hints

Removed (visible in page history) because developing and running legacy PyPI is deprecated. -- SumanaHarihareswara 2019-08-23 10:57:10

CheeseShopDev (last edited 2019-08-23 10:57:11 by SumanaHarihareswara)

Unable to edit the page? See the FrontPage for instructions.