|
Size: 5057
Comment: Cheese Shop -> Package Index
|
← Revision 82 as of 2019-08-23 10:57:11 ⇥
Size: 5122
Comment: removing legacy PyPI content, updating links
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 1: | Line 1: |
| This page documents the development of the Python Package Index (formerly known as Cheeseshop): | This page is about development of the [[http://pypi.org|Python Package Index]] (formerly known as Cheeseshop): |
| Line 3: | Line 3: |
| . http://pypi.python.org = How to submit packages = This page isn't about submitting packages. You want to see CheeseShopTutorial instead. = Requests for new Trove categories = Python frameworks with plugins or packages that target the framework can get their own category. The category should only be added *after* such packages exist. Complimentary packages can link to each other from their descriptions, they do not need a category to link them together; only when packages are provided by different people does a category need to be created. To ask for a category email catalog-sig@python.org . = XML-RPC Interface = The Package Index has an XML-RPC interface which may be used by any 3rd-party application developer. See CheeseShopXmlRpc for details. |
* More Info: CheeseShop * PyPI APIs: [[PyPIJSON|JSON]], [[PyPIXmlRpc|XMLRPC]] |
| Line 14: | Line 7: |
| This information is for developers working on the PyPI code. The PyPI code resides in SVN at: | |
| Line 16: | Line 8: |
| . https://svn.python.org/packages/ (trunk,features,branches) Developers with access may use this URL: |
== PyPI.nextgen: == |
| Line 19: | Line 10: |
| . svn+ssh://svn.python.org/data/repos/packages/trunk/pypi Patches and bug reports should be submitted to: |
Currently, as of 2019-06-27, PyPI has undergone a complete rewrite from scratch, and as a result much of the information on this page is not up-to-date. |
| Line 22: | Line 12: |
| . http://sourceforge.net/tracker/?group_id=66150&atid=513503 Discussion about the Package Index should be had on the Catalog SIG mailing list: |
The development moved from Mercurial (Python) to Git (C, shell). License changed from BSD-3 to Apache 2.0. Project code named 'warehouse' can be downloaded from: |
| Line 25: | Line 14: |
| . http://mail.python.org/mailman/listinfo/catalog-sig Package Index API used by easy_install: |
https://github.com/pypa/warehouse |
| Line 28: | Line 16: |
| . http://peak.telecommunity.com/DevCenter/EasyInstall#package-index-api | == Testing Your Stuff Against PyPI == If you need to test stuff against PyPI (registration, uploading, API activities) then please [[https://packaging.python.org/guides/using-testpypi/|use the alternative server, test.pypi.org]]. |
| Line 30: | Line 22: |
| * [http://www.python.org/peps/pep-0345.html PEP for metadata 1.2] -- not finished and needs more catalog-sig discussion) * command-line tool to query pypi and fetch entries |
Meta-to-do: file the issues below that are still unresolved in [[https://github.com/pypa/warehouse/issues|the GitHub issue tracker]], then remove this section. -- SumanaHarihareswara <<DateTime(2019-08-23T06:57:10-0400)>> * A dump of download counts. * A big structured dump of all package meta-data. * A link from package to RTFD. * [[http://www.python.org/peps/pep-0345.html|PEP for metadata 1.2]] -- not finished and needs more catalog-sig discussion) |
| Line 33: | Line 30: |
| * tooltips for field labels | |
| Line 35: | Line 31: |
| * moderated user reviews and ratings (this would require quite a lot of support from volunteers though) | |
| Line 38: | Line 33: |
| Something that's been requested, but needs much more thought and analysis to see whether it causes any problems: the ability to treat project names and versions as case-insensitive, while removing extraneous characters (as in pkg_resources.safe_name()) for purposes both of searching and determining name uniqueness when registering. | Something that's been requested, but needs much more thought and analysis to see whether it causes any problems: the ability to treat project names and versions as case-insensitive, while removing extraneous characters (as in pkg_resources.safe_name()) for purposes both of searching and determining name uniqueness when registering. === Done === * command-line tool to query pypi and fetch entries: [[http://pypi.python.org/pypi/yolk|yolk]] |
| Line 41: | Line 45: |
| * Edit [http://www.python.org/peps/pep-0243.html PEP 243] to reflect reality. The interface is implemented in the distutils register and upload commands. This code is good enough for documentation, especially because it's the only implementation necessary. == Development Environment Hints == PyPI uses postgresql as a database, with a roll it yourself web framework based on different python modules. It uses apache2 as the web server. |
|
| Line 45: | Line 46: |
| It can run using cgi, fcgi and mod_python. | |
| Line 47: | Line 47: |
| Before restoring database, "pypi" role must exists: | * Edit [[http://www.python.org/peps/pep-0243.html|PEP 243]] to reflect reality. The interface is implemented in the distutils register and upload commands. This code is good enough for documentation, especially because it's the only implementation necessary. * moderated user reviews and ratings (this would require quite a lot of support from volunteers though) |
| Line 49: | Line 50: |
| {{{ createuser pypi }}} These are notes so we can remember how to dump / restore the packages database: |
|
| Line 54: | Line 51: |
| {{{ pg_dump -Fc -b packages >packages-20060706.dump createdb packages pg_restore -O -d packages -Fc packages-20060706.dump }}} Ask RichardJones if you need a database dump. |
== Proposals == |
| Line 61: | Line 53: |
| PageTemplates are obtained from: | |
| Line 63: | Line 54: |
| {{{ svn co svn://svn.zope.org/repos/main/zope.pagetemplate/trunk zope.pagetemplate svn co svn://svn.zope.org/repos/main/zope.tal/trunk zope.tal svn co svn://svn.zope.org/repos/main/zope.interface/trunk zope.interface svn co svn://svn.zope.org/repos/main/zope.i18nmessageid/trunk zope.i18nmessageid svn co svn://svn.zope.org/repos/main/zope.tales/trunk zope.tales }}} We also need to install ez_setup, celementtree and docutils. |
* [[http://wiki.python.org/moin/EnhancedPyPI|EnhancedPyPI]] Enhance multiple package index servers support in Distutils. |
| Line 72: | Line 56: |
| # For debian, this should get you most of the dependencies. apt-get install zope3 python-celementtree python-psycopg | |
| Line 74: | Line 57: |
| Then go ahead and edit pypi.ini and this line in pypi.cgi: | == Previous PyPI version == |
| Line 76: | Line 59: |
| {{{ cfg = config.Config('/tmp/pypi.ini', 'webui') }}} if your config.ini isn't in /tmp/pypi.ini. You can leave it as 'config.ini' if it's in the same directory as pypi.py. |
The legacy version of PyPI is the code that was running on http://pypi.python.org for many years, till mid-2018. [[https://lwn.net/Articles/751458/|This LWN article goes into the history.]] The information below should help you get around the code. |
| Line 81: | Line 61: |
| You will need to add cheesecake_password=yourpasshere into the config.ini in the webui section. | . The PyPI code was hosted under the Python Packaging Authority project: https://bitbucket.org/pypa/pypi and is now on GItHub: https://github.com/pypa/pypi-legacy |
| Line 83: | Line 64: |
| If you wish to test functionality that requires login, then you will need to configure your apache to pass through basic auth. You do this by adding lines '''similar''' to the following to your apache2 conf file: | . Bug and patch tracker https://github.com/pypa/pypi-legacy/issues |
| Line 85: | Line 67: |
| {{{ RewriteCond %{HTTP:Authorization} ^(.+)$ RewriteRule ^/cgi-bin/pypi.cgi(.*) /Library/WebServer/CGI-Executables/pypi.cgi$1 [e=HTTP_CGI_AUTHORIZATION:%1,t=application/x-httpd-cgi,l] }}} If this doesn't work, then you can use these lines to debug: |
. [[http://mail.python.org/mailman/listinfo/distutils-sig|Mailing List]] ([[http://dir.gmane.org/gmane.comp.python.distutils|Gmane]] web interface) |
| Line 91: | Line 70: |
| {{{ RewriteLog /var/log/httpd/rewrite.log RewriteLogLevel 9 }}} |
. API that is used by easy_install http://peak.telecommunity.com/DevCenter/EasyInstall#package-index-api * [[PyPIOAuth]] - authentication library for Google and Launchpad logins == Legacy PyPI architecture and endpoints == PyPI is a WSGI application that can be executed standalone using `python pypi.wsgi` command if all requirements are met. `pypi.wsgi` contains usual WSGI wrapper code and delegates request processing to `WebUI.run()` method from `webui.py`. This method just opens DB and handles exceptions, actual request processing is done in `WebUI.inner_run()`. This method analyzes URL endpoint and executes appropriate handler. As of 2011-04, the rules to match endpoints to handlers are the following: ||/simple ||WebUI.run_simple() ||dump all package names on single html page || ||/simple/(.+)/ ||WebUI.run_simple() ||dump all links for a package in html list || ||/serversig/(.+)/ ||.run_simple_sign() ||save as above, but signed by server || ||/mirrors ||.mirrors() ||display static page with a list of mirrors || ||/daytime ||.daytime() ||display current server time || ... XML-RPC requests are detected by CONTENT_TYPE=`text/xml` variable in CGI environment and processed by `rpc.RequestHandler().__call__()`. List of XML-RPC "endpoints" is available on [[PyPIXmlRpc]] page. == Legacy PyPI Development Environment Hints == Removed (visible in [[https://wiki.python.org/moin/CheeseShopDev?action=recall&rev=81|page history]]) because developing and running legacy PyPI is deprecated. -- SumanaHarihareswara <<DateTime(2019-08-23T06:57:10-0400)>> |
This page is about development of the Python Package Index (formerly known as Cheeseshop):
More Info: CheeseShop
Developing the Package Index
PyPI.nextgen:
Currently, as of 2019-06-27, PyPI has undergone a complete rewrite from scratch, and as a result much of the information on this page is not up-to-date.
The development moved from Mercurial (Python) to Git (C, shell). License changed from BSD-3 to Apache 2.0. Project code named 'warehouse' can be downloaded from:
Testing Your Stuff Against PyPI
If you need to test stuff against PyPI (registration, uploading, API activities) then please use the alternative server, test.pypi.org.
TO-DO list
Meta-to-do: file the issues below that are still unresolved in the GitHub issue tracker, then remove this section. -- SumanaHarihareswara 2019-08-23 10:57:10
- A dump of download counts.
- A big structured dump of all package meta-data.
- A link from package to RTFD.
PEP for metadata 1.2 -- not finished and needs more catalog-sig discussion)
- documented procedures for "taking over" entries should the original owner of the entry go away (and any required system support)
- change notification emails
- per-classifier "wiki" content to allow description and discussion around each classifier (perhaps what packages are available and how they relate to one another)
- screenshot images (with thumbnailing and a "latest screenshot" on the front page?) - or perhaps icons instead of thumbnails for some packages?
Something that's been requested, but needs much more thought and analysis to see whether it causes any problems: the ability to treat project names and versions as case-insensitive, while removing extraneous characters (as in pkg_resources.safe_name()) for purposes both of searching and determining name uniqueness when registering.
Done
command-line tool to query pypi and fetch entries: yolk
Not Going TO-DO
Edit PEP 243 to reflect reality. The interface is implemented in the distutils register and upload commands. This code is good enough for documentation, especially because it's the only implementation necessary.
- moderated user reviews and ratings (this would require quite a lot of support from volunteers though)
Proposals
EnhancedPyPI Enhance multiple package index servers support in Distutils.
Previous PyPI version
The legacy version of PyPI is the code that was running on http://pypi.python.org for many years, till mid-2018. This LWN article goes into the history. The information below should help you get around the code.
The PyPI code was hosted under the Python Packaging Authority project: https://bitbucket.org/pypa/pypi and is now on GItHub: https://github.com/pypa/pypi-legacy
Bug and patch tracker https://github.com/pypa/pypi-legacy/issues
Mailing List (Gmane web interface)
API that is used by easy_install http://peak.telecommunity.com/DevCenter/EasyInstall#package-index-api
PyPIOAuth - authentication library for Google and Launchpad logins
Legacy PyPI architecture and endpoints
PyPI is a WSGI application that can be executed standalone using python pypi.wsgi command if all requirements are met. pypi.wsgi contains usual WSGI wrapper code and delegates request processing to WebUI.run() method from webui.py. This method just opens DB and handles exceptions, actual request processing is done in WebUI.inner_run(). This method analyzes URL endpoint and executes appropriate handler. As of 2011-04, the rules to match endpoints to handlers are the following:
/simple |
WebUI.run_simple() |
dump all package names on single html page |
/simple/(.+)/ |
WebUI.run_simple() |
dump all links for a package in html list |
/serversig/(.+)/ |
.run_simple_sign() |
save as above, but signed by server |
/mirrors |
.mirrors() |
display static page with a list of mirrors |
/daytime |
.daytime() |
display current server time |
...
XML-RPC requests are detected by CONTENT_TYPE=text/xml variable in CGI environment and processed by rpc.RequestHandler().__call__(). List of XML-RPC "endpoints" is available on PyPIXmlRpc page.
Legacy PyPI Development Environment Hints
Removed (visible in page history) because developing and running legacy PyPI is deprecated. -- SumanaHarihareswara 2019-08-23 10:57:10