Differences between revisions 1 and 10 (spanning 9 versions)
Revision 1 as of 2008-01-04 11:51:45
Size: 6175
Editor: 181
Comment:
Revision 10 as of 2008-02-20 23:17:54
Size: 6311
Editor: FredDrake
Comment: fix minor typo
Deletions are marked like this. Additions are marked like this.
Line 5: Line 5:
 * a list of enhancement to make Disutils packaging system supports several index servers;
 * changes to be made in the central server index implemetation, a.k.a PyPI;
 * a guideline for implementing a PyPI-compatible server.
 * a list of enhancements to make Distutils packaging system support several index servers;
 * changes to be made in the central server index implementation, a.k.a PyPI;
Line 12: Line 11:
Python package distribution since Python 2.4. Together with Distutils Python package distribution since Python 2.4. Together with Distutils 
Line 14: Line 13:
releasing their packages in eggs, and are making them avaible in PyPI. releasing their packages in eggs, and are making them available in PyPI.
Line 16: Line 15:
Web frameworks such as Plone and the underlying framework, are
now entirely distributed in eggs, and use PyPI as the main place to
publish them. Some tools like zc.buildout automates the build of
Web frameworks such as Plone and the underlying framework, are 
now entirely distributed in eggs, and use PyPI as the main place to 
publish them. Some tools like {{{zc.buildout}}} automate the build of 
Line 21: Line 20:
This central approach is great for Python advocacy, and for the cohesion
of the community of developers, but brings a few issues:
This central approach is great for Python advocacy, and for the cohesion 
of the community of developers, but brings a few issues. The ones
addressed in this document are
:
Line 24: Line 24:
 * when PyPI is down or slow, many developers are unable to build their     application, unless they have a local PyPI cache;
 * when people from a sub-community, like Plone, want to promote their
  
work, they need to shout it out in several places, to make sure
  
it has reach people focused on the given framework. These places
  
are acting a bit like the PyPI, so a common releasing standard
   
make thing easier to automate.
 * when PyPI is unreachable, developers are unable to build their application, unless they have a local PyPI cache;
 * when people from a sub-community, like Plone, want to promote their work, they need to shout it out in several places, to make sure it has reach people focused on the given framework. These places are acting a bit like the PyPI, so a common releasing standard make things easier to automate.
Line 32: Line 27:
This PEP provides a solution to transparently support several index
servers. It is a very low risk and very simple to implement. It is also
This document provides a solution to transparently support several index 
servers. It is a very low risk and very simple to implement. It is also 
Line 40: Line 35:
* making .pypirc supports multiple servers
* making PyPI permissive for Trove classification
* providing a base layer for PyPI-like servers implementation
Line 44: Line 36:
= Making .pypirc supports multiple servers =  * making {{{.pypirc}}} support multiple servers
* making PyPI permissive for Trove classification
Line 46: Line 39:
Right now, the .pypirc file is intended to keep username and password = Making .pypirc support multiple servers =

Right now, the {{{.pypirc}}} file is intended to keep username and password 
Line 53: Line 48:
Line 56: Line 50:
The default repository is PyPI, and when another repository has to be used,
there are two options. 
The default repository is PyPI, and when another repository has to be used, 
there are two options.
Line 59: Line 53:
Either adding the repository url in the command line::
Either adding the repository url in the command line:
Line 66: Line 59:
Either adding it in the .pypirc file::
or adding it in the {{{.pypirc}}} file:
Line 76: Line 68:
In both cases, it is not possible when your username differs from a server to
another, to keep a username/password for each server. Furthermore the realm
In both cases, if your username differs from a server to another, it is not
possible
to keep a username/password for each server. Furthermore the realm 
Line 82: Line 74:
A simple way to enhance it, is to be able to add several sections in .pypirc. A simple way to enhance it, is to be able to add several sections in
{{{
.pypirc}}}. The root section would be the {{{[distutils]}}}
section, with a list of sections that represent a server.
Line 84: Line 78:
For instance::
For example:
Line 88: Line 81:
  [server-login]   [distutils]
  index-servers =
    pypi
    my-other-server
    my-other-server-with-its-own-realm

  [pypi]
  repository:http://pypi.python.org/pypi/
Line 91: Line 91:
Line 104: Line 105:
This will allow backward-compatibility, and make register and upload able
to interact with several servers when they are called.
Line 107: Line 106:
== Asking the user at the prompt == When a user calls the {{{register}}} or the {{{upload}}} command, it will
use the default server located in the {{{pypi}}} section, or
the server given by the {{{-r}}} option:
Line 109: Line 110:
When a user call register or uplad with such a file, it will be asked
for each server wheter he wants to perform the action over it.
Line 112: Line 111:
XXX to be completed if if accepted, until then, iw.dist is a
working prototype available at the cheeseshop.
{{{
$ python setup.py register sdist upload -r http://example.com/repository # registering and uploading at example.com
$ python setup.py register sdist upload # registering and uploading at PyPI
}}}
Line 115: Line 116:
see: http://www.nabble.com/enhancing-.pypirc-file-for-multiple-servers-handling-to14504516.html#a14527503 The {{{-r}}} option will also accept the name of the section:

{{{
$ python setup.py register sdist upload -r my-other-server # registering and uploading at example.com
$ python setup.py register sdist upload -r pypi # registering and uploading at PyPI
}}}

Default values in this file will be:

 * {{{distutils:index-servers}}} : {{{"pypi"}}}
 * {{{pypi:repository}}} : {{{http://pypi.python.org/pypi/}}}


Backward compatibility will be kept, and a file that uses the old format:

{{{
  [server-login]
  username:tarek2
  password:secret
}}}

...will be translated as:

{{{
  [distutils]
  index-servers =
    pypi

  [pypi]
  username:tarek2
  password:secret
}}}
Line 121: Line 154:
This is intended because the server provide a "package center" that has its own domain-specific categories. This is intended because the server can provide a "package center" that has its own domain-specific categories.
Line 130: Line 163:
But a slightly different one in another server. Let's say, the ACME company But a slightly different one in another server. Let's say, the ACME company:
Line 140: Line 173:
a warning is popped at the prompt:: a warning is popped at the prompt. The unknown category is not created on the server:
each server keeps its own classifiers.
Line 145: Line 179:
 $ python setup.py register

 Do
you want to register at PyPI (y/N) ? y
 $ python setup.py register # registering at PyPI
Line 152: Line 184:
   Do you want to register at ACME (y/N) ? y
Line 155: Line 185:
        ...         $ python setup.py register -r http://example.com/repository # registering at example.com
Line 161: Line 191:
This will allow visual checking when a typo is made.

XXX code the patch for PyPI code if accepted

== Providing a base layer for PyPI-like servers implementation ==

The PyPI current implementation is done in a cgi-like Python application,
that works with a Postgres database. The code though, is not based on an abstract
layer. For instance, when a package is uploaded, the request's form is checked
then the data are pushed into the Database through SQL query.

This means that everytime Disutils wants to change its behavior, PyPI has
to change too.

A base server-side layer could be added in Distutils itself, to provide:
* a model for the storage;
* a base class for the API the server has to provide.

With such a layer, it would also facilitate the standardization
of all PyPI-like servers.

This section presents this base layer: XXX to be completed if accepted
This will allow visual checking when a typo is made. The package will then be available
in each server, but only under the categories known to the server.

Abstract

This document provides:

  • a list of enhancements to make Distutils packaging system support several index servers;
  • changes to be made in the central server index implementation, a.k.a PyPI;

Rationale

The PyPI is playing a very important role in the standardization of Python package distribution since Python 2.4. Together with Distutils and the third-party library Setuptools, most Python programmers are releasing their packages in eggs, and are making them available in PyPI.

Web frameworks such as Plone and the underlying framework, are now entirely distributed in eggs, and use PyPI as the main place to publish them. Some tools like zc.buildout automate the build of Python application by looking for eggs in PyPI and downloading them.

This central approach is great for Python advocacy, and for the cohesion of the community of developers, but brings a few issues. The ones addressed in this document are:

  • when PyPI is unreachable, developers are unable to build their application, unless they have a local PyPI cache;
  • when people from a sub-community, like Plone, want to promote their work, they need to shout it out in several places, to make sure it has reach people focused on the given framework. These places are acting a bit like the PyPI, so a common releasing standard make things easier to automate.

This document provides a solution to transparently support several index servers. It is a very low risk and very simple to implement. It is also finalizing a feature that was primarily intended by Distutils and PyPI, but not easy to use as-is: making it a central server, but allowing people to register and upload their package elsewhere.

The rest of the document presents the actions to take, and the work to be done for it:

  • making .pypirc support multiple servers

  • making PyPI permissive for Trove classification

Making .pypirc support multiple servers

Right now, the .pypirc file is intended to keep username and password for registering a package. The file looks like

  [server-login]
  username:tarek
  password:secret

The default repository is PyPI, and when another repository has to be used, there are two options.

Either adding the repository url in the command line:

  $ python setup.py register -r http://example.com/repository

or adding it in the .pypirc file:

  [server-login]
  username:tarek2
  password:secret
  repository:http://example.com/repository

In both cases, if your username differs from a server to another, it is not possible to keep a username/password for each server. Furthermore the realm associated with the server is hardcoded to "pypi".

Several sections in .pypirc

A simple way to enhance it, is to be able to add several sections in .pypirc. The root section would be the [distutils] section, with a list of sections that represent a server.

For example:

  [distutils]
  index-servers =
    pypi
    my-other-server
    my-other-server-with-its-own-realm

  [pypi]
  repository:http://pypi.python.org/pypi/
  username:tarek2
  password:secret


  [my-other-server]
  username:tarek2
  password:secret
  repository:http://example.com/repository

  [my-other-server-with-its-own-realm]
  username:tarek3
  password:secret3
  repository:http://example2.com/repository
  realm:acme

When a user calls the register or the upload command, it will use the default server located in the pypi section, or the server given by the -r option:

$ python setup.py register sdist upload -r http://example.com/repository       # registering and uploading at example.com
$ python setup.py register sdist upload        # registering and uploading at PyPI

The -r option will also accept the name of the section:

$ python setup.py register sdist upload -r my-other-server      # registering and uploading at example.com
$ python setup.py register sdist upload -r pypi       # registering and uploading at PyPI

Default values in this file will be:

  • distutils:index-servers : "pypi"

  • pypi:repository : http://pypi.python.org/pypi/

Backward compatibility will be kept, and a file that uses the old format:

  [server-login]
  username:tarek2
  password:secret

...will be translated as:

  [distutils]
  index-servers =
    pypi

  [pypi]
  username:tarek2
  password:secret

Making PyPI permissive for Trove classification

PyPI is based on a Trove classification, see http://www.python.org/dev/peps/pep-0301/#distutils-trove-classification. Another server may have its own trove classification though, that differs from PyPI. This is intended because the server can provide a "package center" that has its own domain-specific categories.

For example, a package might have this classification:

  Development Status :: 5 - Production/Stable
  Intended Audience :: Developers

But a slightly different one in another server. Let's say, the ACME company:

  ACME :: Visibility :: Public
  Development Status :: 5 - Production/Stable
  Intended Audience :: Developers

A permissive Trove classification would allow the registering of the package in both servers, even if the categories does not exist. For each unkown category a warning is popped at the prompt. The unknown category is not created on the server: each server keeps its own classifiers.

When registering it to several servers, the expected output would be::

        $ python setup.py register   # registering at PyPI
        ...
        Warning "ACME :: Visibility :: Public" classifier not found on the server
        200 - OK


        $ python setup.py register -r http://example.com/repository   # registering at example.com
        Warning "Development Status :: 5 - Production/Stable" classifier not found on the server
        Warning "Intended Audience :: Developers" classifier not found on the server
        200 - OK

This will allow visual checking when a typo is made. The package will then be available in each server, but only under the categories known to the server.

EnhancedPyPI (last edited 2008-11-15 14:00:12 by localhost)

Unable to edit the page? See the FrontPage for instructions.