Revision 6 as of 2008-10-12 17:39:19

Clear message

Mirroring infrastructure in PyPI

Abstract

This PEP describes a mirroring infrastructure for PyPI.

Motivation

PyPI is hosting over 4000 projects and is used on a daily basis by people to build applications. Especially systems like easy_install and Buildout make intensive usage of PyPI.

For people making extensive use of PyPI, it can act as a single point of failure. People have started to set up some mirrors, both private and public. Those mirrors are active mirrors, which means that they are browsing PyPI to get synced.

The motivation of this PEP is to set up a registering mechanism in PyPI in order to list all the public PyPI mirrors and to provide an event based system where all mirrors get informed via RPC when a package has been uploaded, modified, or removed, so they can eventually sync themselves.

This PEP describes:

Mirror listing and registering

A new HTML page will be added at http://pypi.python.org/mirrors that can be browsed like the simple index. This page gives a list of the mirrors through a list of links.

These links are the URL of the simple index of each mirror. The page will look like this:

<html>
  <head><title>PyPI mirrors</title></head>
  <body>
  
    <h1>PyPI mirrors</h1>
    
    <p>
    If you want to register a new mirror, send an email
    to the catalog-SIG@python.org with:
    </p>

    <ol>
        <li> The url of your mirror.</li>
        <li> The name and email of the maintainer.</li>
        <li> The url of the freshness date.</li>
    </ol>

    <p>
    The registering is done manually and to become a
    mirror, you need to strictly follow the package index
    API defined here:
    </p>
    
    <p>
    http://peak.telecommunity.com/DevCenter/EasyInstall#package-index-api
    </p>
    
    <ul id="mirror-links">
        <li>
          <a class="link" href="http://example.com/pypi">Mirror #1</a> |
          <a class="freshness" href="http://example.com/freshness">freshness</a> |
          <a class="page" href="http://pypi.python.org/mirrors/1">more infos</a>
        </li>
    </ul>

  </body>
</html>

When a mirror is proposed on the mailing list, it is manually added in a dedicated SQL table in the PyPI application.

The mirror list page is a simple html page that can be browsed by any tool that wants to get a list of registered mirrors.

Each mirror gets its own page url at PyPI with extra information. The name of that page is the ID of the mirror in the PyPI system.

This page contains extra information about the mirror:

exemple:

<html>
  <head><title>PyPI mirror</title></head>
  <body>
  
    <h1 id="name">Mirror #1</h1>
    <ul>
      <li>Maintainer: Some guy</li>
      <li>Url: <a id="url" href="http://example.com/pypi">Mirror #1</a></li>
      <li>Freshness date: <span id="freshness">2008-10-12T15:23:01</span></li>
      <li>HTTP response time: <span id="responseTime">0.7270 seconds</span></li>
    </ul>
  </body>
</html>

Other package indexes that are not mirrors of PyPI are not added in the mirror list in PyPI.

Although they can provide themselve the same mirroring list mechanism for their own mirrors.

Freshness date

CPAN uses a freshness date system where the mirror last synchronisation date is made available.

Each mirror is maintaining an url with a simple txt/plain content that represents the last synchronisation date the mirror maintains.

The date is provided in GMT time, using the iso 8601 format (http://en.wikipedia.org/wiki/ISO_8601)

Costs

Someone has to manage the list of mirrors. This work should not take too much time. I am willing to be that maintainer if the people that maintain the server don't have the time, or don't trust me.

Unable to edit the page? See the FrontPage for instructions.