• Distutils
• ManifestPluginSystem

Problem

Right now there are several ways to build the MANIFEST file that comes with a source distribution, besides the scripts.

People tend to use:

• the MANIFEST.in templating system
• a script that scans the (D)VCS structures in order to build the list of files. (setuptools)
• package_data and data_files arguments

Moreover, Distutils will include some elements by default, like README.txt and so one.

There's no consensus on what is the good practice yet, but;

• some people don't want to use and maintain a MANIFEST.in file. They want an automatic tool.
• you have to combine several techniques when you are dealing with special cases, such as:
• if you use a VCS script, you might want to exclude some files that are listed in your repository. You need to use MANIFEST.in for that.
• if you name explicitely the files in package_data, there are no way to include recursively some directories, and to exclude some files. That is why you also need to use MANIFEST.in
• MANIFEST.in is fine, but doesn't provide an automatic way to include elements like a VCS script would.

The problem is that a packager that wants to work with a package might be unable to get the list of files of this package if he doesn't work exactly like the author.

Example:

• Bob develops his package using Subversion. He doesn't bather about the file list because when he creates his package, it is automatically built using a setuptools script that scans the .svn folder. He works with svn 1.5.
• Bill gets a source distribution, and wants to work on it. He is working with the brand new 1.6 version of Subversion, which is incompatible with the current setuptools VCS script. He is unable to create a source distribution anymore, unless he ask someone else to build the MANIFEST file for him.
• John works with this brand new DVCS, incompatible with Subversion. Unfortunately, no one has written a script in setuptools. Like Bill, he is unable to launch the sdist command because he will probably get a incomplete list of files.

Result :

• Bill and John will have to maintain their own versions of setup.py (and maybe MANIFEST.in) until they merge.

Proposal

This proposal does not solve the problem, because it's unsolvable. Unless everyone use the same way to list the files, people have to work around. This proposal make Distutils simpler to use, no matter how people build their file list.

Let's add in Distutils:

• a generic plugin system in the core
• some plugins to build a file list :

  • a template plugin that knows how to build a filelist using the MANIFEST.in templating system

  • a default plugin that uses the templateplugin and do extra things (to provide what Distutils provides today)

  • a static plugin, that points to a MANIFEST static file

• a new manifest metadata

plugin system

The plugin system is inspired from Setuptools entry points.

Maybe setuptools entry_points code could be added in Distutils, but this section describes the need nevertheless.

Some functions are added in Distutils, in a module called plugins:

• get_plugin(group, name) : returns an object for (group, name)
• register_plugin(group, name, object): register an object, for (group, name)
• unregister_plugin(group, name): removes an object for (group, name)
• list_plugins(group=None, doc=False): returns a list of all objects for the given group.
• list_groups(): return a list of all groups

The filelist module implements a default plugin, in the distutils:filelist group, that contains the current Distutils algorithm that builds the MANIFEST file:

    >>> register_plugin('distutils:filelist', 'default',
    ...                 distutils.filelist.FileList)

Distutils is refactored in order to use this plugin. For instance the default plugin is the FileList class. It's used by the sdist command for example.

The default plugin uses another plugin called template, that uses the MANIFEST.in templating system:

    >>> register_plugin('distutils:filelist', 'template',
    ...                 distutils.filelist.ManifestTemplate)

Another plugin is also provided, called static:

    >>> register_plugin('distutils:filelist', 'static',
                        distutils.filelist.StaticFile)

This plugin allows to provide a path to a simple file that contains the list of file (like a MANIFEST file).

Last, list_plugins('distutils:filelist'), returns a list of all plugins names of a given group:

    >>> list_plugins('distutils:filelist')
    ['default', 'template', 'static']

An optional parameter can be use to get more information:

    >>> list_plugins('distutils:filelist', doc=True)
    [('default', 'Default plugin to get the file list of a package'),
     ('static', 'Returns a file list given a path of a file that contains the list')]

Without parameters, list_plugins() returns all objects.

How a plugin is called and used ?

A plugin is called with a in-out argument called filelist, an optional root_path argument and a list of keywords:

    >>> get_plugin('distutils:filelist', 'default')(filelist, root_path='/somewhere')

If root_path is None, the plugin might want to use the current working directory to start his scanning work.

In sdist, the plugin is called with the current working directory and all options passed to sdist.

How to choose one or several plugin

sdist will have a new option, called manifest, this option is a list of plugin names. If not given, ['default'] is used by default.

Someone that wants to use another plugin will simply add it in the list:

    >>> setup(name='foo', manifest=['hg'])

He might also want to use several plugins. For instance, build a file list with the hg plugin, then exclude some files with the template plugin:

    >>> setup(name='foo', manifest=['hg', 'template'])

The sdist command will loop over all plugins to build the file list using this simple code::

    >>> def get_filelist():
    ...    filelist = []
    ...    for name in list_plugins('distutils:filelist'):
    ...        plugin = get_plugin(name)
    ...        plugin(filelist)
    ...    return filelist

Each plugin is able to add or remove a file from the file list.

the manifest metadata

The manifest metadata already exists in a sense. It's not in PKG-INFO itself, but lands into the .egg-info directory once it's built.

A new display option will allow people to get it without having to call sdist or egg_info, exactly like --long-description provides:

    $ python setup.py --manifest

This will call the get_filelist() function and display its result in the standard output.