Introduction to distutils

I have recently used distutils for the first time and thought I would share what I have learned.

Please note: I am not a pro. All this is based on my own hacking and struggling to get things to work. I am sure there will be alternative ways to do things and that I will make mistakes. Caveat Emptor.

(Based on GNU/Linux)

The layout of folders

A proper layout of your files and folders can really save you trouble later on. I use this layout:

top
|-- package
|   |-- __init__.py
|   |-- module.py
|   `-- things
|       |-- cross.png
|       |-- fplogo.png
|       `-- tick.png
|-- runner
|-- MANIFEST.in
|-- README
`-- setup.py

UPDATE: How to add other files, other data and directories

I have a tentative solution for when you need extra files and folders in your distributed tarball. This is surprisingly tricky to do. Please see the section at the end titled "Other Files"

The files

runner

I assume that there will be a single script file that you will use to start your Python app. I call it 'runner' (without the .py) in this example. It will import the actual package. Here is the code:

runner

##This will work in development on a relative folder basis
##It will then work when installed in site-packages on a target system
##where the runner script is in /usr/bin (or wherever)
##
##So, you don't need anything special - no fancy path tricks.

import package.module

package.module.start ()

module.py

Next is the package. In this case there is only one module in the package (import package.module), so here is the module:

module.py

import os, sys

def determine_path ():
    """Borrowed from wxglade.py"""
    try:
        root = __file__
        if os.path.islink (root):
            root = os.path.realpath (root)
        return os.path.dirname (os.path.abspath (root))
    except:
        print "I'm sorry, but something is wrong."
        print "There is no __file__ variable. Please contact the author."
        sys.exit ()
        
def start ():
    print "module is running"
    print determine_path ()
    print "My various data files and so on are:"
    files = [f for f in os.listdir(determine_path () + "/things")]
    print files
    
if __name__ == "__main__":
    print "Decide what to do"

This uses the magic variable __file__ that will return the path and name of the module. Without it you would have to jump through some hoops to locate the module on any given distro or O/S.

It shows that it can find the data files in the things folder.

`__init__.py`

The __init__.py file is empty.

README

The README file is named that way because it's one of the ways that distutils automatically looks for it. You should put details about your app into it. It's not a required file.

MANIFEST.in

The MANIFEST.in file took me a while to understand. It's the file that distutils uses to collect all the files in your project that will go into the final installer tarball (the file that gets distributed).

NB: whatever is in MANIFEST.in will be in your installer. Whatever is not in there, will not be in your installer.

Shortly, in your setup.py file, you will see other references to the various files in your project. It seems redundant to refer to all the files you want twice, but this is a fact: MANIFEST.in == what will go into your installer.

Here is the file:

MANIFEST.in

include runner README
recursive-include package/things *

It has a limited syntax (see the docs),but basically:

You do not have to include any .py files in your package-root folder. You do have to include the runner script because it has no .py extension. (I am not sure if it would pick up .py files in the root.)

setup.py

Finally we get to the setup.py file:

setup.py

from distutils.core import setup

#This is a list of files to install, and where
#(relative to the 'root' dir, where setup.py is)
#You could be more specific.
files = ["things/*"]

setup(name = "appname",
    version = "100",
    description = "yadda yadda",
    author = "myself and I",
    author_email = "email@someplace.com",
    url = "whatever",
    #Name the folder where your packages live:
    #(If you have other packages (dirs) or modules (py files) then
    #put them into the package directory - they will be found 
    #recursively.)
    packages = ['package'],
    #'package' package must contain files (see list above)
    #I called the package 'package' thus cleverly confusing the whole issue...
    #This dict maps the package name =to=> directories
    #It says, package *needs* these files.
    package_data = {'package' : files },
    #'runner' is in the root.
    scripts = ["runner"],
    long_description = """Really long text here.""" 
    #
    #This next part it for the Cheese Shop, look a little down the page.
    #classifiers = []     
) 

Making the installation tarball

Well, that's the lot. It's quite simple really, as long as you keep to the pattern. If you want extra folders and sub-packages and so forth, it gets a little hairy. But, I suppose, if you want all that then you will have no trouble extending it from this tutorial.

To make the installation tarball, you simply run:

python setup.py sdist

That will crank through all the files in your MANIFEST.in and build the tar.gz file in a folder called dist off your root.

Look for errors. You might have to fix lines in your MANIFEST.in file.

Here is what I see:

output

distutils:$ python setup.py sdist
running sdist
reading manifest file 'MANIFEST'
creating appname-100
creating appname-100/package
creating appname-100/package/things
making hard links in appname-100...
hard linking README -> appname-100
hard linking runner -> appname-100
hard linking setup.py -> appname-100
hard linking package/__init__.py -> appname-100/package
hard linking package/module.py -> appname-100/package
hard linking package/things/cross.png -> appname-100/package/things
hard linking package/things/fplogo.png -> appname-100/package/things
hard linking package/things/tick.png -> appname-100/package/things
tar -cf dist/appname-100.tar appname-100
gzip -f9 dist/appname-100.tar
removing 'appname-100' (and everything under it)

Testing the tarball

I perform these steps:

cd dist
tar xzf appname-100.tar.gz 
cd appname-100
sudo python setup.py install
password:######
running install
running build
running build_py
running build_scripts
running install_lib
running install_scripts
changing mode of /usr/bin/runner to 755

Then I cd over to /usr/lib/python2.4/site-packages and have a look for a folder named 'package'. I also try to run 'runner' and see what ensues.

It should look (something) like this:

distutils:$ runner
module is running
/usr/lib/python2.4/site-packages/package
My various data files and so on are:
['cross.png', 'fplogo.png', 'tick.png']

Registering with the Cheese Shop (PyPI)

The Cheese Shop is a repository of Python apps. You can add your app to it quite easily, I was surprised how quickly this worked. ( Requires Python 2.3 or later )

Make sure you have the right meta-data in your setup.py file. Please consult the Python Docs for this info. Here is a sample of what you should include:

    classifiers=[
      'Development Status :: 4 - Beta',
      'Environment :: X11 Applications :: GTK',
      'Intended Audience :: End Users/Desktop',
      'Intended Audience :: Developers',
      'License :: OSI Approved :: GNU General Public License (GPL)',
      'Operating System :: POSIX :: Linux',
      'Programming Language :: Python',
      'Topic :: Desktop Environment',
      'Topic :: Text Processing :: Fonts'
      ]

Then you type:

$ python setup.py register
running register
We need to know who you are, so please choose either:
 1. use your existing login,
 2. register as a new user,
 3. have the server generate a new password for you (and email it to you), or
 4. quit
Your selection [default 1]:  2
Username: putyournamehere
Password:
 Confirm:
   EMail: youremailhere
You will receive an email shortly.
Follow the instructions in it to complete registration.

After all that, you can quickly find you app in the Cheese Shop via your browser, or sign up for their RSS feed.

Submittting to the Cheese Shop (PyPI)

Requires Python 2.5 or later:

distutils:$ python setup.py sdist upload

TODO: need a sample output here

To GPG-sign the upload, use:

distutils:$ python setup.py sdist upload --sign

TODO: need a sample output here

Other Files

WARNING I have found a problem with this approach that causes all the data directories to be installed to the /usr/ directory. I have no idea why and not much help seems forthcoming. I will update when I get a clue. UPDATE TO WARNING Okay, I think I have fixed the problem, new code has been added to the example below.

Jan 4 2008. When you need other stuff in your tarball (.py files included) and distutils is causing your brain to segfault, this may help you. Robin Dunn (he of wxPython fame) pointed me at his source code and I spent a little time hacking it.

I'll use Fonty Python as the example - I'll show you the tree and explain it a little, then I'll follow that with the code that does the job.

The tree

Here's the Fonty Python tree (as it stands, leaning wildly, right now). 'trunk' is the distribution root. I have trimmed it somewhat.

trunk:$ tree
|-- COPYING
|-- README
|-- fontypython
|-- fontypythonmodules
|   |-- __init__.py
|   |-- cli.py
|   |-- help
|   |   |-- README
|   |   |-- common
|   |   |   |-- README
|   |   |   |-- fp_ticking.png
|   |   |   `-- logo_dec_2007.png
|   |   |-- en
|   |   |   |-- README
|   |   |   |-- fp_1to8.png
|   |   |   `-- help.html
|   |   `-- fr
|   |       |-- README
|   |       |-- fp_1to8.png
|   |       `-- help.html
|   |-- i18n.py
|   |-- locale
|   |   |-- fr
|   |   |   `-- LC_MESSAGES
|   |   |       `-- all.mo
|   |   `-- it
|   |       `-- LC_MESSAGES
|   |           `-- all.mo
|   |-- pofiles
|   |   |-- README
|   |   |-- fp_all.po
|   |   |-- fr_all.merged.po
|   |   |-- it_all.merged.po
|   |   `-- who_did_what
|   |-- things
|   |   |-- README
|   |   `-- tick.png
|   `-- wxgui.py
|-- fp
`-- setup.py

So, there are .py files, text files, locale files, images and html files. I want them all in the tarball in the right order.

The MANIFEST.in file

include fontypython fp README COPYING
recursive-include fontypythonmodules/pofiles *
recursive-include fontypythonmodules/things *
recursive-include fontypythonmodules/locale *
recursive-include fontypythonmodules/help *

I tell it to recurse all those subfolders. I include 'fp', 'fontypython', 'README' and 'COPYING' because they are in the root - perhaps there's a better way to do it, but this works.

The Magic Code

The magic stuff is in the two functions called opj and find_data_files that you put in your setup.py file. They work together to build a list of all the files in a given path. This list is in the right shape for distutils to swallow. You pass it in via the data_files variable in the setup() call.

(Ignore all my fontypythonmodules dot stuff - that's just to get strings from another module.)

After making these changes, a normal python setup.py sdist works without oddball errors and the tarball has all the files you'd expect it to.

import os, sys, glob, fnmatch

## Added 10 Jan 2008
from distutils.core import setup, Extension
import distutils.command.install_data

## Code borrowed from wxPython's setup and config files
## Thanks to Robin Dunn for the suggestion.
## I am not 100% sure what's going on, but it works!
def opj(*args):
    path = os.path.join(*args)
    return os.path.normpath(path)

## Added 10 Jan 2008
# Specializations of some distutils command classes
class wx_smart_install_data(distutils.command.install_data.install_data):
    """need to change self.install_dir to the actual library dir"""
    def run(self):
        install_cmd = self.get_finalized_command('install')
        self.install_dir = getattr(install_cmd, 'install_lib')
        return distutils.command.install_data.install_data.run(self)

def find_data_files(srcdir, *wildcards, **kw):
    # get a list of all files under the srcdir matching wildcards,
    # returned in a format to be used for install_data
    def walk_helper(arg, dirname, files):
        if '.svn' in dirname:
            return
        names = []
        lst, wildcards = arg
        for wc in wildcards:
            wc_name = opj(dirname, wc)
            for f in files:
                filename = opj(dirname, f)

                if fnmatch.fnmatch(filename, wc_name) and not os.path.isdir(filename):
                    names.append(filename)
        if names:
            lst.append( (dirname, names ) )

    file_list = []
    recursive = kw.get('recursive', True)
    if recursive:
        os.path.walk(srcdir, walk_helper, (file_list, wildcards))
    else:
        walk_helper((file_list, wildcards),
                    srcdir,
                    [os.path.basename(f) for f in glob.glob(opj(srcdir, '*'))])
    return file_list

## This is a list of files to install, and where:
## Make sure the MANIFEST.in file points to all the right 
## directories too.
files = find_data_files('fontypythonmodules/', '*.*')

from distutils.core import setup

setup(name = "fontypython",
    version = fontypythonmodules.fpversion.version,
    description = fontypythonmodules.strings.description,
    author = "Donn.C.Ingle",
    author_email = fontypythonmodules.strings.contact,
    license = "GPL-3.0",
    url = "https://savannah.nongnu.org/projects/fontypython/",
    packages = ['fontypythonmodules'],
    
    data_files = files,
    
    ## Borrowed from wxPython too:
    ## Causes the data_files to be installed into the modules directory.
    ## Override some of the default distutils command classes with my own.
    cmdclass = { 'install_data':    wx_smart_install_data },
    
    #'fontypython' and 'fp' are in the root.
    scripts = ["fontypython", "fp"],
    long_description = fontypythonmodules.strings.long_description,
    classifiers=[
      'Development Status :: 4 - Beta',
      'Environment :: X11 Applications :: GTK',
      'Intended Audience :: End Users/Desktop',
      'Intended Audience :: Developers',
      'License :: OSI Approved :: GNU General Public License (GPL)',
      'Operating System :: POSIX :: Linux',
      'Programming Language :: Python',
      'Topic :: Desktop Environment',
      'Topic :: Text Processing :: Fonts'
      ]
)

NEW VERSION OF def walk_helper()

Given that you may want to exclude files that contain more than just .pyc in them, we need something handy. Replace the def in the code above with this code.

## A list of partials within a filename that would disqualify it
## from appearing in the tarball.
badnames=[".pyc","~","no_"]
def walk_helper(arg, dirname, files):
    if 'CVS' in dirname: ## ADD/CHANGE as you need here too.
        return
    names = []
    lst, wildcards = arg
    for wc in wildcards:
        wc_name = opj(dirname, wc)
        for f in files:
            filename = opj(dirname, f)
            #if ".pyc" not in filename:
            ## This hairy looking line excludes the filename
            ## if any part of one of  badnames is in it:
            if not any(bad in filename for bad in badnames):
                if fnmatch.fnmatch(filename, wc_name) and not os.path.isdir(filename):
                    names.append(filename)
    if names:
        lst.append( (dirname, names ) )

The secret is the list badnames which you can fill as you please.

Fin

I hope this helps someone. Happy coding.

DonnIngle


CategoryFaq CategoryDistutilsCookbook

Distutils/Tutorial (last edited 2016-08-31 21:42:42 by GeneWood)

Unable to edit the page? See the FrontPage for instructions.