Finding Modules in a Directory

Is there a better way than just listing the contents of the directory, and taking those tiles that end with ".pyc" or ".py"..?

But perhaps there isn't.

   1 import os
   2 
   3 def find_modules(path="."):
   4     """Return names of modules in a directory.
   5 
   6     Returns module names in a list. Filenames that end in ".py" or
   7     ".pyc" are considered to be modules. The extension is not included
   8     in the returned list.
   9     """
  10     modules = set()
  11     for filename in os.listdir(path):
  12         module = None
  13         if filename.endswith(".py"):
  14             module = filename[:-3]
  15         elif filename.endswith(".pyc"):
  16             module = filename[:-4]
  17         if module is not None:
  18             s.add(module)
  19     return list(modules)

Importing the Modules

How do you import a module, once you have it's name?

With the ImpModule! It dynamically loads named modules.

   1 import imp
   2 
   3 def load_module(name, path=["."]):
   4     """Return a named module found in a given path."""
   5     (file, pathname, description) = imp.find_module(name, path)
   6     return imp.load_module(name, file, pathname, description)
   7 
   8 modules = [load_module(name) for name in find_modules()]

Finding the Things Inside a Module

Once you have your module, you can look inside it, with .__dict__.

   1 module.__dict__

Finding Functions Within a Module

We just look for dictionary values that are of type types.FunctionType.

   1 def functions_in_module(module)
   2     functions = []
   3     for obj in module.__dict__.values():
   4         if isinstance(obj, types.FunctionType):
   5             functions.append(obj)
   6     return functions

Discussion

I got this error when executing find_modules() in a package directory. That is the directory contained an __init.py__ file:

  File "C:\Python254\lib\site-packages\joedorocak\find_modules.py", line 27, in find_modules
    s.add(module)
NameError: global name 's' is not defined

It looks to me like s needs to be initialized (some place near "modules = set()"). I'm not sure what the protocol is here, so I'm just going to leave this comment in the discussion.

Here's what seems to work for me. I got rid of 's' altogether.

def find_modules(path="."):
    """Return names of modules in a directory.

    Returns module names in a list. Filenames that end in ".py" or
    ".pyc" are considered to be modules. The extension is not included
    in the returned list.
    """
    modules = set()
    for filename in os.listdir(path):
        module = None
        if filename.endswith(".py"):
            module = filename[:-3]
        elif filename.endswith(".pyc"):
            module = filename[:-4]
        if module is not None:
            modules.add(module)
    return list(modules)

All the best,

JoeDorocak

Page

User

Finding Modules in a Directory

Importing the Modules

Finding the Things Inside a Module

Finding Functions Within a Module

See Also

Discussion

-  ⇤ ← Revision 3 as of 2005-04-15 04:37:07 → 
  Size: 2821
  Editor: aaron
  Comment: find_module, load_module
+   ← Revision 12 as of 2010-06-14 22:06:18 → ⇥
  Size: 3164
  Editor: c-98-207-20-119
  Comment:
-Deletions are marked like this.
+Additions are marked like this.
 Line 1:
-I frequently want to find all the modules in some directory, with some property, and do something with them.
+Here's how to find all the modules in some directory, and import them.
 Line 3:
-If you know how, please inform me. Here are some things I'm researching.
+<<TableOfContents>>

== Finding Modules in a Directory ==

Is there a better way than just listing the contents of the directory, and taking those tiles that end with ".pyc" or ".py"..?

But perhaps there isn't.

{{{
#!python
import os

def find_modules(path="."):
    """Return names of modules in a directory.

    Returns module names in a list. Filenames that end in ".py" or
    ".pyc" are considered to be modules. The extension is not included
    in the returned list.
    """
    modules = set()
    for filename in os.listdir(path):
        module = None
        if filename.endswith(".py"):
            module = filename[:-3]
        elif filename.endswith(".pyc"):
            module = filename[:-4]
        if module is not None:
            s.add(module)
    return list(modules)
}}}

== Importing the Modules ==

How do you import a module, once you have it's name?

With the ImpModule! It dynamically loads named modules.

{{{
#!python
import imp

def load_module(name, path=["."]):
    """Return a named module found in a given path."""
    (file, pathname, description) = imp.find_module(name, path)
    return imp.load_module(name, file, pathname, description)

modules = [load_module(name) for name in find_modules()]
}}}
-Line 6:
+Line 53:
+Once you have your module, you can look inside it, with {{{.__dict__}}}.
-Line 12:
+Line 61:
-== Identifying Functions ==

{{{
#!python
import types

def is_it_a_function(obj):
    return isinstance(obj, types.FunctionType)
}}}
-Line 24:
+Line 63:
-So, putting them together,...
+We just look for dictionary values that are of type {{{types.FunctionType}}}.
-Line 36:
+Line 75:
-== Finding Modules in a Directory ==
+== See Also ==
-Line 38:
+Line 77:
-Is there a better way than just listing the contents of the directory, and taking those tiles that end with ".pyc" or ".py"..?
+The DocXmlRpcServer page includes code demonstrating the use of these techniques.
-Line 40:
+Line 79:
-== Importing the Modules ==
+= Discussion =
I got this error when executing find_modules() in a package directory. That is the directory  contained an {{{ __init.py__}}} file:
{{{
  File "C:\Python254\lib\site-packages\joedorocak\find_modules.py", line 27, in find_modules
    s.add(module)
NameError: global name 's' is not defined
}}}
It looks to me like s needs to be initialized (some place near "modules = set()"). I'm not sure what the protocol is here, so I'm just going to leave this comment in the discussion.
-Line 42:
+Line 88:
-How do you do it dynamically, just given a filename?

Once you've imported it, how do you get a handle on it? (That is, how do you get it's __dict__?)

There seems to be [http://www.python.org/doc/current/lib/module-imp.html a module imp] that can be used to dynamically load a named module.

== Playing with imp Module ==

Make a file "eraseme.py":
+Here's what seems to work for me. I got rid of 's' altogether.
-Line 53:
+Line 91:
-#!python
print "Successfully imported!"
+def find_modules(path="."):
    """Return names of modules in a directory.

    Returns module names in a list. Filenames that end in ".py" or
    ".pyc" are considered to be modules. The extension is not included
    in the returned list.
    """
    modules = set()
    for filename in os.listdir(path):
        module = None
        if filename.endswith(".py"):
            module = filename[:-3]
        elif filename.endswith(".pyc"):
            module = filename[:-4]
        if module is not None:
            modules.add(module)
    return list(modules)
-Line 57:
+Line 111:
-Then, at the python shell,
+All the best,
-Line 59:
+Line 113:
-{{{
#!python
>>> import imp
>>> returned = imp.find_module("eraseme", ["."])
>>> returned
(<open file 'eraseme.py', mode 'U' at 0xb7f62220>, 'eraseme.py', ('.py', 'U', 1))
>>>
}}}

The {{{find_module}}} line searches for "eraseme" in the current working directory (".").

The first two returned items are self-explanatory, but what about that tuple- {{{('.py', 'U', 1)}}} ..?

The first (".py") is obviously the extension.

'U' means, "I opened the file in UniversalNewline mode." Basically, universal newline mode is like read ("r"), except that it interprets all newline forms the same way.

Finally, the 1 is a code that matches against imp.PY_SOURCE (1), imp.PY_COMPILED (2), or imp.C_EXTENSION (3). Basically, it's telling us that the .py file is a source file.

Once you have this stuff, it's easy to load:

{{{
#!python
>>> eraseme = imp.load_module("eraseme", returned[0], returned[1], returned[2])
successfully imported!
>>> eraseme
<module 'eraseme' from './eraseme.py'
}}}

Note that, you don't have to play by the rules. If you wanted to, you could have said:

{{{
#!python
>>> module = imp.load_module("eggs", returned[0], "spam.py", returned[2])
successfully imported!
>>> module
<module 'eggs' from 'spam.py'>
>>> dir(module)
['__builtins__', '__doc__', '__file__', '__name__']
>>> module.__name__
'eggs'
>>> module.__file__
'spam.py'
>>>
}}}

...even though the file's ''real'' name is "eggs.py".
+JoeDorocak