Differences between revisions 6 and 7
Revision 6 as of 2011-07-08 09:57:21
Size: 4877
Editor: 2001:878:200:1001:e2f8:47ff:fe30:30c8
Comment:
Revision 7 as of 2011-07-08 11:57:05
Size: 7032
Editor: 2001:878:200:1001:e2f8:47ff:fe30:30c8
Comment:
Deletions are marked like this. Additions are marked like this.
Line 33: Line 33:
Line 65: Line 64:
 ``__import__()``  ``__import__(self, name, globals={}, locals={}, fromlist=[], level=0)``
 Reimplementation of the builtin ``__import__()`` function. The import of a module will proceed using the state stored in the ImportEngine instance rather than the global import state. For full documentation of ``__import__`` funtionality, see [2]_ . ``__import__()`` from ``ImportEngine`` and its subclasses can be used to customise the behaviour of the ``import`` statement by replacing ``__builtin__.__import__`` with ``ImportEngine.__import__``.
Line 67: Line 67:
 ``load_module()`` ``import_module(name, package=None)``
 A reimplementation of ``importlib.import_module()`` which uses the import state stored in the ImportEngine instance. See [3]_ for full reference.
Line 69: Line 70:
 ``from_engine(self, other)``
 
 
Method to create a new import object from another ImportEngine instance. The new object is initialised with a copy of the state in other. When called on engine.sysengine, ``from_engine`` can be used to create an ImportEngine object with a **copy** of the global import state.
``from_engine(self, other)``
  Method to create a new import object from another ImportEngine instance. The new object is initialised with a copy of the state in other. When called on ``engine.sysengine`` as ``other``, ``from_engine`` can be used to create an ImportEngine object with a **copy** of the global import state.
Line 74: Line 74:
  Convenience class to provide engine-like access to the global state. Provides ``__import__()``, ``import_module()`` and ``from_engine()`` methods like ``ImportEngine`` but writes through to the global state in ``sys``.
Line 80: Line 81:
 instance of GlobalImportEngine  Instance of GlobalImportEngine provided for convenience (eg. for use by module finders and loaders).
Line 89: Line 90:
The engine parameter is optional so that the 'new style' finders and loaders can be made backwards compatible by falling back on engine.sysengine with the following simple pattern: The only difference between 'new style' and PEP 302 compatible finders/loaders The engine parameter is optional so that the 'new style' finders and loaders can be made backwards compatible by falling back on engine.sysengine with the following simple pattern:
Line 99: Line 100:
An implementation based on Brett Cannon's importlib has been developed by Greg Slodkowicz as part of the 2011 Google Summer of Code. The code repository is located at https://bitbucket.org/jergosh/gsoc_import_engine. An implementation based on Brett Cannon's importlib has been developed by Greg Slodkowicz as part of the 2011 Google Summer of Code. The code repository is located at https://bitbucket.org/jergosh/gsoc_import_engine/.
Line 104: Line 105:
The existing importlib implementation depends on several functions and object from imp, Python's builtin implementation of __import__. These
Naturally, this is a problem from the ImportEngine point of view. The offending functions are:
BuiltinImporter
ExtensionImporter
The existing importlib implementation depends on several functions and object from imp, Python's builtin implementation of __import__. These functions are unaware of ImportEngine and place the newly imported module in ``sys.modules``. Naturally, this is a problem from the ImportEngine point of view. The offending methods are:

* imp.init_builtin()
* imp.load_dynamic()

Ideally, these methods would be re-implemented to be aware of ImportEngine, but seeing as they provide low-level dynamic library loading, it would likely be difficult to do so in pure Python. Another option would be to equip these methods with extra parameters, either passing an ImportEngine instance directly or a modules dictionary in which to put the module. Such selective intervention into ``imp`` would be hard to justify.

Similarly, ``imp.NullImporter`` implements a ``load_module`` method which is incompatible with 'new style' loaders. Since the NullImporter class does next to nothing (i. e. always returns None), it has been reimplemented in Python. The only way this could cause problems would be explicitly checking if a module's importer is an imp.NullImporter (which occurs only in some unittests).

These problems would disappear if import engines were to be fully integrated into Python, i. e. on the level of the built-in __import__ as opposed to only into importlib.

Page for design and implementation details for the Import Engine GSoC 2011 project.

http://www.google-melange.com/gsoc/proposal/review/google/gsoc2011/gslodkowicz/1

PEP: XXX

Title: Python Import Engine

Version: $Revision$

Last-Modified: $Date$

Author: Nick Coghlan <ncoghlan@gmail.com>, Greg Slodkowicz <jergosh@gmail.com>

Status: Draft

Type: Standards Track

Content-Type: text/x-rst

Created: 4-Jul-2011

Post-History: XXX

Abstract

This PEP proposes incorporating an 'import engine' class which would encapsulate all state related to importing modules into a single object. Currently the bulk of importing work is done by means of module finders and loaders, and their interfaces would have to be updated for them to work with import engine objects. In that sense, this PEP constitutes an revision of finder and loader interfaces described in PEP 302 [1].

Rationale

Historically, any modification to the import functionality required re-implementing __import__() entirely. PEP 302 provides a major improvement by introducing separation between imports of different types of modules. As a result, additional process-global state stored in the sys module. This, along with earlier import-related state, comprises:

  • sys.modules
  • sys.path
  • sys.path_hooks
  • sys.meta_path
  • sys.path_importer_cache
  • the import lock (imp.lock_held()/acquire_lock()/release_lock())

Isolating this state would allow multiple import states to be conveniently stored within a process. Placing the import functionality in a self-contained object would allow subclassing to add additional features (e.g. module import notifications or fine-grained control over which modules can be imported). The engine would also be subclassed to use the import engine API to interact with the existing process-global state.

Proposal

We propose introducing an ImportEngine class to encapsulate import functionality. This includes the __import__() function which can be bootstrapped into Python to be executed when the import statement is encountered in the code and also load_module(), equivalent to imp.load_module() and importlib.load_module().

Since the new style finders and loaders should also have the option to modify the global import state, we introduce a GlobalImportState class with interface identical to ImportEngine but taking advantage of the global state. This can be easily implemented using class properties.

Design and Implementation

API

The proposed extension would consist of the following objects:

engine.ImportEngine
__import__(self, name, globals={}, locals={}, fromlist=[], level=0) Reimplementation of the builtin __import__() function. The import of a module will proceed using the state stored in the ImportEngine instance rather than the global import state. For full documentation of __import__ funtionality, see [2] . __import__() from ImportEngine and its subclasses can be used to customise the behaviour of the import statement by replacing __builtin__.__import__ with ImportEngine.__import__.
import_module(name, package=None)
A reimplementation of importlib.import_module() which uses the import state stored in the ImportEngine instance. See [3] for full reference.
from_engine(self, other)
Method to create a new import object from another ImportEngine instance. The new object is initialised with a copy of the state in other. When called on engine.sysengine as other, from_engine can be used to create an ImportEngine object with a copy of the global import state.

GlobalImportEngine(ImportEngine) Convenience class to provide engine-like access to the global state. Provides __import__(), import_module() and from_engine() methods like ImportEngine but writes through to the global state in sys.

Global variables

engine.sysengine
Instance of GlobalImportEngine provided for convenience (eg. for use by module finders and loaders).

Necessary changes to finder/loader interfaces:

find_module (cls, fullname, path=None, engine=None)

load module (cls, fullname, path=None, engine=None)

The only difference between 'new style' and PEP 302 compatible finders/loaders The engine parameter is optional so that the 'new style' finders and loaders can be made backwards compatible by falling back on engine.sysengine with the following simple pattern:

find_module(cls, fullname, path=None, engine=None)
  if not engine:
    engine = engine.sysengine

  ...

An implementation based on Brett Cannon's importlib has been developed by Greg Slodkowicz as part of the 2011 Google Summer of Code. The code repository is located at https://bitbucket.org/jergosh/gsoc_import_engine/.

Open Issues

The existing importlib implementation depends on several functions and object from imp, Python's builtin implementation of __import__. These functions are unaware of ImportEngine and place the newly imported module in sys.modules. Naturally, this is a problem from the ImportEngine point of view. The offending methods are:

  • imp.init_builtin()
  • imp.load_dynamic()

Ideally, these methods would be re-implemented to be aware of ImportEngine, but seeing as they provide low-level dynamic library loading, it would likely be difficult to do so in pure Python. Another option would be to equip these methods with extra parameters, either passing an ImportEngine instance directly or a modules dictionary in which to put the module. Such selective intervention into imp would be hard to justify.

Similarly, imp.NullImporter implements a load_module method which is incompatible with 'new style' loaders. Since the NullImporter class does next to nothing (i. e. always returns None), it has been reimplemented in Python. The only way this could cause problems would be explicitly checking if a module's importer is an imp.NullImporter (which occurs only in some unittests).

These problems would disappear if import engines were to be fully integrated into Python, i. e. on the level of the built-in __import__ as opposed to only into importlib.

References

[1]PEP 302, New Import Hooks, J van Rossum, Moore (http://www.python.org/dev/peps/pep-0302)
[2]__import__() builtin function, The Python Standard Library documentation (http://docs.python.org/library/functions.html#__import__)
[3]Importlib documentation, Cannon (http://docs.python.org/dev/library/importlib)

SummerOfCode/PythonImportEnginePlanning (last edited 2011-07-11 09:25:27 by 2001:878:200:1001:e2f8:47ff:fe30:30c8)

Unable to edit the page? See the FrontPage for instructions.