Differences between revisions 14 and 15
Revision 14 as of 2007-06-01 20:54:46
Size: 6654
Editor: trl
Comment:
Revision 15 as of 2007-06-13 12:16:16
Size: 7035
Editor: ext9
Comment: Added papyros
Deletions are marked like this. Additions are marked like this.
Line 25: Line 25:
 * [http://code.google.com/p/papyros/ papyros] - lightweight master-slave based parallel processing. Clients submit jobs to a master object which is monitored by one or more slave objects that do the real work. Two main implementations are currently provided, one using multiple threads and one multiple processes in one or more hosts through [http://pyro.sourceforge.net/ Pyro].

Parallel Processing and Multiprocessing in Python

A number of Python-related libraries exist for the programming of solutions either employing multiple CPUs or multicore CPUs in a [http://en.wikipedia.org/wiki/Symmetric_multiprocessing symmetric multiprocessing (SMP)] or shared memory environment, or potentially huge numbers of computers in a cluster or grid environment. This page seeks to provide references to the different libraries and solutions available.

Symmetric Multiprocessing

Some libraries, often to preserve some similarity with more familiar concurrency models (such as Python's threading API), employ parallel processing techniques which limit their relevance to SMP-based hardware, mostly due to the usage of process creation functions such as the UNIX fork system call. However, a technique called process migration may permit such libraries to be useful in certain kinds of computational clusters as well, notably single-system image cluster solutions ([http://openmosix.sourceforge.net/ OpenMosix] being one such example).

  • [http://lfw.org/python/delegate.html delegate] - fork-based process creation with pickled data sent through pipes

  • [http://www.python.org/pypi/parallel parallel/pprocess] - fork-based process creation with asynchronous channel-based communications employing pickled data (currently only POSIX/UNIX/Linux, perhaps Cygwin)

  • [http://poshmodule.sourceforge.net/ POSH] Python Object Sharing is an extension module to Python that allows objects to be placed in shared memory. POSH allows concurrent processes to communicate simply by assigning objects to shared container objects. (POSIX/UNIX/Linux only)

  • [http://www.parallelpython.com/ pp] - process-based, job-oriented solution with cluster support (Windows, Linux, Unix)

  • [http://www.python.org/pypi/processing processing] - fork-based process creation (using threads on other platforms), implementing an API like the standard library's threading API and providing familiar objects such as queues and semaphores through the use of a manager process (platform-independent - needs threading or a fork implementation)

  • [http://www.python.org/pypi/remoteD remoteD] - fork-based process creation with a dictionary-based communications paradigm (platform independent, according to PyPI entry)

Advantages of such approaches include convenient process creation and the ability to share resources. Indeed, the fork system call permits efficient sharing of common read-only data structures on modern UNIX-like operating systems.

Cluster Computing

Unlike SMP architectures and especially in contrast to thread-based concurrency, cluster (and grid) architectures offer high scalability due to the relative absence of shared resources, although this can make the programming paradigms seem somewhat alien to uninitiated developers. In this domain, some overlap with other distributed computing technologies may be observed (see DistributedProgramming for more details).

Grid Computing

Editorial Notes

The above lists should be arranged in ascending alphabetical order - please respect this when adding new frameworks or tools.

ParallelProcessing (last edited 2021-05-17 13:47:48 by MordicusEtCubitus)

Unable to edit the page? See the FrontPage for instructions.