Differences between revisions 10 and 11
Revision 10 as of 2014-04-18 04:53:56
Size: 3144
Comment: update link
Revision 11 as of 2014-04-18 04:54:58
Size: 3138
Comment: update link
Deletions are marked like this. Additions are marked like this.
Line 7: Line 7:
 * [[http://asperous.github.io/pipeless|Pipeless]] - "A simple Python library for building a basic data pipeline."  * [[http://asperous.us/pipeless/|Pipeless]] - "A simple Python library for building a basic data pipeline."

According to wikipedia: flow-based programming (FBP) is a programming paradigm that defines applications as networks of "black box" processes, which exchange data across predefined connections by message passing, where the connections are specified externally to the processes. These black box processes can be reconnected endlessly to form different applications without having to be changed internally. FBP is thus naturally component-oriented.

This page describes Python packages for FBP.

Flow Based Programming

  • Pyleaf - "An ASCII-art based pipeline language for Python. Produces intuitive diagrams in simple text. Supports memoization, parallelization, caching, code-data consistency, HTML report generation."

  • Pipeless - "A simple Python library for building a basic data pipeline."

  • Pype - "Pypes provides a scalable, standards based, extensible platform for building ETL solutions. Most commercial platforms have steep learning curves and try to generalize too much of the process. Pypes provides a simple yet powerful framework for designing custom data processing workflows using components you write. In turn, it takes care of scalability and scheduling semantics."

  • papy - "The papy package provides an implementation of the flow-based programming paradigm in Python that enables the construction and deployment of distributed workflows."

  • another pype - A simpler module for chaining operations

  • zFlow - "Flow-based Programming Library using Python generators, loosely based on J.P. Morrison's book of the same name."

  • PyF - "PyF is a python open source framework and platform dedicated to large data processing, mining, transforming, reporting and more."

  • Kamaelia - "In Kamaelia you build systems from simple components that talk to each other. This speeds development, massively aids maintenance and also means you build naturally concurrent software. It's intended to be accessible by any developer, including novices. It also makes it fun :)"

  • Ruffus - "The Ruffus python module provides automatic support for: Managing dependencies, Parallel jobs, Re-starting from arbitrary points, especially after errors, Display of the pipeline as a flowchart, Reporting"

  • Bein - "Bein is a workflow manager and miniature LIMS system built in the Bioinformatics and Biostatistics Core Facility of the EPFL. It fills the gap for the working scientist between the classical shell and big workflow managers like Galaxy and major LIMS systems like OpenBIS."

  • florun - "florun is a visual workflow editor and runner"

  • DAGPype - "DAGPype is a pipe-syntax framework for data processing from within scripts or the Python interactive interpreter"

FlowBasedProgramming (last edited 2014-04-18 04:54:58 by DaleAthanasias)

Unable to edit the page? See the FrontPage for instructions.