Revision 22 as of 2011-09-26 20:00:14

Clear message

Join the Testing In Python (TIP) mailing list for Python testing tools discussions!

(!) This wiki page is originated from PyCheeseCake and it was originally created by Grig Gheorghiu

Unit Testing Tools

Tool

Category

Author

Part of

Claim to fame

unittest

unit testing

Steve Purcell

Python standard lib

first unit test framework to be included in Python standard library; easy to use by people familiar with the xUnit frameworks; strong support for test organization and reuse via test suites

doctest

unit testing

Tim Peters

Python standard lib

copy and paste output from shell session; Agile Documentation: unit tests themselves can serve as documentation when combined with epydoc; also see Django's approach

py.test

unit testing

Holger Krekel

py lib

no API!;automatic collection of tests; simple asserts; strong support for test fixture/state management via setup/teardown hooks; strong debugging support via customized traceback

nose

unittest extensions

Jason Pellerin

unit test framework, used most notably by TurboGears and Pylons; provides an alternate test discovery and running process for unittest, one that is intended to mimic the behavior of py.test as much as is reasonably possible without resorting to too much magic. More friendly with unittest.TestCase-based tests than py.test. There are also many plugins available.

testify

unittest extensions

Yelp team

unit test framework, provides Enhanced test fixture setup, Split test suites into buckets for easy parallelization, PEP8 naming conventions & Fancy color test runner with lots of logging / reporting option.

Trial

unittest extensions

Jonathan Lange

Twisted

Extension of unittest to support writing asynchronous unit tests using Deferreds and new result types ('skip' and 'todo'). Includes a command-line program that does test discovery and integrates with doctest and coverage.

subunit

unittest extensions

Robert Collins

Transparently adds support for running unittest test cases/suites in a separate process : prevents system wide changes by a test destabilising the test runner. It also allows reporting from tests in another process into the unittest framework, giving a single integrated test environment.

testresources

unittest extensions

Robert Collins

Provides a mechanism for managing 'resources' - expensive bits of infrastructure - that are needed by multiple tests. Resources are constructed and free on demand, but with an optional TestSuite?, the test run order is optimised to reduce the number of resource constructions and releases needed. Compatible with unittest.

testtools

unittest extensions

Jonathan Lange

Useful extensions to unittest derived from custom extensions by projects such as Twisted and Bazaar.

Sancho

unit testing

MEMS and Nanotechnology Exchange

MEMS Exchange tools

Sancho 2.1 runs tests, and provides output for tests that fail; Sancho 2.1 does not count tests passed or failed; targets projects that do not maintain failing tests

zope.testing

unit testing

Zope3 community

Powerful test runner that includes support for post-mortem debugging of test failures. Also includes profiling and coverage reporting. This is a standalone package that has no dependencies on Zope and works just fine with projects that don't use Zope.

pry

unit testing

Aldo Cortesi

Elegant unit testing framework with built-in coverage analysis, profiling, micro-benchmarking and a powerful command-line interface.

pythoscope

unit testing

Michal Kwiatkowski

Tool that will automatically, or semi-automatically, generate unit tests for legacy systems written in Python.

testlib

unittest extensions

Logilab

logilab-common

Gives more power to standard unittest. More assert* methods; support for module level setup/teardown; skip test feature...

pytest

tests runner

Logilab

logilab-common

Tests finder / runner. Selectivly run tests; Stop on first failure; Run pdb on failed tests; Colorized reports; Run tests with coverage / profile enabled...

dutest

unittest extensions

Olemis Lang

PyOOP package (distributed separately too)

An object oriented interface to retrieve unittest test cases out of doctests. Hides initialization from doctests by allowing setUp and tearDown for each interactive example. Allows control over all the options provided by doctest. Specialized classes allow selective test discovery across a package hierarchy.

The following tools are not currently being developed or maintained as far as we can see. They are here for completeness, with last activity date and an indication of what documentation there is. If you know better, please edit.

Tool

Last Activity

Category

Author

Claim to fame

TestOOB

2009

unittest extensions

Ori Peleg

unittest enhancements; test filtering via regex/glob patterns; reporting in XML/HTML; colorized output; runs pdb on failing tests; run in parallel in threads/processes; verbose asserts; report failures immediately; and a little more;

pyUnitPerf

2009

performance-based unit testing

Grig Gheorghiu

port of JUnitPerf; transparently adds performance testing capabilities to existing unitttest-based test suites

Peckcheck

2008

specification-based unit testing

Darius Bacon

like the unittest module, except that test methods may take arguments; you declare each argument with a default value which names a test-data generator and the peckcheck module will peck at your test methods with a bunch of generated values

testosterone

2008

unit testing

Chad Whitacre

A manly test runner; command-line and curses(3) interfaces; the iPod of test runners. (More seriously, makes it easier to browse and inspect your test failures)

qunittest

2004

unit testing

Edgewall

Easy-to-use (PyQt) GUI framework and application for use with the Python unit testing framework. It can be used to conveniently execute unit tests and suites, and then display the results in a useful fashion.

Mock Testing Tools

See also here for a side-by-side syntax comparison between some of the more popular tools in this space.

Tool

Category

Author

Claim to fame

Ludibrio

mocks, stubs, spy, and dummies

Gustavo Rezende

Elegant test doubles framework in Python (mocks, stubs, spy, and dummies)

Python Mock

mock testing

Dave Kirby

Python Mock enables the easy creation of mock objects that can be used to emulate the behaviour of any class that the code under test depends on. You can set up expectations about the calls that are made to the mock object, and examine the history of calls made. This makes it easier to unit test classes in isolation. Download it here.

PyMock

mock testing

Jeff Younker

PyMock is based on the Java easymock package. It uses a recording and replay model rather than using a specification language. Easymock lives up to its name compared to other mocking packages. PyMock takes advantage of python's dynamic nature for futher improvements.

mock

mock testing

Michael Foord aka Fuzzyman

Provides "action -> assertion" mocking patter, instead of standard "record -> replay" pattern

pMock

mock testing

Graham Carlyle

Inspired by the Java jMock library, pMock makes the writing of unit tests using mock object techniques easier. Development of pmock has long since stopped and so it can be considered dead.

minimock

mock testing

Ian Bicking

Embeds mock testing constructs inside doctest tests.

svnmock

mock testing

Collin Winter

enables easier testing of Python programs that make use of Subversion's Python bindings

Mocker

mock testing

Gustavo Niemeyer

Graceful platform for test doubles in Python (mocks, stubs, fakes, and dummies). Well-documented and fairly feature-complete.

Stubble

stub testing

Iwan Vosloo

Stubble allows you to write arbitrary classes for use as stubs instead of read classes while testing. Stubble lets you link a stub class loosely to the real class which it is a stub for. This information is then used to ensure that tests will break if there is a discrepancy between the interface supported by your stub class and that of the real class it stands in for.

Mox

mock testing

smiddlek, dglasser

Mox is based on EasyMock, a Java mock object framework. Mox will make mock objects for you, so you don't have to create your own! It mocks the public/protected interfaces of Python objects. You set up your mock objects expected behavior using a domain specific language (DSL), which makes it easy to use, understand, and refactor!

MockTest

mock testing

Tim Cuthbertson (gfxmonk)

Mocktest allows you to mock / stub objects and make expectations about the methods you expect to be called as well as the arguments they should be called with. Expectations are very readable and expressive, and checked automatically. Any stubbed methods are reverted after each test case. Still under development, so subject to change

Fudge

mock and stub testing

Kumar McMillan

A module for using fake objects (mocks, stubs, etc) to test real ones. Uses a declrative syntax like jMock whereby you set up expectations for how an object should be used. An error will raise if an expectation is not met.

Mockito for Python

mock and stub testing

Szczepan Faber

A port of the Mockito mocking framework to Python. (Technically speaking, Mockito is a Test Spy framework.)

CaptureMock

mock testing

Geoff Bache

True record-replay approach to mocking. Requires no coding, just telling it which modules/attributes you want to mock. Then stores the behaviour in an external file, which can be used to test the code without those modules installed.

Fuzz Testing Tools

According to Wikipedia, "fuzz testing" (or "fuzzing") is a software testing technique whose basic idea is to attach the inputs of a program to a source of random data ("fuzz"). If the program fails (for example, by crashing, or by failing built-in code assertions), then there are defects to correct. The great advantage of fuzz testing is that the test design is extremely simple, and free of preconceptions about system behavior.

Tool

Author

Claim to fame

Pester

Ivan Moore

Tests your tests by mutating source code and finding tests that don't fail!

Peach Fuzzer Framework

Michael Eddington

Peach can fuzz just about anything from .NET, COM/ActiveX, SQL, shared libraries/DLL's, network applications, web, you name it.

antiparser

dmckinney

The purpose of antiparser is to provide an API that can be used to model network protocols and file formats by their composite data types. Once a model has been created, the antiparser has various methods for creating random sets of data that deviates in ways that will ideally trigger software bugs or security vulnerabilities.

Taof (The Art Of Fuzzing)

rodrigomarcos

Taof is a GUI cross-platform Python generic network protocol fuzzer. It has been designed for minimizing set-up time during fuzzing sessions and it is specially useful for fast testing of proprietary or undocumented protocols.

Fusil (Fusil the fuzzer)

Victor Stinner

It helps to start process with a prepared environment (limit memory, environment variables, redirect stdout, etc.), start network client or server, and create mangled files. Fusil has many probes to detect program crash: watch process exit code, watch process stdout and syslog for text patterns (eg. "segmentation fault"), watch session duration, watch cpu usage (process and system load), etc.

Web Testing Tools

First, let's define some categories of Web testing tools:

Tool

Category

Author

Part of

Claim to fame

twill

Browser simulation & In-process

Titus Brown

offers simple commands for navigating Web pages, posting forms and asserting conditions; can be used as shell script or Python module; can be used for unit-testing your own Web app; stress-test functionality; port of PBP; uses John J. Lee's mechanize

webunit

Browser simulation

Richard Jones

HTTP, HTTPS, GET, POST, basic auth all handled; control over expected status codes; uses unittest as the underlying framework

FunkLoad

Browser simulation

Benoit Delbosc / Nuxeo

functional/performance/load/stress testing for Web applications; easy test creation using TCPWatch as proxy recorder; reporting capabilities; based on webunit

zope.testbrowser

Browser simulation

Benji York

Zope 3

designed to be used in doctests that mirror user actions (see the README.txt for examples); can also be used as standalone Web testing tool outside of Zope 3

webtest

Browser simulation

Robert Brewer

CherryPy

extensions to unittest for web frameworks; uses http://docs.python.org/lib/module-httplib.html httplib]

PAMIE

Browser automation

Robert Marchetti

automates Internet Explorer by manipulating IE's Document Object Model via COM

PyXPCOM

Browser automation

Mark Hammond

provides Python XPCOM bindings for automating Mozilla browsers

paste.test.fixture

In-process

Ian Bicking

Paste

lets you test your WSGI applications without a server; similar in feel to zope.testbrowser

DjangoTesting

In-process

Georg Bauer

DjangoStuff

very simple and small testing framework for Django applications, based on the ideas of the Ruby on Rails testing framework; fully based on unittest, so you don't need anything beside Django and Python to make use of it

ibofobi.utils.test

In-process

Sune Kirkeby

testing framwework for Django applications; built on top of doctest instead of unittest; fixtures are written in YAML not Python code; implemented simple request/response tests on top of Beautiful Soup

wsgi-intercept

In-process

Kumar McMillan

This library lets you intercept calls to any specific host/port combination and redirect them into a WSGI application

windmill

Browser Automation

Windmill Community

Cross-Platform and Cross-Browser web automation tool, supports Firefox, Safari, Chrome and IE on Mac, Linux and Windows. Written in Python, test authoring libraries in Python and JavaScript. Active community, #windmill on irc.freenode.org

Selenium

Browser Automation

Selenium Community

Cross-Platform and Cross-Browser web automation tool, supports Firefox, Safari, Chrome and IE on Mac, Linux and Windows. Written in Java with Python bindings. Active community, #selenium on irc.freenode.org Documentation

Acceptance/Business Logic Testing Tools

Tool

Author

Claim to fame

PyFIT

John Roth

port of FIT; supports both FIT and FitNesse; acceptance tests for business logic written as HTML or Wiki tables

TextTest

Geoff Bache

works by comparing plain text logged by programs with a previous 'gold standard' version of that text; the tool itself is written in Python but it can be used to test programs written in any language; comes with extensive self tests which serve as examples of how to use it. Plugins for testing GUIs (see PyUseCase)

Lettuce

Gabriel Falcão

port of http://cukes.info/ supporting acceptance tests in Cucumber format, including tabular feature files and JUnit xml output.

|| FitLoader || Ian Bicking || A (rather alpha-ish) library for loading tests from Excel or CSV f

Unable to edit the page? See the FrontPage for instructions.