Differences between revisions 6 and 7
Revision 6 as of 2011-08-11 04:40:18
Size: 33215
Comment:
Revision 7 as of 2011-08-11 04:45:14
Size: 33866
Comment:
Deletions are marked like this. Additions are marked like this.
Line 3: Line 3:
Join the [http://lists.idyll.org/listinfo/testing-in-python Testing In Python (TIP)] mailing list for Python testing tools discussions! Join the [[http://lists.idyll.org/listinfo/testing-in-python| Testing In Python (TIP)]] mailing list for Python testing tools discussions!
Line 8: Line 8:
|| [http://pyunit.sourceforge.net/pyunit.html unittest] || unit testing || [http://www.pythonconsulting.com/ Steve Purcell] || [http://docs.python.org/lib/module-unittest.html Python standard lib] || first unit test framework to be included in Python standard library; easy to use by people familiar with the xUnit frameworks; strong support for test organization and reuse via test suites ||
|| [http://docs.python.org/lib/module-doctest.html doctest] || unit testing || [http://www.python.org/peps/pep-0020.html Tim Peters] || [http://docs.python.org/lib/module-doctest.html Python standard lib] || copy and paste output from shell session; [http://agiletesting.blogspot.com/2005/02/agile-documentation-with-doctest-and.html Agile Documentation]: unit tests themselves can serve as documentation when combined with epydoc; also see [http://agiletesting.blogspot.com/2005/08/agile-documentation-in-django-project.html Django's approach] ||
|| [http://codespeak.net/py/current/doc/test.html py.test] || unit testing || [http://codespeak.net/ Holger Krekel] || [http://codespeak.net/py/current/doc/home.html py lib] || no API!;automatic collection of tests; simple asserts; strong support for test fixture/state management via setup/teardown hooks; strong debugging support via customized traceback ||
|| [http://testoob.sourceforge.net/ TestOOB] || unittest extensions || [http://freshmeat.net/%7Eoripel/ Ori Peleg] || || unittest enhancements; test filtering via regex/glob patterns; reporting in XML/HTML; colorized output; runs pdb on failing tests; run in parallel in threads/processes; verbose asserts; report failures immediately; and a little more; ||
|| [http://somethingaboutorange.com/mrl/projects/nose/ nose] || unittest extensions || [http://somethingaboutorange.com/mrl/ Jason Pellerin] || || unit test framework, used most notably by [http://www.turbogears.org/ TurboGears] and [http://pylonshq.com/ Pylons]; provides an alternate test discovery and running process for unittest, one that is intended to mimic the behavior of py.test as much as is reasonably possible without resorting to too much magic. More friendly with unittest.TestCase-based tests than py.test. There are also many [http://nose-plugins.jottit.com/ plugins] available. ||
|| [http://twistedmatrix.com/trac/wiki/TwistedTrial Trial] || unittest extensions || [http://mumak.net/wiki Jonathan Lange] || [http://www.twistedmatrix.com/ Twisted] || Extension of unittest to support writing asynchronous unit tests using Deferreds and new result types ('skip' and 'todo'). Includes a command-line program that does test discovery and integrates with doctest and coverage. ||
|| [http://www.robertcollins.net/unittest/subunit subunit] || unittest extensions || [http://www.robertcollins.net/ Robert Collins] || || Transparently adds support for running unittest test cases/suites in a separate process : prevents system wide changes by a test destabilising the test runner. It also allows reporting from tests in another process into the unittest framework, giving a single integrated test environment. ||
|| [http://www.robertcollins.net/unittest/testresources testresources] || unittest extensions || [http://www.robertcollins.net/ Robert Collins] || || Provides a mechanism for managing 'resources' - expensive bits of infrastructure - that are needed by multiple tests. Resources are constructed and free on demand, but with an optional [http://pycheesecake.org/wiki/TestSuite TestSuite?], the test run order is optimised to reduce the number of resource constructions and releases needed. Compatible with unittest. ||
|| [https://launchpad.net/testtools testtools] || unittest extensions || [http://mumak.net/wiki Jonathan Lange] || || Useful extensions to unittest derived from custom extensions by projects such as Twisted and Bazaar. ||
|| [http://www.mems-exchange.org/software/sancho/ Sancho] || unit testing || [http://www.mems-exchange.org/ MEMS and Nanotechnology Exchange] || [http://www.mems-exchange.org/software/ MEMS Exchange tools] || Sancho 2.1 runs tests, and provides output for tests that fail; Sancho 2.1 does not count tests passed or failed; targets projects that do not maintain failing tests ||
|| [http://sourceforge.net/projects/pyunitperf pyUnitPerf] || performance-based unit testing || [http://agiletesting.blogspot.com/ Grig Gheorghiu] || || port of [http://www.clarkware.com/software/JUnitPerf.html JUnitPerf]; transparently adds [http://agiletesting.blogspot.com/2004/12/performance-testing-with-pyunitperf.html performance testing capabilities] to existing unitttest-based test suites ||
|| [http://www.accesscom.com/%7Edarius/software/clickcheck.html Peckcheck] || specification-based unit testing || [http://www.accesscom.com/%7Edarius/personal/contact.html Darius Bacon] || || like the unittest module, except that test methods may take arguments; you declare each argument with a default value which names a test-data generator and the peckcheck module will peck at your test methods with a bunch of generated values ||
|| [http://projects.edgewall.com/qunittest/ qunittest] || unit testing || [http://www.edgewall.com/ Edgewall] || || Easy-to-use (PyQt) GUI framework and application for use with the Python unit testing framework. It can be used to conveniently execute unit tests and suites, and then display the results in a useful fashion. ||
|| [http://www.zetadev.com/software/testosterone/ testosterone] || unit testing || [http://www.zetadev.com/ Chad Whitacre] || || A manly test runner; command-line and curses(3) interfaces; the iPod of test runners. (More seriously, makes it easier to browse and inspect your test failures) ||
|| [http://cheeseshop.python.org/pypi/zope.testing zope.testing] || unit testing || Zope3 community || || Powerful test runner that includes support for post-mortem debugging of test failures. Also includes profiling and coverage reporting. This is a standalone package that has no dependencies on Zope and works just fine with projects that don't use Zope. ||
|| [http://dev.nullcube.com/ pry] || unit testing || [http://www.nullcube.com/ Aldo Cortesi] || || Elegant unit testing framework with built-in coverage analysis, profiling, micro-benchmarking and a powerful command-line interface. ||
|| [http://pythoscope.org/ pythoscope] || unit testing || [http://joker.linuxstuff.pl/ Michal Kwiatkowski] || || Tool that will automatically, or semi-automatically, generate unit tests for legacy systems written in Python. ||
|| [http://www.logilab.org/project/logilab-common/ testlib] || unittest extensions || [http://www.logilab.fr/ Logilab] || logilab-common || Gives more power to standard unittest. More assert* methods; support for module level setup/teardown; skip test feature... ||
|| [http://www.logilab.org/project/logilab-common/ pytest] || tests runner || [http://www.logilab.fr/ Logilab] || logilab-common || Tests finder / runner. Selectivly run tests; Stop on first failure; Run pdb on failed tests; Colorized reports; Run tests with coverage / profile enabled... ||
|| [http://pypi.python.org/pypi/dutest dutest] || unittest extensions || [https://opensvn.csie.org/traccgi/swlcu/ Olemis Lang] || [http://flioops.sf.net/ PyOOP package] (distributed separately too) || An object oriented interface to retrieve unittest test cases out of doctests. Hides initialization from doctests by allowing setUp and tearDown for each interactive example. Allows control over all the options provided by doctest. Specialized classes allow selective test discovery across a package hierarchy. ||
|| [[http://pyunit.sourceforge.net/pyunit.html| unittest]] || unit testing || [[http://www.pythonconsulting.com/| Steve Purcell]] || [[http://docs.python.org/lib/module-unittest.html| Python standard lib]] || first unit test framework to be included in Python standard library; easy to use by people familiar with the xUnit frameworks; strong support for test organization and reuse via test suites ||
|| [[http://docs.python.org/lib/module-doctest.html| doctest]] || unit testing || [[http://www.python.org/peps/pep-0020.html| Tim Peters]] || [[http://docs.python.org/lib/module-doctest.html| Python standard lib]] || copy and paste output from shell session; [[http://agiletesting.blogspot.com/2005/02/agile-documentation-with-doctest-and.html| Agile Documentation]]: unit tests themselves can serve as documentation when combined with epydoc; also see [[http://agiletesting.blogspot.com/2005/08/agile-documentation-in-django-project.html| Django's approach]] ||
|| [[http://codespeak.net/py/current/doc/test.html| py.test]] || unit testing || [[http://codespeak.net/| Holger Krekel]] || [[http://codespeak.net/py/current/doc/home.html| py lib]] || no API!;automatic collection of tests; simple asserts; strong support for test fixture/state management via setup/teardown hooks; strong debugging support via customized traceback ||
|| [[http://testoob.sourceforge.net/| TestOOB]] || unittest extensions || [[http://freshmeat.net/%7Eoripel/| Ori Peleg]] || || unittest enhancements; test filtering via regex/glob patterns; reporting in XML/HTML; colorized output; runs pdb on failing tests; run in parallel in threads/processes; verbose asserts; report failures immediately; and a little more; ||
|| [[http://somethingaboutorange.com/mrl/projects/nose/| nose]] || unittest extensions || [[http://somethingaboutorange.com/mrl/| Jason Pellerin]] || || unit test framework, used most notably by [[http://www.turbogears.org/| TurboGears]] and [[http://pylonshq.com/| Pylons]]; provides an alternate test discovery and running process for unittest, one that is intended to mimic the behavior of py.test as much as is reasonably possible without resorting to too much magic. More friendly with unittest.TestCase-based tests than py.test. There are also many [[http://nose-plugins.jottit.com/| plugins]] available. ||
|| [[http://twistedmatrix.com/trac/wiki/TwistedTrial| Trial]] || unittest extensions || [[http://mumak.net/wiki| Jonathan Lange]] || [[http://www.twistedmatrix.com/| Twisted]] || Extension of unittest to support writing asynchronous unit tests using Deferreds and new result types ('skip' and 'todo'). Includes a command-line program that does test discovery and integrates with doctest and coverage. ||
|| [[http://www.robertcollins.net/unittest/subunit| subunit]] || unittest extensions || [[http://www.robertcollins.net/| Robert Collins]] || || Transparently adds support for running unittest test cases/suites in a separate process : prevents system wide changes by a test destabilising the test runner. It also allows reporting from tests in another process into the unittest framework, giving a single integrated test environment. ||
|| [[http://www.robertcollins.net/unittest/testresources| testresources]] || unittest extensions || [[http://www.robertcollins.net/| Robert Collins]] || || Provides a mechanism for managing 'resources' - expensive bits of infrastructure - that are needed by multiple tests. Resources are constructed and free on demand, but with an optional [[http://pycheesecake.org/wiki/TestSuite| TestSuite?]], the test run order is optimised to reduce the number of resource constructions and releases needed. Compatible with unittest. ||
|| [[https://launchpad.net/testtools| testtools]] || unittest extensions || [[http://mumak.net/wiki| Jonathan Lange]] || || Useful extensions to unittest derived from custom extensions by projects such as Twisted and Bazaar. ||
|| [[http://www.mems-exchange.org/software/sancho/| Sancho]] || unit testing || [[http://www.mems-exchange.org/| MEMS and Nanotechnology Exchange]] || [[http://www.mems-exchange.org/software/| MEMS Exchange tools]] || Sancho 2.1 runs tests, and provides output for tests that fail; Sancho 2.1 does not count tests passed or failed; targets projects that do not maintain failing tests ||
|| [[http://sourceforge.net/projects/pyunitperf| pyUnitPerf]] || performance-based unit testing || [[http://agiletesting.blogspot.com/| Grig Gheorghiu]] || || port of [[http://www.clarkware.com/software/JUnitPerf.html| JUnitPerf]]; transparently adds [[http://agiletesting.blogspot.com/2004/12/performance-testing-with-pyunitperf.html| performance testing capabilities]] to existing unitttest-based test suites ||
|| [[http://www.accesscom.com/%7Edarius/software/clickcheck.html| Peckcheck]] || specification-based unit testing || [[http://www.accesscom.com/%7Edarius/personal/contact.html| Darius Bacon]] || || like the unittest module, except that test methods may take arguments; you declare each argument with a default value which names a test-data generator and the peckcheck module will peck at your test methods with a bunch of generated values ||
|| [[http://projects.edgewall.com/qunittest/| qunittest]] || unit testing || [[http://www.edgewall.com/| Edgewall]] || || Easy-to-use (PyQt) GUI framework and application for use with the Python unit testing framework. It can be used to conveniently execute unit tests and suites, and then display the results in a useful fashion. ||
|| [[http://www.zetadev.com/software/testosterone/| testosterone]] || unit testing || [[http://www.zetadev.com/| Chad Whitacre]] || || A manly test runner; command-line and curses(3) interfaces; the iPod of test runners. (More seriously, makes it easier to browse and inspect your test failures) ||
|| [[http://cheeseshop.python.org/pypi/zope.testing| zope.testing]] || unit testing || Zope3 community || || Powerful test runner that includes support for post-mortem debugging of test failures. Also includes profiling and coverage reporting. This is a standalone package that has no dependencies on Zope and works just fine with projects that don't use Zope. ||
|| [[http://dev.nullcube.com/| pry]] || unit testing || [[http://www.nullcube.com/| Aldo Cortesi]] || || Elegant unit testing framework with built-in coverage analysis, profiling, micro-benchmarking and a powerful command-line interface. ||
|| [[http://pythoscope.org/| pythoscope]] || unit testing || [[http://joker.linuxstuff.pl/| Michal Kwiatkowski]] || || Tool that will automatically, or semi-automatically, generate unit tests for legacy systems written in Python. ||
|| [[http://www.logilab.org/project/logilab-common/| testlib]] || unittest extensions || [[http://www.logilab.fr/| Logilab]] || logilab-common || Gives more power to standard unittest. More assert* methods; support for module level setup/teardown; skip test feature... ||
|| [[http://www.logilab.org/project/logilab-common/| pytest]] || tests runner || [[http://www.logilab.fr/| Logilab]] || logilab-common || Tests finder / runner. Selectivly run tests; Stop on first failure; Run pdb on failed tests; Colorized reports; Run tests with coverage / profile enabled... ||
|| [[http://pypi.python.org/pypi/dutest| dutest]] || unittest extensions || [[https://opensvn.csie.org/traccgi/swlcu/| Olemis Lang]] || [[http://flioops.sf.net/| PyOOP package]] (distributed separately too) || An object oriented interface to retrieve unittest test cases out of doctests. Hides initialization from doctests by allowing setUp and tearDown for each interactive example. Allows control over all the options provided by doctest. Specialized classes allow selective test discovery across a package hierarchy. ||
Line 32: Line 32:
|| [http://github.com/nsigustavo/ludibrio/ Ludibrio] || mocks, stubs, spy, and dummies || Gustavo Rezende || Elegant test doubles framework in Python (mocks, stubs, spy, and dummies) ||
|| [http://python-mock.sourceforge.net/ Python Mock] || mock testing || [http://www.thedeveloperscoach.com/ Dave Kirby] || Python Mock enables the easy creation of mock objects that can be used to emulate the behaviour of any class that the code under test depends on. You can set up expectations about the calls that are made to the mock object, and examine the history of calls made. This makes it easier to unit test classes in isolation. [http://sourceforge.net/projects/python-mock/ Download it here.] ||
|| [http://theblobshop.com/pymock PyMock] || mock testing || [http://theblobshop.com/ Jeff Younker] || PyMock is based on the Java [http://www.easymock.org/ easymock] package. It uses a recording and replay model rather than using a specification language. Easymock lives up to its name compared to other mocking packages. PyMock takes advantage of python's dynamic nature for futher improvements. ||
|| [http://www.voidspace.org.uk/python/mock.html mock] || mock testing || Michael Foord aka [http://www.voidspace.org.uk/python/weblog/index.shtml Fuzzyman] || Provides "action -> assertion" mocking patter, instead of standard "record -> replay" pattern ||
|| [http://pmock.sourceforge.net/ pMock] || mock testing || Graham Carlyle || Inspired by the Java [http://www.jmock.org/ jMock] library, pMock makes the writing of unit tests using mock object techniques easier. Development of pmock has long since stopped and so it can be considered dead. ||
|| [http://blog.ianbicking.org/minimock.html minimock] || mock testing || [http://blog.ianbicking.org/ Ian Bicking] || Embeds mock testing constructs inside doctest tests. ||
|| [http://oakwinter.com/code/svnmock/ svnmock] || mock testing || [http://oakwinter.com/code/ Collin Winter] || enables easier testing of Python programs that make use of [http://subversion.tigris.org/ Subversion's] Python bindings ||
|| [http://labix.org/mocker Mocker] || mock testing || [http://blog.labix.org/ Gustavo Niemeyer] || Graceful platform for test doubles in Python (mocks, stubs, fakes, and dummies). Well-documented and fairly feature-complete. ||
|| [http://www.reahl.org/project?name=stubble Stubble] || stub testing || [http://www.reahl.org/ Iwan Vosloo] || Stubble allows you to write arbitrary classes for use as stubs instead of read classes while testing. Stubble lets you link a stub class loosely to the real class which it is a stub for. This information is then used to ensure that tests will break if there is a discrepancy between the interface supported by your stub class and that of the real class it stands in for. ||
|| [http://code.google.com/p/pymox/ Mox] || mock testing || smiddlek, dglasser || Mox is based on EasyMock, a Java mock object framework. Mox will make mock objects for you, so you don't have to create your own! It mocks the public/protected interfaces of Python objects. You set up your mock objects expected behavior using a domain specific language (DSL), which makes it easy to use, understand, and refactor! ||
|| [http://github.com/gfxmonk/mocktest/tree/master MockTest] || mock testing || Tim Cuthbertson (gfxmonk) || Mocktest allows you to mock / stub objects and make expectations about the methods you expect to be called as well as the arguments they should be called with. Expectations are very readable and expressive, and checked automatically. Any stubbed methods are reverted after each test case. Still under development, so subject to change ||
|| [http://farmdev.com/projects/fudge/ Fudge] || mock and stub testing || [http://farmdev.com/ Kumar McMillan] || A module for using fake objects (mocks, stubs, etc) to test real ones. Uses a declrative syntax like jMock whereby you set up expectations for how an object should be used. An error will raise if an expectation is not met. ||
|| [http://code.google.com/p/mockito/wiki/MockitoForPython Mockito for Python] || mock and stub testing || [http://monkeyisland.pl/ Szczepan Faber] || A port of the [http://code.google.com/p/mockito/ Mockito] mocking framework to Python. (Technically speaking, Mockito is a Test Spy framework.) ||
|| [[http://github.com/nsigustavo/ludibrio/| Ludibrio]] || mocks, stubs, spy, and dummies || Gustavo Rezende || Elegant test doubles framework in Python (mocks, stubs, spy, and dummies) ||
|| [[http://python-mock.sourceforge.net/| Python Mock]] || mock testing || [[http://www.thedeveloperscoach.com/| Dave Kirby]] || Python Mock enables the easy creation of mock objects that can be used to emulate the behaviour of any class that the code under test depends on. You can set up expectations about the calls that are made to the mock object, and examine the history of calls made. This makes it easier to unit test classes in isolation. [[http://sourceforge.net/projects/python-mock/| Download it here.]] ||
|| [[http://theblobshop.com/pymock| PyMock]] || mock testing || [[http://theblobshop.com/| Jeff Younker]] || PyMock is based on the Java [[http://www.easymock.org/| easymock]] package. It uses a recording and replay model rather than using a specification language. Easymock lives up to its name compared to other mocking packages. PyMock takes advantage of python's dynamic nature for futher improvements. ||
|| [[http://www.voidspace.org.uk/python/mock.html| mock]] || mock testing || Michael Foord aka [[http://www.voidspace.org.uk/python/weblog/index.shtml| Fuzzyman]] || Provides "action -> assertion" mocking patter, instead of standard "record -> replay" pattern ||
|| [[http://pmock.sourceforge.net/| pMock]] || mock testing || Graham Carlyle || Inspired by the Java [[http://www.jmock.org/| jMock]] library, pMock makes the writing of unit tests using mock object techniques easier. Development of pmock has long since stopped and so it can be considered dead. ||
|| [[http://blog.ianbicking.org/minimock.html| minimock]] || mock testing || [[http://blog.ianbicking.org/| Ian Bicking]] || Embeds mock testing constructs inside doctest tests. ||
|| [[http://oakwinter.com/code/svnmock/| svnmock]] || mock testing || [[http://oakwinter.com/code/| Collin Winter]] || enables easier testing of Python programs that make use of [[http://subversion.tigris.org/| Subversion's]] Python bindings ||
|| [[http://labix.org/mocker| Mocker]] || mock testing || [[http://blog.labix.org/| Gustavo Niemeyer]] || Graceful platform for test doubles in Python (mocks, stubs, fakes, and dummies). Well-documented and fairly feature-complete. ||
|| [[http://www.reahl.org/project?name=stubble| Stubble]] || stub testing || [[http://www.reahl.org/| Iwan Vosloo]] || Stubble allows you to write arbitrary classes for use as stubs instead of read classes while testing. Stubble lets you link a stub class loosely to the real class which it is a stub for. This information is then used to ensure that tests will break if there is a discrepancy between the interface supported by your stub class and that of the real class it stands in for. ||
|| [[http://code.google.com/p/pymox/| Mox]] || mock testing || smiddlek, dglasser || Mox is based on EasyMock, a Java mock object framework. Mox will make mock objects for you, so you don't have to create your own! It mocks the public/protected interfaces of Python objects. You set up your mock objects expected behavior using a domain specific language (DSL), which makes it easy to use, understand, and refactor! ||
|| [[http://github.com/gfxmonk/mocktest/tree/master| MockTest]] || mock testing || Tim Cuthbertson (gfxmonk) || Mocktest allows you to mock / stub objects and make expectations about the methods you expect to be called as well as the arguments they should be called with. Expectations are very readable and expressive, and checked automatically. Any stubbed methods are reverted after each test case. Still under development, so subject to change ||
|| [[http://farmdev.com/projects/fudge/| Fudge]] || mock and stub testing || [[http://farmdev.com/| Kumar McMillan]] || A module for using fake objects (mocks, stubs, etc) to test real ones. Uses a declrative syntax like jMock whereby you set up expectations for how an object should be used. An error will raise if an expectation is not met. ||
|| [[http://code.google.com/p/mockito/wiki/MockitoForPython| Mockito for Python]] || mock and stub testing || [[http://monkeyisland.pl/| Szczepan Faber]] || A port of the [[http://code.google.com/p/mockito/| Mockito]] mocking framework to Python. (Technically speaking, Mockito is a Test Spy framework.) ||
Line 48: Line 48:
According to Wikipedia, [http://en.wikipedia.org/wiki/Fuzz_testing "fuzz testing" (or "fuzzing")] is a software testing technique whose basic idea is to attach the inputs of a program to a source of random data ("fuzz"). If the program fails (for example, by crashing, or by failing built-in code assertions), then there are defects to correct. The great advantage of fuzz testing is that the test design is extremely simple, and free of preconceptions about system behavior. According to Wikipedia, [[http://en.wikipedia.org/wiki/Fuzz_testing| "fuzz testing" (or "fuzzing")]] is a software testing technique whose basic idea is to attach the inputs of a program to a source of random data ("fuzz"). If the program fails (for example, by crashing, or by failing built-in code assertions), then there are defects to correct. The great advantage of fuzz testing is that the test design is extremely simple, and free of preconceptions about system behavior.
Line 51: Line 51:
|| [http://jester.sourceforge.net/ Pester] || Ivan Moore || Tests your tests by mutating source code and finding tests that don't fail! ||
|| [http://peachfuzz.sourceforge.net/ Peach Fuzzer Framework] || Michael Eddington || Peach can fuzz just about anything from .NET, COM/ActiveX, SQL, shared libraries/DLL's, network applications, web, you name it. ||
|| [http://antiparser.sourceforge.net/ antiparser] || dmckinney || The purpose of antiparser is to provide an API that can be used to model network protocols and file formats by their composite data types. Once a model has been created, the antiparser has various methods for creating random sets of data that deviates in ways that will ideally trigger software bugs or security vulnerabilities. ||
|| [http://sourceforge.net/projects/taof Taof (The Art Of Fuzzing)] || rodrigomarcos || Taof is a GUI cross-platform Python generic network protocol fuzzer. It has been designed for minimizing set-up time during fuzzing sessions and it is specially useful for fast testing of proprietary or undocumented protocols. ||
|| [http://fusil.hachoir.org/trac Fusil (Fusil the fuzzer)] || [http://www.haypocalc.com/wiki/Victor_Stinner Victor Stinner] || It helps to start process with a prepared environment (limit memory, environment variables, redirect stdout, etc.), start network client or server, and create mangled files. Fusil has many probes to detect program crash: watch process exit code, watch process stdout and syslog for text patterns (eg. "segmentation fault"), watch session duration, watch cpu usage (process and system load), etc. ||
|| [[http://jester.sourceforge.net/| Pester]] || Ivan Moore || Tests your tests by mutating source code and finding tests that don't fail! ||
|| [[http://peachfuzz.sourceforge.net/| Peach Fuzzer Framework]] || Michael Eddington || Peach can fuzz just about anything from .NET, COM/ActiveX, SQL, shared libraries/DLL's, network applications, web, you name it. ||
|| [[http://antiparser.sourceforge.net/| antiparser]] || dmckinney || The purpose of antiparser is to provide an API that can be used to model network protocols and file formats by their composite data types. Once a model has been created, the antiparser has various methods for creating random sets of data that deviates in ways that will ideally trigger software bugs or security vulnerabilities. ||
|| [[http://sourceforge.net/projects/taof| Taof (The Art Of Fuzzing)]] || rodrigomarcos || Taof is a GUI cross-platform Python generic network protocol fuzzer. It has been designed for minimizing set-up time during fuzzing sessions and it is specially useful for fast testing of proprietary or undocumented protocols. ||
|| [[http://fusil.hachoir.org/trac| Fusil (Fusil the fuzzer)]] || [[http://www.haypocalc.com/wiki/Victor_Stinner| Victor Stinner]] || It helps to start process with a prepared environment (limit memory, environment variables, redirect stdout, etc.), start network client or server, and create mangled files. Fusil has many probes to detect program crash: watch process exit code, watch process stdout and syslog for text patterns (eg. "segmentation fault"), watch session duration, watch cpu usage (process and system load), etc. ||
Line 63: Line 63:
  * '''In-process''' or unit-test-type tools ([[http://bit.ly/n42I1X courtesy of Ian Bicking]]): call an application in the same process, instead of generating an HTTP request; so an exception in the application would go all the way up to the command runner (py.test, unittest, etc).   * '''In-process''' or unit-test-type tools ([[[http://bit.ly/n42I1X| courtesy of Ian Bicking]]]): call an application in the same process, instead of generating an HTTP request; so an exception in the application would go all the way up to the command runner (py.test, unittest, etc).
Line 67: Line 67:
|| [http://twill.idyll.org/ twill] || Browser simulation & In-process || [http://www.advogato.org/person/titus/ Titus Brown] || || offers simple commands for navigating Web pages, posting forms and asserting conditions; can be used as shell script or Python module; can be used for [http://www.idyll.org/%7Et/www-tools/twill.html#unit-testing unit-testing your own Web app]; stress-test functionality; port of [http://pbp.berlios.de/ PBP]; uses John J. Lee's [http://wwwsearch.sf.net/ mechanize] ||
|| [http://mechanicalcat.net/tech/webunit/ webunit] || Browser simulation || [http://www.mechanicalcat.net/richard/log Richard Jones] || || HTTP, HTTPS, GET, POST, basic auth all handled; control over expected status codes; uses unittest as the underlying framework ||
|| [http://funkload.nuxeo.org/ FunkLoad] || Browser simulation || Benoit Delbosc / [http://www.nuxeo.com/ Nuxeo] || || functional/performance/load/stress testing for Web applications; easy test creation using [http://hathawaymix.org/Software/TCPWatch/ TCPWatch] as proxy recorder; reporting capabilities; based on [http://mechanicalcat.net/tech/webunit/ webunit] ||
|| [http://cheeseshop.python.org/pypi?:action=display&name=zope.testbrowser zope.testbrowser] || Browser simulation || [http://www.benjiyork.com/ Benji York] || [http://www.zope.org/DevHome/Wikis/DevSite/Projects/ComponentArchitecture/FrontPage Zope 3] || designed to be used in doctests that mirror user actions (see the README.txt for [http://svn.zope.org/*checkout*/Zope3/trunk/src/zope/testbrowser/README.txt examples]); can also be used as standalone Web testing tool outside of Zope 3 ||
|| [http://www.cherrypy.org/file/trunk/cherrypy/test/webtest.py webtest] || Browser simulation || [http://www.aminus.org/blogs/index.php/fumanchu Robert Brewer] || [http://www.cherrypy.org/ CherryPy] || extensions to unittest for web frameworks; uses http://docs.python.org/lib/module-httplib.html httplib] ||
|| [http://pamie.sourceforge.net/ PAMIE] || Browser automation || Robert Marchetti || || automates Internet Explorer by manipulating IE's Document Object Model via COM ||
|| [http://kb.mozillazine.org/PyXPCOM PyXPCOM] || Browser automation || [http://starship.python.net/crew/mhammond/ Mark Hammond] || || provides Python XPCOM bindings for automating Mozilla browsers ||
|| [http://pythonpaste.org/testing-applications.html paste.test.fixture] || In-process || [http://blog.ianbicking.org/ Ian Bicking] || [http://pythonpaste.org/ Paste] || lets you test your [http://www.python.org/peps/pep-0333.html WSGI] applications without a server; similar in feel to zope.testbrowser ||
|| [http://simon.bofh.ms/cgi-bin/trac-django-projects.cgi/wiki/DjangoTesting DjangoTesting] || In-process || [http://hugo.muensterland.org/ Georg Bauer] || [http://simon.bofh.ms/cgi-bin/trac-django-projects.cgi/wiki/DjangoStuff DjangoStuff] || very simple and small testing framework for [http://www.djangoproject.com/ Django] applications, based on the ideas of the [http://www.rubyonrails.org/ Ruby on Rails] testing framework; fully based on unittest, so you don't need anything beside Django and Python to make use of it ||
|| [http://ibofobi.dk/blog/archive/2005/11/django-doctest/ ibofobi.utils.test] || In-process || [http://ibofobi.dk/blog/ Sune Kirkeby] || || testing framwework for [http://www.djangoproject.com/ Django] applications; built on top of doctest instead of unittest; fixtures are written in YAML not Python code; implemented simple request/response tests on top of [http://www.crummy.com/software/BeautifulSoup/ Beautiful Soup] ||
|| [http://code.google.com/p/wsgi-intercept/ wsgi-intercept] || In-process || [http://code.google.com/u/kumar.mcmillan/ Kumar McMillan] || || This library lets you intercept calls to any specific host/port combination and redirect them into a WSGI application ||
|| [http://www.getwindmill.com/ windmill] || Browser Automation || [http://www.getwindmill.com/ Windmill Community] || || Cross-Platform and Cross-Browser web automation tool, supports Firefox, Safari, Chrome and IE on Mac, Linux and Windows. Written in Python, test authoring libraries in Python and JavaScript. Active community, #windmill on irc.freenode.org ||
|| [[http://twill.idyll.org/| twill]] || Browser simulation & In-process || [[http://www.advogato.org/person/titus/| Titus Brown]] || || offers simple commands for navigating Web pages, posting forms and asserting conditions; can be used as shell script or Python module; can be used for [[http://www.idyll.org/%7Et/www-tools/twill.html#unit-testing| unit-testing your own Web app]]; stress-test functionality; port of [[http://pbp.berlios.de/| PBP]]; uses John J. Lee's [[http://wwwsearch.sf.net/| mechanize]] ||
|| [[http://mechanicalcat.net/tech/webunit/| webunit]] || Browser simulation || [[http://www.mechanicalcat.net/richard/log| Richard Jones]] || || HTTP, HTTPS, GET, POST, basic auth all handled; control over expected status codes; uses unittest as the underlying framework ||
|| [[http://funkload.nuxeo.org/| FunkLoad]] || Browser simulation || Benoit Delbosc / [[http://www.nuxeo.com/| Nuxeo]] || || functional/performance/load/stress testing for Web applications; easy test creation using [[http://hathawaymix.org/Software/TCPWatch/| TCPWatch]] as proxy recorder; reporting capabilities; based on [[http://mechanicalcat.net/tech/webunit/| webunit]] ||
|| [[http://cheeseshop.python.org/pypi?:action=display&name=zope.testbrowser| zope.testbrowser]] || Browser simulation || [[http://www.benjiyork.com/| Benji York]] || [[http://www.zope.org/DevHome/Wikis/DevSite/Projects/ComponentArchitecture/FrontPage| Zope 3]] || designed to be used in doctests that mirror user actions (see the README.txt for [[http://svn.zope.org/*checkout*/Zope3/trunk/src/zope/testbrowser/README.txt| examples]]); can also be used as standalone Web testing tool outside of Zope 3 ||
|| [[http://www.cherrypy.org/file/trunk/cherrypy/test/webtest.py| webtest]] || Browser simulation || [[http://www.aminus.org/blogs/index.php/fumanchu| Robert Brewer]] || [[http://www.cherrypy.org/| CherryPy]] || extensions to unittest for web frameworks; uses http://docs.python.org/lib/module-httplib.html httplib] ||
|| [[http://pamie.sourceforge.net/| PAMIE]] || Browser automation || Robert Marchetti || || automates Internet Explorer by manipulating IE's Document Object Model via COM ||
|| [[http://kb.mozillazine.org/PyXPCOM| PyXPCOM]] || Browser automation || [[http://starship.python.net/crew/mhammond/| Mark Hammond]] || || provides Python XPCOM bindings for automating Mozilla browsers ||
|| [[http://pythonpaste.org/testing-applications.html| paste.test.fixture]] || In-process || [[http://blog.ianbicking.org/| Ian Bicking]] || [[http://pythonpaste.org/| Paste]] || lets you test your [[http://www.python.org/peps/pep-0333.html| WSGI]] applications without a server; similar in feel to zope.testbrowser ||
|| [[http://simon.bofh.ms/cgi-bin/trac-django-projects.cgi/wiki/DjangoTesting| DjangoTesting]] || In-process || [[http://hugo.muensterland.org/| Georg Bauer]] || [[http://simon.bofh.ms/cgi-bin/trac-django-projects.cgi/wiki/DjangoStuff| DjangoStuff]] || very simple and small testing framework for [[http://www.djangoproject.com/| Django]] applications, based on the ideas of the [[http://www.rubyonrails.org/| Ruby on Rails]] testing framework; fully based on unittest, so you don't need anything beside Django and Python to make use of it ||
|| [[http://ibofobi.dk/blog/archive/2005/11/django-doctest/| ibofobi.utils.test]] || In-process || [[http://ibofobi.dk/blog/| Sune Kirkeby]] || || testing framwework for [[http://www.djangoproject.com/| Django]] applications; built on top of doctest instead of unittest; fixtures are written in YAML not Python code; implemented simple request/response tests on top of [[http://www.crummy.com/software/BeautifulSoup/| Beautiful Soup]] ||
|| [[http://code.google.com/p/wsgi-intercept/| wsgi-intercept]] || In-process || [[http://code.google.com/u/kumar.mcmillan/| Kumar McMillan]] || || This library lets you intercept calls to any specific host/port combination and redirect them into a WSGI application ||
|| [[http://www.getwindmill.com/| windmill]] || Browser Automation || [[http://www.getwindmill.com/| Windmill Community]] || || Cross-Platform and Cross-Browser web automation tool, supports Firefox, Safari, Chrome and IE on Mac, Linux and Windows. Written in Python, test authoring libraries in Python and JavaScript. Active community, #windmill on irc.freenode.org ||
Line 80: Line 80:
'''Note''': Although not a Python-specific testing tool, [http://openqa.org/selenium Selenium] is nevertheless an important addition to any tester's arsenal. It relates to Python in that it can use a Twisted-based server to run tests against Web applications. Read a [http://agiletesting.blogspot.com/2005/03/web-app-testing-with-python-part-2.html tutorial] about it, followed by a [http://agiletesting.blogspot.com/2005/03/quick-update-on-selenium-in-twisted.html quick update]. With [http://joker.linuxstuff.pl/documentation/make_selenium make_selenium.py] script you can write Selenium tests in Python. '''Note''': Although not a Python-specific testing tool, [[http://openqa.org/selenium| Selenium]] is nevertheless an important addition to any tester's arsenal. It relates to Python in that it can use a Twisted-based server to run tests against Web applications. Read a [[http://agiletesting.blogspot.com/2005/03/web-app-testing-with-python-part-2.html| tutorial]] about it, followed by a [[http://agiletesting.blogspot.com/2005/03/quick-update-on-selenium-in-twisted.html| quick update]]. With [[http://joker.linuxstuff.pl/documentation/make_selenium| make_selenium.py]] script you can write Selenium tests in Python.
Line 85: Line 85:
|| [http://fitnesse.org/FitServers.PythonFit PyFIT] || John Roth || port of [http://fit.c2.com/ FIT]; supports both FIT and [http://fitnesse.org/ FitNesse]; acceptance tests for business logic written as HTML or Wiki tables ||
|| [http://texttest.org/ TextTest] || Geoff Bache || works by comparing plain text logged by programs with a previous 'gold standard' version of that text; the tool itself is written in Python but it can be used to test programs written in any language; comes with extensive self tests which serve as examples of how to use it, i.e. how to test a non-trivial application with a pyGTK GUI. ||
|| [http://svn.colorstudy.com/FitLoader/trunk/ FitLoader] || Ian Bicking || A (rather alpha-ish) library for loading tests from Excel or CSV files; not really a framework, but useful for acceptance tests written by non-programmers ||
|| [http://robotframework.org/ Robot Framework] || [http://groups.google.com/group/robotframework-devel Pekka Klärck] || Robot Framework is a keyword-driven test automation framework for acceptance level testing and acceptance test-driven development (ATDD). It has an easy-to-use tabular syntax for creating test cases and its testing capabilities can be extended by test libraries implemented either with Python or Java (requires Jython). Users can also create new keywords from existing ones using the same simple syntax that is used for creating test cases. ||
|| [[http://fitnesse.org/FitServers.PythonFit| PyFIT]] || John Roth || port of [[http://fit.c2.com/| FIT]]; supports both FIT and [[http://fitnesse.org/| FitNesse]]; acceptance tests for business logic written as HTML or Wiki tables ||
|| [[http://texttest.org/| TextTest]] || Geoff Bache || works by comparing plain text logged by programs with a previous 'gold standard' version of that text; the tool itself is written in Python but it can be used to test programs written in any language; comes with extensive self tests which serve as examples of how to use it, i.e. how to test a non-trivial application with a pyGTK GUI. ||
|| [[http://svn.colorstudy.com/FitLoader/trunk/| FitLoader]] || Ian Bicking || A (rather alpha-ish) library for loading tests from Excel or CSV files; not really a framework, but useful for acceptance tests written by non-programmers ||
|| [[http://robotframework.org/| Robot Framework]] || [[http://groups.google.com/group/robotframework-devel| Pekka Klärck]] || Robot Framework is a keyword-driven test automation framework for acceptance level testing and acceptance test-driven development (ATDD). It has an easy-to-use tabular syntax for creating test cases and its testing capabilities can be extended by test libraries implemented either with Python or Java (requires Jython). Users can also create new keywords from existing ones using the same simple syntax that is used for creating test cases. ||
Line 93: Line 93:
|| [http://gintas.pov.lt/guitest/ guitest] || [http://gintasm.blogspot.com/ Gintautas Miliauskas] || Python helper library for testing Python GUI applications, with [http://www.pygtk.org/ pyGTK support] being the most mature ||
|| [http://sourceforge.net/projects/pyguiunit/ pyGUIUnit] || [http://pydev.blogspot.com/ Fabio Zadrozny] || aims to be a gui unittesting library for python; initially provided solely for [http://www.riverbankcomputing.co.uk/pyqt/ PyQt], but it may be extended in the future ||
|| [http://people.redhat.com/zcerza/dogtail/ dogtail] || [http://people.redhat.com/zcerza/ Zack Cerza] || Created by Redhat engineers on linux. Uses the X11 accessability framework (AT-SPI) to drive applications so works well with the gnome desktop on Unixes. Has [http://people.redhat.com/zcerza/dogtail/media.html flash movies] ||
|| [http://ldtp.freedesktop.org/wiki/ ldtp] || [http://ldtp.freedesktop.org/wiki/TeamMembers team members] || Also uses the X11 accessability framework (AT-SPI) to drive applications so works well with the gnome desktop on Unixes. Has extensive tests for the evolution groupware client. ||
|| [http://sourceforge.net/projects/pywinauto/ pywinauto] || Mark McMahon || Simple Windows (NT/2K/XP) GUI automation with Python. There are tests included for Localization testing but there is no limitation to this. Most of the code at the moment is for recovering information from Windows windows and performing actions on those controls. The idea is to have high level methods for standard controls rather then rely on Sending keystrokes to the applications. ||
|| [http://sourceforge.net/project/showfiles.php?group_id=65529 pyAA] || [http://www.cs.unc.edu/%7Eparente/ Peter Parente] || pyAA is an object oriented Python wrapper around the client-side functionality in the Microsoft Active Accessibility (MSAA) library. MSAA is a library for the Windows platform that allows client applications inspect, control, and monitor events and controls in graphical user interfaces (GUIs) and server applications to expose runtime information about their user interfaces. See the [http://www.cs.unc.edu/%7Eparente/tech/tr09.shtml User interface automation with pyAA] tutorial for more info. ||
|| [http://www.tizmoi.net/watsup/intro.html WATSUP] || Dr Tim Couper || Windows Application Test System Using Python - another Windows GUI automation tool. ||
|| [http://www.brunningonline.net/simon/blog/archives/winGuiAuto.py.html winGuiAuto] || Simon Brunning || Low-level library for Windows GUI automation used by PAMIE and WATSUP. ||
|| [http://www.async.com.br/projects/kiwi/ kiwi ui testing] || Johan Dahlin || PyGTK UI Testing framework, recording and playback system. ||
|| [[http://gintas.pov.lt/guitest/| guitest]] || [[http://gintasm.blogspot.com/| Gintautas Miliauskas]] || Python helper library for testing Python GUI applications, with [[http://www.pygtk.org/| pyGTK support]] being the most mature ||
|| [[http://sourceforge.net/projects/pyguiunit/| pyGUIUnit]] || [[http://pydev.blogspot.com/| Fabio Zadrozny]] || aims to be a gui unittesting library for python; initially provided solely for [[http://www.riverbankcomputing.co.uk/pyqt/| PyQt]], but it may be extended in the future ||
|| [[http://people.redhat.com/zcerza/dogtail/| dogtail]] || [[http://people.redhat.com/zcerza/| Zack Cerza]] || Created by Redhat engineers on linux. Uses the X11 accessability framework (AT-SPI) to drive applications so works well with the gnome desktop on Unixes. Has [[http://people.redhat.com/zcerza/dogtail/media.html| flash movies]] ||
|| [[http://ldtp.freedesktop.org/wiki/| ldtp]] || [[http://ldtp.freedesktop.org/wiki/TeamMembers| team members]] || Also uses the X11 accessability framework (AT-SPI) to drive applications so works well with the gnome desktop on Unixes. Has extensive tests for the evolution groupware client. ||
|| [[http://sourceforge.net/projects/pywinauto/| pywinauto]] || Mark McMahon || Simple Windows (NT/2K/XP) GUI automation with Python. There are tests included for Localization testing but there is no limitation to this. Most of the code at the moment is for recovering information from Windows windows and performing actions on those controls. The idea is to have high level methods for standard controls rather then rely on Sending keystrokes to the applications. ||
|| [[http://sourceforge.net/project/showfiles.php?group_id=65529| pyAA]] || [[http://www.cs.unc.edu/%7Eparente/| Peter Parente]] || pyAA is an object oriented Python wrapper around the client-side functionality in the Microsoft Active Accessibility (MSAA) library. MSAA is a library for the Windows platform that allows client applications inspect, control, and monitor events and controls in graphical user interfaces (GUIs) and server applications to expose runtime information about their user interfaces. See the [[http://www.cs.unc.edu/%7Eparente/tech/tr09.shtml| User interface automation with pyAA]] tutorial for more info. ||
|| [[http://www.tizmoi.net/watsup/intro.html| WATSUP]] || Dr Tim Couper || Windows Application Test System Using Python - another Windows GUI automation tool. ||
|| [[http://www.brunningonline.net/simon/blog/archives/winGuiAuto.py.html| winGuiAuto]] || Simon Brunning || Low-level library for Windows GUI automation used by PAMIE and WATSUP. ||
|| [[http://www.async.com.br/projects/kiwi/| kiwi ui testing]] || Johan Dahlin || PyGTK UI Testing framework, recording and playback system. ||
Line 106: Line 106:
|| [http://pychecker.sourceforge.net/ PyChecker] || [http://sourceforge.net/users/ericnewton/ Eric Newton] and [http://nnorwitz.blogspot.com/ Neal Norwitz] || finds problems that are typically caught by a compiler for less dynamic languages; imports each module before checking it ||
|| [http://www.logilab.org/project/pylint Pylint] || Sylvain Thénault / [http://www.logilab.org/categories/python Logilab] || includes [http://pychecker.sourceforge.net/ PyChecker] checks, plus more features, like checking line-code's length, checking if variable names are well-formed according to your coding standard, or checking if declared interfaces are truly implemented ||
|| [http://divmod.org/trac/wiki/DivmodPyflakes Pyflakes] || Phil Frost / [http://divmod.org/trac Divmod] || similar to [http://pychecker.sourceforge.net/ PyChecker]; focused on identifying common errors quickly without executing Python code; its primary advantage over [http://pychecker.sourceforge.net/ PyChecker] is that it is fast: it runs on most large projects in only a few seconds ||
|| [http://clonedigger.sourceforge.net/ Clone Digger] || Peter Bulychev || Clone Digger aims to detect similar code in Python and Java programs. The synonyms for the term `similar code` are `clone` and `duplicate code`. ||
|| [[http://pychecker.sourceforge.net/| PyChecker]] || [[http://sourceforge.net/users/ericnewton/| Eric Newton]] and [[http://nnorwitz.blogspot.com/| Neal Norwitz]] || finds problems that are typically caught by a compiler for less dynamic languages; imports each module before checking it ||
|| [[http://www.logilab.org/project/pylint| Pylint]] || Sylvain Thénault / [[http://www.logilab.org/categories/python| Logilab]] || includes [[http://pychecker.sourceforge.net/| PyChecker]] checks, plus more features, like checking line-code's length, checking if variable names are well-formed according to your coding standard, or checking if declared interfaces are truly implemented ||
|| [[http://divmod.org/trac/wiki/DivmodPyflakes| Pyflakes]] || Phil Frost / [[http://divmod.org/trac| Divmod]] || similar to [[http://pychecker.sourceforge.net/| PyChecker]]; focused on identifying common errors quickly without executing Python code; its primary advantage over [[http://pychecker.sourceforge.net/| PyChecker]] is that it is fast: it runs on most large projects in only a few seconds ||
|| [[http://clonedigger.sourceforge.net/| Clone Digger]] || Peter Bulychev || Clone Digger aims to detect similar code in Python and Java programs. The synonyms for the term `similar code` are `clone` and `duplicate code`. ||
Line 114: Line 114:
|| [http://www.nedbatchelder.com/code/modules/coverage.html coverage] || [http://www.nedbatchelder.com/blog/index.html Ned Batchelder] || measures code coverage during Python execution; uses the code analysis tools and tracing hooks provided in the Python standard library to determine which lines are executable, and which have been executed ||
|| [http://darcs.idyll.org/%7Et/projects/figleaf/README.html figleaf] || [http://ivory.idyll.org/blog/ Titus Brown] || figleaf is a Python code coverage analysis tool, built somewhat on the model of Ned Batchelder's fantastic coverage module. The goals of figleaf are to be a minimal replacement of 'coverage.py' that supports more configurable coverage gathering and reporting; figleaf is useful for situations where you are recording code coverage in multiple execution runs and/or want to tweak the reporting output ||
|| [http://cheeseshop.python.org/pypi/trace2html trace2html] || Olivier Grisel || HTML test coverage reporting tool with white and blacklisting support ||
|| [http://www.fiber-space.de/EasyExtend/doc/coverage/coverage.html coverage langlet] || [http://www.fiber-space.de/ Kay Schluehr] || The coverage langlet weaves monitoring commands, so called ''sensors'', into source code during global source transformation. When a statement is covered the weaved sensor responds. The coverage langlet is part of [http://www.fiber-space.de/EasyExtend/doc/EE.html EasyExtend] ||
|| [http://dev.nullcube.com/ pry] || [http://www.nullcube.com/ Aldo Cortesi] || Elegant unit testing framework with built-in coverage analysis, profiling, micro-benchmarking and a powerful command-line interface. ||
|| [[http://www.nedbatchelder.com/code/modules/coverage.html| coverage]] || [[http://www.nedbatchelder.com/blog/index.html| Ned Batchelder]] || measures code coverage during Python execution; uses the code analysis tools and tracing hooks provided in the Python standard library to determine which lines are executable, and which have been executed ||
|| [[http://darcs.idyll.org/%7Et/projects/figleaf/README.html| figleaf]] || [[http://ivory.idyll.org/blog/| Titus Brown]] || figleaf is a Python code coverage analysis tool, built somewhat on the model of Ned Batchelder's fantastic coverage module. The goals of figleaf are to be a minimal replacement of 'coverage.py' that supports more configurable coverage gathering and reporting; figleaf is useful for situations where you are recording code coverage in multiple execution runs and/or want to tweak the reporting output ||
|| [[http://cheeseshop.python.org/pypi/trace2html| trace2html]] || Olivier Grisel || HTML test coverage reporting tool with white and blacklisting support ||
|| [[http://www.fiber-space.de/EasyExtend/doc/coverage/coverage.html| coverage langlet]] || [[http://www.fiber-space.de/| Kay Schluehr]] || The coverage langlet weaves monitoring commands, so called ''sensors'', into source code during global source transformation. When a statement is covered the weaved sensor responds. The coverage langlet is part of [[http://www.fiber-space.de/EasyExtend/doc/EE.html| EasyExtend]] ||
|| [[http://dev.nullcube.com/| pry]] || [[http://www.nullcube.com/| Aldo Cortesi]] || Elegant unit testing framework with built-in coverage analysis, profiling, micro-benchmarking and a powerful command-line interface. ||
Line 125: Line 125:
|| [http://buildbot.sourceforge.net/ buildbot] || [http://www.lothar.com/warner.html Brian Warner] || buildbot is a system to automate the compile/test cycle required by most software projects to validate code changes. By automatically rebuilding and testing the tree each time something has changed, build problems are pinpointed quickly, before other developers are inconvenienced by the failure. ||
|| [http://bitten.cmlenz.net/ Bitten] || [http://cmlenz.net/ Christopher Lenz] || Bitten is a Python-based framework for collecting various software metrics via continuous integration. It builds on [http://projects.edgewall.com/trac Trac] to provide an integrated web-based user interface. ||
|| [http://cheeseshop.python.org/pypi/svnchecker/ SVNChecker] || Heinrich Wendel of [http://www.dlr.de/sc/ German Aerospace Center] || SVNChecker is a framework for Subversion pre-commit hooks in order to implement checks of the to be commited files before they are commited. For example, you can check for the code style or unit tests. The output of the checks can be send by mail or be written into a file or simply print to the console. ||
|| [http://www.logilab.org/project/apycot apycot] || [http://www.logilab.fr/ Logilab] || an Automated Pythonic Code Tester: designed to run tests on a code repository on a daily basis. It comes with a set of predefined test, essentially for python packages, and a set of predefined reports to display execution results. However, it has been designed to be higly extensible, so you could write your own test or report using the Python language ||
|| [http://github.com/ctb/pony-build pony-build] || [http://ivory.idyll.org/blog/ Titus Brown] || pony-build is a simple continuous integration package that lets you run a server to display client build results. It consists of two components, a server (which is run in some central & accessible location), and one or more clients (which must be able to contact the server via HTTP). Philosophy statement: good development tools for Python should be easy to install, easy to hack, and not overly constraining. Two out of three ain't bad ;). ||
|| [[http://buildbot.sourceforge.net/| buildbot]] || [[http://www.lothar.com/warner.html| Brian Warner]] || buildbot is a system to automate the compile/test cycle required by most software projects to validate code changes. By automatically rebuilding and testing the tree each time something has changed, build problems are pinpointed quickly, before other developers are inconvenienced by the failure. ||
|| [[http://bitten.cmlenz.net/| Bitten]] || [[http://cmlenz.net/| Christopher Lenz]] || Bitten is a Python-based framework for collecting various software metrics via continuous integration. It builds on [[http://projects.edgewall.com/trac| Trac]] to provide an integrated web-based user interface. ||
|| [[http://cheeseshop.python.org/pypi/svnchecker/| SVNChecker]] || Heinrich Wendel of [[http://www.dlr.de/sc/| German Aerospace Center]] || SVNChecker is a framework for Subversion pre-commit hooks in order to implement checks of the to be commited files before they are commited. For example, you can check for the code style or unit tests. The output of the checks can be send by mail or be written into a file or simply print to the console. ||
|| [[http://www.logilab.org/project/apycot| apycot]] || [[http://www.logilab.fr/| Logilab]] || an Automated Pythonic Code Tester: designed to run tests on a code repository on a daily basis. It comes with a set of predefined test, essentially for python packages, and a set of predefined reports to display execution results. However, it has been designed to be higly extensible, so you could write your own test or report using the Python language ||
|| [[http://github.com/ctb/pony-build| pony-build]] || [[http://ivory.idyll.org/blog/| Titus Brown]] || pony-build is a simple continuous integration package that lets you run a server to display client build results. It consists of two components, a server (which is run in some central & accessible location), and one or more clients (which must be able to contact the server via HTTP). Philosophy statement: good development tools for Python should be easy to install, easy to hack, and not overly constraining. Two out of three ain't bad ;). ||
Line 134: Line 134:
|| [http://code.google.com/p/fixture/ fixture] || [http://farmdev.com/ Kumar McMillan] || || module for loading and referencing test data ||
|| [http://www.djangoproject.com/documentation/testing/#fixtures Django Fixtures] || Russell Keith-Magee || [http://www.djangoproject.com/ django] || A test case for a database-backed website isn't much use if there isn't any data in the database. To make it easy to put test data into the database, Django provides a fixtures framework. ||
|| [[http://code.google.com/p/fixture/| fixture]] || [[http://farmdev.com/| Kumar McMillan]] || || module for loading and referencing test data ||
|| [[http://www.djangoproject.com/documentation/testing/#fixtures| Django Fixtures]] || Russell Keith-Magee || [[http://www.djangoproject.com/| django]] || A test case for a database-backed website isn't much use if there isn't any data in the database. To make it easy to put test data into the database, Django provides a fixtures framework. ||
Line 140: Line 140:
|| [http://cthedot.de/retest/ retest] || [http://cthedot.de/blog/ Christof Hoeke] || || enables tests of Python regular expressions in a web browser; it uses SimpleHTTPServer and AJAX ||
|| [http://pyreb.nongnu.org/ pyreb] || [http://cowo.yoda2000.net/ Giuseppe "Cowo" Corbelli] || || a wxPython GUI to the re module ||
|| [http://kodos.sourceforge.net/ kodos ] || [http://www.phil-schwartz.com/ Phil Schwartz] || || a QT (KDE) Regular Expression Debugger ||
|| [http://svn.pythonpaste.org/Paste/trunk/paste/fixture.py paste TestFileEnvironment] || [http://blog.ianbicking.org/ Ian Bicking] || [http://pythonpaste.org/ Paste] || A simple environment for testing command-line applications, running commands and seeing what files they write to ||
|| [http://guppy-pe.sourceforge.net/#Heapy Heapy] || [http://sncs.se/ Sverker Nilsson] || [http://guppy-pe.sourceforge.net/ Guppy-PE] || supports profiling, debugging and optimization regarding memory related issues in Python programs ||
|| [http://pysizer.8325.org/ PySizer] || Nick Smallbone || || a memory usage profiler for Python code ||
|| [http://sourceforge.net/projects/pymetrics/ PyMetrics] || Reginald B. Charney || || Generates metrics (e.g. McCabe, LoC, %comments, etc.) for Python code. ||
|| [http://journyx.com/curt/complex/ complexity] || Curt Finch || || Analyzes cyclomatic complexity of Python code (written in Perl) ||
|| [http://www.qmtest.com/ qmtest] || [http://codesourcery.com/ CodeSourcery] || || test and automation framework, in python ||
|| [http://www.fiber-space.de/EasyExtend/doc/consoletest/consoletest.html consoletest] || [http://www.fiber-space.de/ Kay Schluehr] || [http://www.fiber-space.de/EasyExtend/doc/EE.html EasyExtend] || Records interactive sessions and extracts test cases when they get replayed ||
|| [http://engineering.meta-comm.com/allpairs.aspx AllPairs] || [http://engineering.meta-comm.com/allpairs.aspx MetaCommunications] || || Test combinations generator (in python) that allows to create set of tests using 'pairwise combinations' method, which reduces a number of combinations of variables into a lesser set which covers most situations. See http://www.pairwise.org/ for more info on this functional testing technique. ||
|| [http://staff.washington.edu/jon/pymodel/www/ PyModel] || [http://staff.washington.edu/jon/ Jon Jacky] || || Model-based testing framework where the models are written in Python. [http://staff.washington.edu/jon/pymodel/www/ PyModel] supports offline and on-the-fly testing. It uses composition for scenario control. Coverage can be guided by a programmable strategy. ||
|| [[http://cthedot.de/retest/| retest]] || [[http://cthedot.de/blog/| Christof Hoeke]] || || enables tests of Python regular expressions in a web browser; it uses SimpleHTTPServer and AJAX ||
|| [[http://pyreb.nongnu.org/| pyreb]] || [[http://cowo.yoda2000.net/| Giuseppe "Cowo" Corbelli]] || || a wxPython GUI to the re module ||
|| [[http://kodos.sourceforge.net/| kodos ]] || [[http://www.phil-schwartz.com/| Phil Schwartz]] || || a QT (KDE) Regular Expression Debugger ||
|| [[http://svn.pythonpaste.org/Paste/trunk/paste/fixture.py| paste TestFileEnvironment]] || [[http://blog.ianbicking.org/| Ian Bicking]] || [[http://pythonpaste.org/| Paste]] || A simple environment for testing command-line applications, running commands and seeing what files they write to ||
|| [[http://guppy-pe.sourceforge.net/#Heapy| Heapy]] || [[http://sncs.se/| Sverker Nilsson]] || [[http://guppy-pe.sourceforge.net/| Guppy-PE]] || supports profiling, debugging and optimization regarding memory related issues in Python programs ||
|| [[http://pysizer.8325.org/| PySizer]] || Nick Smallbone || || a memory usage profiler for Python code ||
|| [[http://sourceforge.net/projects/pymetrics/| PyMetrics]] || Reginald B. Charney || || Generates metrics (e.g. McCabe, LoC, %comments, etc.) for Python code. ||
|| [[http://journyx.com/curt/complex/| complexity]] || Curt Finch || || Analyzes cyclomatic complexity of Python code (written in Perl) ||
|| [[http://www.qmtest.com/| qmtest]] || [[http://codesourcery.com/| CodeSourcery]] || || test and automation framework, in python ||
|| [[http://www.fiber-space.de/EasyExtend/doc/consoletest/consoletest.html| consoletest]] || [[http://www.fiber-space.de/| Kay Schluehr]] || [[http://www.fiber-space.de/EasyExtend/doc/EE.html| EasyExtend]] || Records interactive sessions and extracts test cases when they get replayed ||
|| [[http://engineering.meta-comm.com/allpairs.aspx| AllPairs]] || [[http://engineering.meta-comm.com/allpairs.aspx| MetaCommunications]] || || Test combinations generator (in python) that allows to create set of tests using 'pairwise combinations' method, which reduces a number of combinations of variables into a lesser set which covers most situations. See http://www.pairwise.org/ for more info on this functional testing technique. ||
|| [[http://staff.washington.edu/jon/pymodel/www/| PyModel]] || [[http://staff.washington.edu/jon/| Jon Jacky]] || || Model-based testing framework where the models are written in Python. [[http://staff.washington.edu/jon/pymodel/www/| PyModel]] supports offline and on-the-fly testing. It uses composition for scenario control. Coverage can be guided by a programmable strategy. ||

Join the Testing In Python (TIP) mailing list for Python testing tools discussions!

Unit Testing Tools

Tool

Category

Author

Part of

Claim to fame

unittest

unit testing

Steve Purcell

Python standard lib

first unit test framework to be included in Python standard library; easy to use by people familiar with the xUnit frameworks; strong support for test organization and reuse via test suites

doctest

unit testing

Tim Peters

Python standard lib

copy and paste output from shell session; Agile Documentation: unit tests themselves can serve as documentation when combined with epydoc; also see Django's approach

py.test

unit testing

Holger Krekel

py lib

no API!;automatic collection of tests; simple asserts; strong support for test fixture/state management via setup/teardown hooks; strong debugging support via customized traceback

TestOOB

unittest extensions

Ori Peleg

unittest enhancements; test filtering via regex/glob patterns; reporting in XML/HTML; colorized output; runs pdb on failing tests; run in parallel in threads/processes; verbose asserts; report failures immediately; and a little more;

nose

unittest extensions

Jason Pellerin

unit test framework, used most notably by TurboGears and Pylons; provides an alternate test discovery and running process for unittest, one that is intended to mimic the behavior of py.test as much as is reasonably possible without resorting to too much magic. More friendly with unittest.TestCase-based tests than py.test. There are also many plugins available.

Trial

unittest extensions

Jonathan Lange

Twisted

Extension of unittest to support writing asynchronous unit tests using Deferreds and new result types ('skip' and 'todo'). Includes a command-line program that does test discovery and integrates with doctest and coverage.

subunit

unittest extensions

Robert Collins

Transparently adds support for running unittest test cases/suites in a separate process : prevents system wide changes by a test destabilising the test runner. It also allows reporting from tests in another process into the unittest framework, giving a single integrated test environment.

testresources

unittest extensions

Robert Collins

Provides a mechanism for managing 'resources' - expensive bits of infrastructure - that are needed by multiple tests. Resources are constructed and free on demand, but with an optional TestSuite?, the test run order is optimised to reduce the number of resource constructions and releases needed. Compatible with unittest.

testtools

unittest extensions

Jonathan Lange

Useful extensions to unittest derived from custom extensions by projects such as Twisted and Bazaar.

Sancho

unit testing

MEMS and Nanotechnology Exchange

MEMS Exchange tools

Sancho 2.1 runs tests, and provides output for tests that fail; Sancho 2.1 does not count tests passed or failed; targets projects that do not maintain failing tests

pyUnitPerf

performance-based unit testing

Grig Gheorghiu

port of JUnitPerf; transparently adds performance testing capabilities to existing unitttest-based test suites

Peckcheck

specification-based unit testing

Darius Bacon

like the unittest module, except that test methods may take arguments; you declare each argument with a default value which names a test-data generator and the peckcheck module will peck at your test methods with a bunch of generated values

qunittest

unit testing

Edgewall

Easy-to-use (PyQt) GUI framework and application for use with the Python unit testing framework. It can be used to conveniently execute unit tests and suites, and then display the results in a useful fashion.

testosterone

unit testing

Chad Whitacre

A manly test runner; command-line and curses(3) interfaces; the iPod of test runners. (More seriously, makes it easier to browse and inspect your test failures)

zope.testing

unit testing

Zope3 community

Powerful test runner that includes support for post-mortem debugging of test failures. Also includes profiling and coverage reporting. This is a standalone package that has no dependencies on Zope and works just fine with projects that don't use Zope.

pry

unit testing

Aldo Cortesi

Elegant unit testing framework with built-in coverage analysis, profiling, micro-benchmarking and a powerful command-line interface.

pythoscope

unit testing

Michal Kwiatkowski

Tool that will automatically, or semi-automatically, generate unit tests for legacy systems written in Python.

testlib

unittest extensions

Logilab

logilab-common

Gives more power to standard unittest. More assert* methods; support for module level setup/teardown; skip test feature...

pytest

tests runner

Logilab

logilab-common

Tests finder / runner. Selectivly run tests; Stop on first failure; Run pdb on failed tests; Colorized reports; Run tests with coverage / profile enabled...

dutest

unittest extensions

Olemis Lang

PyOOP package (distributed separately too)

An object oriented interface to retrieve unittest test cases out of doctests. Hides initialization from doctests by allowing setUp and tearDown for each interactive example. Allows control over all the options provided by doctest. Specialized classes allow selective test discovery across a package hierarchy.

Mock Testing Tools

Tool

Category

Author

Claim to fame

Ludibrio

mocks, stubs, spy, and dummies

Gustavo Rezende

Elegant test doubles framework in Python (mocks, stubs, spy, and dummies)

Python Mock

mock testing

Dave Kirby

Python Mock enables the easy creation of mock objects that can be used to emulate the behaviour of any class that the code under test depends on. You can set up expectations about the calls that are made to the mock object, and examine the history of calls made. This makes it easier to unit test classes in isolation. Download it here.

PyMock

mock testing

Jeff Younker

PyMock is based on the Java easymock package. It uses a recording and replay model rather than using a specification language. Easymock lives up to its name compared to other mocking packages. PyMock takes advantage of python's dynamic nature for futher improvements.

mock

mock testing

Michael Foord aka Fuzzyman

Provides "action -> assertion" mocking patter, instead of standard "record -> replay" pattern

pMock

mock testing

Graham Carlyle

Inspired by the Java jMock library, pMock makes the writing of unit tests using mock object techniques easier. Development of pmock has long since stopped and so it can be considered dead.

minimock

mock testing

Ian Bicking

Embeds mock testing constructs inside doctest tests.

svnmock

mock testing

Collin Winter

enables easier testing of Python programs that make use of Subversion's Python bindings

Mocker

mock testing

Gustavo Niemeyer

Graceful platform for test doubles in Python (mocks, stubs, fakes, and dummies). Well-documented and fairly feature-complete.

Stubble

stub testing

Iwan Vosloo

Stubble allows you to write arbitrary classes for use as stubs instead of read classes while testing. Stubble lets you link a stub class loosely to the real class which it is a stub for. This information is then used to ensure that tests will break if there is a discrepancy between the interface supported by your stub class and that of the real class it stands in for.

Mox

mock testing

smiddlek, dglasser

Mox is based on EasyMock, a Java mock object framework. Mox will make mock objects for you, so you don't have to create your own! It mocks the public/protected interfaces of Python objects. You set up your mock objects expected behavior using a domain specific language (DSL), which makes it easy to use, understand, and refactor!

MockTest

mock testing

Tim Cuthbertson (gfxmonk)

Mocktest allows you to mock / stub objects and make expectations about the methods you expect to be called as well as the arguments they should be called with. Expectations are very readable and expressive, and checked automatically. Any stubbed methods are reverted after each test case. Still under development, so subject to change

Fudge

mock and stub testing

Kumar McMillan

A module for using fake objects (mocks, stubs, etc) to test real ones. Uses a declrative syntax like jMock whereby you set up expectations for how an object should be used. An error will raise if an expectation is not met.

Mockito for Python

mock and stub testing

Szczepan Faber

A port of the Mockito mocking framework to Python. (Technically speaking, Mockito is a Test Spy framework.)

Fuzz Testing Tools

According to Wikipedia, "fuzz testing" (or "fuzzing") is a software testing technique whose basic idea is to attach the inputs of a program to a source of random data ("fuzz"). If the program fails (for example, by crashing, or by failing built-in code assertions), then there are defects to correct. The great advantage of fuzz testing is that the test design is extremely simple, and free of preconceptions about system behavior.

Tool

Author

Claim to fame

Pester

Ivan Moore

Tests your tests by mutating source code and finding tests that don't fail!

Peach Fuzzer Framework

Michael Eddington

Peach can fuzz just about anything from .NET, COM/ActiveX, SQL, shared libraries/DLL's, network applications, web, you name it.

antiparser

dmckinney

The purpose of antiparser is to provide an API that can be used to model network protocols and file formats by their composite data types. Once a model has been created, the antiparser has various methods for creating random sets of data that deviates in ways that will ideally trigger software bugs or security vulnerabilities.

Taof (The Art Of Fuzzing)

rodrigomarcos

Taof is a GUI cross-platform Python generic network protocol fuzzer. It has been designed for minimizing set-up time during fuzzing sessions and it is specially useful for fast testing of proprietary or undocumented protocols.

Fusil (Fusil the fuzzer)

Victor Stinner

It helps to start process with a prepared environment (limit memory, environment variables, redirect stdout, etc.), start network client or server, and create mangled files. Fusil has many probes to detect program crash: watch process exit code, watch process stdout and syslog for text patterns (eg. "segmentation fault"), watch session duration, watch cpu usage (process and system load), etc.

Web Testing Tools

First, let's define some categories of Web testing tools:

  • Browser simulation tools: simulate browsers by implementing the HTTP request/response protocol and by parsing the resulting HTML

  • Browser automation tools: automate browsers by driving them for example via COM calls in the case of Internet Explorer, or XPCOM in the case of Mozilla

  • In-process or unit-test-type tools (courtesy of Ian Bicking]): call an application in the same process, instead of generating an HTTP request; so an exception in the application would go all the way up to the command runner (py.test, unittest, etc).

Tool

Category

Author

Part of

Claim to fame

twill

Browser simulation & In-process

Titus Brown

offers simple commands for navigating Web pages, posting forms and asserting conditions; can be used as shell script or Python module; can be used for unit-testing your own Web app; stress-test functionality; port of PBP; uses John J. Lee's mechanize

webunit

Browser simulation

Richard Jones

HTTP, HTTPS, GET, POST, basic auth all handled; control over expected status codes; uses unittest as the underlying framework

FunkLoad

Browser simulation

Benoit Delbosc / Nuxeo

functional/performance/load/stress testing for Web applications; easy test creation using TCPWatch as proxy recorder; reporting capabilities; based on webunit

zope.testbrowser

Browser simulation

Benji York

Zope 3

designed to be used in doctests that mirror user actions (see the README.txt for examples); can also be used as standalone Web testing tool outside of Zope 3

webtest

Browser simulation

Robert Brewer

CherryPy

extensions to unittest for web frameworks; uses http://docs.python.org/lib/module-httplib.html httplib]

PAMIE

Browser automation

Robert Marchetti

automates Internet Explorer by manipulating IE's Document Object Model via COM

PyXPCOM

Browser automation

Mark Hammond

provides Python XPCOM bindings for automating Mozilla browsers

paste.test.fixture

In-process

Ian Bicking

Paste

lets you test your WSGI applications without a server; similar in feel to zope.testbrowser

DjangoTesting

In-process

Georg Bauer

DjangoStuff

very simple and small testing framework for Django applications, based on the ideas of the Ruby on Rails testing framework; fully based on unittest, so you don't need anything beside Django and Python to make use of it

ibofobi.utils.test

In-process

Sune Kirkeby

testing framwework for Django applications; built on top of doctest instead of unittest; fixtures are written in YAML not Python code; implemented simple request/response tests on top of Beautiful Soup

wsgi-intercept

In-process

Kumar McMillan

This library lets you intercept calls to any specific host/port combination and redirect them into a WSGI application

windmill

Browser Automation

Windmill Community

Cross-Platform and Cross-Browser web automation tool, supports Firefox, Safari, Chrome and IE on Mac, Linux and Windows. Written in Python, test authoring libraries in Python and JavaScript. Active community, #windmill on irc.freenode.org

Note: Although not a Python-specific testing tool, Selenium is nevertheless an important addition to any tester's arsenal. It relates to Python in that it can use a Twisted-based server to run tests against Web applications. Read a tutorial about it, followed by a quick update. With make_selenium.py script you can write Selenium tests in Python.

Acceptance/Business Logic Testing Tools

Tool

Author

Claim to fame

PyFIT

John Roth

port of FIT; supports both FIT and FitNesse; acceptance tests for business logic written as HTML or Wiki tables

TextTest

Geoff Bache

works by comparing plain text logged by programs with a previous 'gold standard' version of that text; the tool itself is written in Python but it can be used to test programs written in any language; comes with extensive self tests which serve as examples of how to use it, i.e. how to test a non-trivial application with a pyGTK GUI.

FitLoader

Ian Bicking

A (rather alpha-ish) library for loading tests from Excel or CSV files; not really a framework, but useful for acceptance tests written by non-programmers

Robot Framework

Pekka Klärck

Robot Framework is a keyword-driven test automation framework for acceptance level testing and acceptance test-driven development (ATDD). It has an easy-to-use tabular syntax for creating test cases and its testing capabilities can be extended by test libraries implemented either with Python or Java (requires Jython). Users can also create new keywords from existing ones using the same simple syntax that is used for creating test cases.

GUI Testing Tools

Tool

Author

Claim to fame

guitest

Gintautas Miliauskas

Python helper library for testing Python GUI applications, with pyGTK support being the most mature

pyGUIUnit

Fabio Zadrozny

aims to be a gui unittesting library for python; initially provided solely for PyQt, but it may be extended in the future

dogtail

Zack Cerza

Created by Redhat engineers on linux. Uses the X11 accessability framework (AT-SPI) to drive applications so works well with the gnome desktop on Unixes. Has flash movies

ldtp

team members

Also uses the X11 accessability framework (AT-SPI) to drive applications so works well with the gnome desktop on Unixes. Has extensive tests for the evolution groupware client.

pywinauto

Mark McMahon

Simple Windows (NT/2K/XP) GUI automation with Python. There are tests included for Localization testing but there is no limitation to this. Most of the code at the moment is for recovering information from Windows windows and performing actions on those controls. The idea is to have high level methods for standard controls rather then rely on Sending keystrokes to the applications.

pyAA

Peter Parente

pyAA is an object oriented Python wrapper around the client-side functionality in the Microsoft Active Accessibility (MSAA) library. MSAA is a library for the Windows platform that allows client applications inspect, control, and monitor events and controls in graphical user interfaces (GUIs) and server applications to expose runtime information about their user interfaces. See the User interface automation with pyAA tutorial for more info.

WATSUP

Dr Tim Couper

Windows Application Test System Using Python - another Windows GUI automation tool.

winGuiAuto

Simon Brunning

Low-level library for Windows GUI automation used by PAMIE and WATSUP.

kiwi ui testing

Johan Dahlin

PyGTK UI Testing framework, recording and playback system.

Source Code Checking Tools

Tool

Author

Claim to fame

PyChecker

Eric Newton and Neal Norwitz

finds problems that are typically caught by a compiler for less dynamic languages; imports each module before checking it

Pylint

Sylvain Thénault / Logilab

includes PyChecker checks, plus more features, like checking line-code's length, checking if variable names are well-formed according to your coding standard, or checking if declared interfaces are truly implemented

Pyflakes

Phil Frost / Divmod

similar to PyChecker; focused on identifying common errors quickly without executing Python code; its primary advantage over PyChecker is that it is fast: it runs on most large projects in only a few seconds

Clone Digger

Peter Bulychev

Clone Digger aims to detect similar code in Python and Java programs. The synonyms for the term similar code are clone and duplicate code.

Code Coverage Tools

Tool

Author

Claim to fame

coverage

Ned Batchelder

measures code coverage during Python execution; uses the code analysis tools and tracing hooks provided in the Python standard library to determine which lines are executable, and which have been executed

figleaf

Titus Brown

figleaf is a Python code coverage analysis tool, built somewhat on the model of Ned Batchelder's fantastic coverage module. The goals of figleaf are to be a minimal replacement of 'coverage.py' that supports more configurable coverage gathering and reporting; figleaf is useful for situations where you are recording code coverage in multiple execution runs and/or want to tweak the reporting output

trace2html

Olivier Grisel

HTML test coverage reporting tool with white and blacklisting support

coverage langlet

Kay Schluehr

The coverage langlet weaves monitoring commands, so called sensors, into source code during global source transformation. When a statement is covered the weaved sensor responds. The coverage langlet is part of EasyExtend

pry

Aldo Cortesi

Elegant unit testing framework with built-in coverage analysis, profiling, micro-benchmarking and a powerful command-line interface.

Continuous Integration Tools

Although not properly a part of testing tools, continuous integration tools are nevertheless an important addition to a tester's arsenal.

Tool

Author

Claim to fame

buildbot

Brian Warner

buildbot is a system to automate the compile/test cycle required by most software projects to validate code changes. By automatically rebuilding and testing the tree each time something has changed, build problems are pinpointed quickly, before other developers are inconvenienced by the failure.

Bitten

Christopher Lenz

Bitten is a Python-based framework for collecting various software metrics via continuous integration. It builds on Trac to provide an integrated web-based user interface.

SVNChecker

Heinrich Wendel of German Aerospace Center

SVNChecker is a framework for Subversion pre-commit hooks in order to implement checks of the to be commited files before they are commited. For example, you can check for the code style or unit tests. The output of the checks can be send by mail or be written into a file or simply print to the console.

apycot

Logilab

an Automated Pythonic Code Tester: designed to run tests on a code repository on a daily basis. It comes with a set of predefined test, essentially for python packages, and a set of predefined reports to display execution results. However, it has been designed to be higly extensible, so you could write your own test or report using the Python language

pony-build

Titus Brown

pony-build is a simple continuous integration package that lets you run a server to display client build results. It consists of two components, a server (which is run in some central & accessible location), and one or more clients (which must be able to contact the server via HTTP). Philosophy statement: good development tools for Python should be easy to install, easy to hack, and not overly constraining. Two out of three ain't bad ;).

Test Fixtures

Tool

Author

Part of

Claim to fame

fixture

Kumar McMillan

module for loading and referencing test data

Django Fixtures

Russell Keith-Magee

django

A test case for a database-backed website isn't much use if there isn't any data in the database. To make it easy to put test data into the database, Django provides a fixtures framework.

Miscellaneous Python Testing Tools

Tool

Author

Part of

Claim to fame

retest

Christof Hoeke

enables tests of Python regular expressions in a web browser; it uses SimpleHTTPServer and AJAX

pyreb

Giuseppe "Cowo" Corbelli

a wxPython GUI to the re module

kodos

Phil Schwartz

a QT (KDE) Regular Expression Debugger

paste TestFileEnvironment

Ian Bicking

Paste

A simple environment for testing command-line applications, running commands and seeing what files they write to

Heapy

Sverker Nilsson

Guppy-PE

supports profiling, debugging and optimization regarding memory related issues in Python programs

PySizer

Nick Smallbone

a memory usage profiler for Python code

PyMetrics

Reginald B. Charney

Generates metrics (e.g. McCabe, LoC, %comments, etc.) for Python code.

complexity

Curt Finch

Analyzes cyclomatic complexity of Python code (written in Perl)

qmtest

CodeSourcery

test and automation framework, in python

consoletest

Kay Schluehr

EasyExtend

Records interactive sessions and extracts test cases when they get replayed

AllPairs

MetaCommunications

Test combinations generator (in python) that allows to create set of tests using 'pairwise combinations' method, which reduces a number of combinations of variables into a lesser set which covers most situations. See http://www.pairwise.org/ for more info on this functional testing technique.

PyModel

Jon Jacky

Model-based testing framework where the models are written in Python. PyModel supports offline and on-the-fly testing. It uses composition for scenario control. Coverage can be guided by a programmable strategy.

PythonTestingToolsTaxonomy (last edited 2023-10-19 08:09:31 by IwanVosloo)

Unable to edit the page? See the FrontPage for instructions.