Python is often used to crawl the internet. One of the useful application for that is finding dead links.

== Theory ==

The basic method to check if link is dead is sending a HEAD request.

Advanced stuff requires crawling links that are alive, and skipping those that are already visited.

== Links to check ==

 * [[http://math.nist.gov/~RPozo/ngraph/webcrawler.html|PCrawler]] - NIST modular crawler, Public Domain, needs some love.
 * [[https://www.mediawiki.org/wiki/Manual:Pywikibot/weblinkchecker.py|weblinkchecker.py]] - Wikipedia's Pywikibot link checker, MIT license.
 *  https://pypi.python.org/pypi/LinkChecker - many features, GPL.