HarvestMan is a world wide web crawler(robot) program written entirely in python using urllib2.
It uses python threads to achieve a very fast download of web-sites.
The current version is 1.0.1. HarvestMan is totally free software.
[http://members.lycos.co.uk/anandpillai "HarvestMan Homepage"]
("Harvestman": Borrowed from the name of a kind of small spider found in parts of N.A)