HarvestMan is a world wide web crawler(robot) program written entirely in python using urllib2.
It uses python threads to achieve a very fast download of web-sites.
The current version is 0.8 beta. HarvestMan is totally free software.
[http://members.lycos.co.uk/anandpillai "HarvestMan Homepage"]
(Harvestman is a kind of small spider found in parts of N.A)