|
Size: 2117
Comment:
|
← Revision 28 as of 2014-05-31 20:10:33 ⇥
Size: 1928
Comment: undo spam
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 7: | Line 7: |
| [http://www.freshmeat.net/projects/harvestman/ "Freshmeat Project Page"] | [[http://freecode.com/projects/harvestman|"Freecode Project Page"]] |
| Line 9: | Line 9: |
| [http://harvestman.objectis.net/ "HarvestMan Home Page"] | [[http://harvestman.freezope.org/|"HarvestMan Home Page"]] link gone |
| Line 11: | Line 11: |
| version:: 1.1.2 (''[[Date(2003-08-13T00:00:00)]]'') licence:: OSL 1.1 (Open Software License Version 1.1) Python versions:: 2.2.2, 2.2.3 (Might work for earlier versions also, not tested) |
version:: 1.4 (''<<Date(2005-05-27T00:00:00)>>'') licence:: GNU GPL Python versions:: 2.2, 2.3, 2.4 |
| Line 21: | Line 21: |
| on the internet.It can be used to download files from intranet | on the internet. It can be used to download files from intranet |
| Line 61: | Line 61: |
| "The Harvesters", |
|
| Line 64: | Line 62: |
| Nirmal K Chidambaram ---- CategoryHomepage |
Description
A www crawler(robot) program in python.
Information
"HarvestMan Home Page" link gone
- version
1.4 (2005-05-27)
- licence
- GNU GPL
- Python versions
- 2.2, 2.3, 2.4
- Platforms
- Any platform supported by python
- Binaries
- None
How it spins its web
HarvestMan uses a threading model using python threads to achieve a very fast, but highly customizable download of web-sites on the internet. It can be used to download files from intranet servers. It is the first multithreaded, opensource webcrawler written in python.
Features
- Fully Multithreaded
- Number of threads configurable by user
- Support for robots exclusion protocol
- Filtering of urls using regular expressions
- Filtering of server names using regular expressions
- Control download by specifying depth of fetching
- Configure by number of files downloadable
- Specify timeout for individual threads
- Control download speed by changing thread/depth options.
HTTP/FTP/HTTPS support & support for servers in LAN.
- XML project files which can be re-read
- Smart reconnection
- Support for proxies/firewalls
- File limits, server limits
- Projects browser page
- Command line/config file support
- Use as a program or as a web-spider module
- OO architecture
Who should use it
HarvestMan is written for the desktop user. It can be used as an internet spidering module also. An API for external users is being written.
Taxonomy
Species: HarvestMan Genus: (Internet) Spiders
Developers
- Anand B Pillai,