Differences between revisions 5 and 6
Revision 5 as of 2003-06-12 11:20:33
Size: 402
Editor: 81
Comment:
Revision 6 as of 2003-06-16 15:59:45
Size: 1945
Editor: 213
Comment:
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
HarvestMan is a world wide web crawler(robot) program written entirely
in python using urllib2.
#pragma section-numbers off
=== Description ===
A www crawler(robot) program in python.
Line 4: Line 5:
It uses python threads to achieve a very fast download of web-sites. === Information ===
Line 6: Line 7:
The current version is 1.0.1.
HarvestMan is totally free software.
   [http://members.lycos.co.uk/anandpillai "HarvestMan Homepage"]
   version:: 1.0.1 (''[[Date(2003-06-14T00:00:00)]]'')
   licence:: An Opensource license, free for personal use.
   Python versions:: Tested on 2.2.2, 2.2.3
Line 9: Line 12:
[http://members.lycos.co.uk/anandpillai "HarvestMan Homepage"] === Deployment Platforms ===
   Tested on Windows 95-98/NT/2000, Mandrakesoft Linux 9.0.
Line 11: Line 15:
("Harvestman": Borrowed from the name of a kind of small spider found in parts of N.A) === Why should I bother ? ====
   HarvestMan uses a new threading model using python threads to
   achieve a very fast, but highly customizable download of web-sites
   on the internet.It can be used to download files from intranet
   servers.
  
   HarvestMan is a truly multi-threaded webcrawler utility
   using the threading support in python language to the fullest.
  
   It is the first webcrawler in python, which is opensource.
Line 13: Line 26:
=== Features ===

    * Fully Multithreaded
    * Number of threads configurable by user
    * Support for robots exclusion protocol
    * Filtering of urls using regular expressions
    * Filtering of server names using regular expressions
    * Control download by specifying depth of fetching
    * Configure by number of files downloadable
    * Specify timeout for individual threads
    * Control download speed by changing thread/depth
      options.

=== Who should use it ====

    HarvestMan is written for the desktop user. It is ideally
    suited for python hackers and learners. We also urge the
    avergae user to download the executable and try it. It is free ! :-)

=== Taxonomy ===
    
    HarvestMan is the name of a kind of small spider found in parts of N.A
    also called "Daddy long legs". Since this program functions as a "spider",
    and also thrives by "harvesting" links from the internet, the name
    "HarvestMan" looked very apt.

    Species: HarvestMan
    Genus: Web-spiderae
    

Description

A www crawler(robot) program in python.

Information

Deployment Platforms

  • Tested on Windows 95-98/NT/2000, Mandrakesoft Linux 9.0.

=== Why should I bother ? ====

  • HarvestMan uses a new threading model using python threads to achieve a very fast, but highly customizable download of web-sites on the internet.It can be used to download files from intranet servers.

    HarvestMan is a truly multi-threaded webcrawler utility using the threading support in python language to the fullest. It is the first webcrawler in python, which is opensource.

Features

  • Fully Multithreaded
  • Number of threads configurable by user
  • Support for robots exclusion protocol
  • Filtering of urls using regular expressions
  • Filtering of server names using regular expressions
  • Control download by specifying depth of fetching
  • Configure by number of files downloadable
  • Specify timeout for individual threads
  • Control download speed by changing thread/depth
    • options.

=== Who should use it ====

  • HarvestMan is written for the desktop user. It is ideally suited for python hackers and learners. We also urge the avergae user to download the executable and try it. It is free ! :-)

Taxonomy

  • HarvestMan is the name of a kind of small spider found in parts of N.A also called "Daddy long legs". Since this program functions as a "spider", and also thrives by "harvesting" links from the internet, the name "HarvestMan" looked very apt.

    Species: HarvestMan Genus: Web-spiderae

HarvestMan (last edited 2014-05-31 20:10:33 by MarcAndreLemburg)

Unable to edit the page? See the FrontPage for instructions.