• WebClientProgramming

Client-Side Web Programming

Libraries

• utidylib and mxTidy -- Python interfaces to html tidy library to clean up HTML documents.

• html5lib A HTML5-compliant library for parsing arbitarily-broken HTML to a range of tree formats including minidom, elementtree (including lxml) and BeautifulSoup

• BeautifulSoup -- a permissive HTML parser.

• Don't use HTMLParser on HTML that might be invalid! That way lies pain. Either clean it up (using tidy), or use a different parser.

• urllib, urllib2, and httplib in the standard library.

• ClientCookie, ClientForm, and Mechanize are higher-level libraries for writing a web client.

• mechanoid a mechanize fork.

• libxml2dom can parse HTML by employing libxml2's liberal HTML parser.

Resources

• Grab a document from the web - from the Python Cookbook

• Python web-client programming general FAQs.

• urllib -- Open arbitrary resources by URL

• urllib2 -- extensible library for opening URLs