Browsing in Python with Mechanize
Browsing with Mechanize
The mechanize module in Python is similar to perl WWW:Mechanize.
It gives you a browser like object to interact with web pages.
Here is an example on how to use it in a program.
import mechanize
br = mechanize.Browser()
br.open("http://www.example.com/")
Follow second link with element text matching regular expression
response1 = br.follow_link(text_regex=r"cheeses*shop", nr=1)
assert br.viewing_html()
print br.title()
print response1.geturl()
print response1.info() # headers
print response1.read() # body
To get the response code from a website, you can the response.code
from mechanize import Browser
browser = Browser()
response = browser.open('http://www.google.com')
print response.code
Get all forms from a website
import mechanize
br = mechanize.Browser()
br.open("http://www.google.com/")
for f in br.forms():
print f
I found this post at http://stockrt.github.com that very accurate describes how
to emulate a browser in Python using mechanize.
Browsing with Python (written of Drew Stephens)
#!/usr/bin/python
import re
from mechanize import Browser
br = Browser()
Ignore robots.txt
br.set_handle_robots( False )
Google demands a user-agent that isn't a robot
br.addheaders = [('User-agent', 'Firefox')]
Retrieve the Google home page, saving the response
br.open( "http://google.com" )
Select the search box and search for 'foo'
br.select_form( 'f' )
br.form[ 'q' ] = 'foo'
Get the search results
br.submit()
Find the link to foofighters.com; why did we run a search?
resp = None
for link in br.links():
siteMatch = re.compile( 'www.foofighters.com' ).search( link.url )
if siteMatch:
resp = br.follow_link( link )
break
Print the site
content = resp.get_data()
print content
The script above is split up to make it easier to read
