Screen scraping 1

Screen scraping is a process whereby your program downloads Web pages and extracts information from them. Conceptually, the technique is very simple. You download the data and analyze it, you could, simply use urllib, get the Web page’s HTML source, and then use regular expressions or some such to extract the information.

use http://www.python.org/community/jobs/ as example

<h2>

<a class="reference external" href="http://www.dubizzle.com">DubizzleMiddle East</a>

(Dubai,United Arab Emirates)

</h2>

from urllib import urlopen
import re
    
p = re.compile('<h2><a .*? href="(.*?)">(.*?)</a>')
text = urlopen("http://www.python.org/community/jobs/").read()
for url, name in p.findall(text):
    print '%s (%s)' %(name, url)