CS109 Lecture 7

CS109 Lecture 7

Data Scraping

Sources

• From a Web Sites
• With An API

Copyrights and permission

• Be careful and polite
• Give credit
• Care about media law
• Don’t be evil

Useful tags

<h1></h1>
<p></p>
<br>
<a href = 'url'>Link</a>

Useful Libraries for Scraping

• urllib
• beautifulsoup
• pattern
• LXML

Get Data From Website

url = 'url'
scource = urllib2.urlopen(url).read()

soup = bs4.BeautifulSoup(source)
soup.findAll('a') # find <a><\a> tag

tag = soup.find('a')
tag.get('href')

C = soup.findAll('p',{'class':'Event'})
t=C[0] 
t.findNextSiblings

Get Data With An API

import json # JavaScript Obejct Notation
import requests
api_key = 'mykey'
url = 'url' + api_key
scource = urllib2.urlopen(url).read()

#---simple example--------
a = {'a':1,'b':2}
s = json.dump(a) 
a2 = json.loads(s) 
#-------------------------
dataDict = json.loads(data)
dtatDict.keys()