python爬虫基础：requests、urllib2、httplib2

最基本的爬虫抓取
requests:
import requests   # python3
response = requests.get("http://wwww.baidu.com")
content = response.content
print("headers:", response.headers)
print("content:", content)


urllib2:
import urllib2  # python2
url = "http://wwww.baidu.com"
req = urllib2.urlopen(url)
content = req.read()
print "headers:", req.headers
print "content:", content


httplib2:
import httplib2   # python3
url = "http://wwww.baidu.com"
http = httplib2.Http()
response_headers, content = http.request(url, 'GET')
print("headers:", response_headers)

print("content:", content)

此外，对于带有查询字段的url,get请求一般会将请求的数据附在url之后，以？分割url和传输数据，多个参数用&连接。

requests中data为dict，json：

    data = {‘data1’：‘xxxx’，‘data2’：‘xxxxx’}

    response = requests.get(url = url,params = data)

urllib2中data为string：

    data = urllib.urlencode(data)

    full_url = url +'?' + data

    req = urllib2.urlopen(full_url)