爬虫基础----requests库

例子

>>>import requests

>>>url= 'http://example.com/'

>>>response = requests.get(url)

>>>response.status_code

200    // requests.status_code查看相应的状态，请求成功返回200，请求失败返回403

>>>response.headers['content-type']

'text/html;charset=utf-8'

>>>response.content

u'Hello,world!'

发送请求

––r = requests.get('https://github.com/timeline.json')[GET]

––r = requests.post("http://httpbin.org/post")[POST]

为URL传递参数

––payload = {'key1':'value1','key2':'value2'}

––r = requests.get("http://httpbin.org/get",params=payload)

––printr.url

––u'http://httpbin.org/get?key2=value2&key1=value1’

响应内容

––r = requests.get('https://github.com/timeline.json')

––print r.text

––'[{"repository":{"open_issues":0,"url":"https://github.com/...

响应状态码

–r = requests.get('http://httpbin.org/get')

–print r.status_code

–200 [OK] OR 403 [禁止访问]

响应头

–print r.headers

–print r.headers['Content-Type']

–'application/json; charset=utf-8’

•cookies

–print r.cookies['example_cookie_name']

超时

–requests.get('http://github.com',timeout=0.001)

–requests.exceptions.Timeout: … // 超时就抛出异常

错误不异常

–Requests异常一般都继承自requests.exceptions.RequestException

–遇到网络问题（DNS查询失败、拒绝连接等）时，Requests会抛出一个ConnectionError异常

–若请求超时，则抛出一个Timeout异常

–遇到罕见的无效HTTP响应时，Requests则会抛出一个HTTPError异常

–若请求超过了设定的最大重定向次数，则会抛出一个TooManyRedirects异常

如何在网页上抓取想要东西

正则表达式(regular expression)

––描述了一种字符串匹配的模式，可以用来检查一个串是否吨有某种子串、将匹配的子串做替换戒者从某个串中取出符合某个条件的子串等

–参考文档：http://www.runoob.com/regexp/regexp-syntax.html

Lxml

––强大的处理HTML不XML的python库

––参考文档：http://lxml.de/tutorial.html

––参考文档：https://www.ibm.com/developerworks/cn/xml/x-hiperfparse