requests模块是一个第三方模块,对urllib模块进行了封装,提供了更友好更强大的API
使用pip安装即可
requests模块的官方文档:
http://docs.python-requests.org/zh_CN/latest/
1. GET请求
In [1]: import requests In [2]: help(requests.get) //使用requests.get提交GET请求 Help on function get in module requests.api: get(url, params=None, **kwargs) Sends a GET request. :param url: URL for the new :class:`Request` object. :param params: (optional) Dictionary or bytes to be sent in the query string for the :class:`Request`. :param \*\*kwargs: Optional arguments that ``request`` takes. :return: :class:`Response <Response>` object :rtype: requests.Response In [3]: r = requests.get('https://github.com/timeline.json') In [6]: dic = {'hostname':'n3', 'ip':'3.3.3.3'} In [7]: r = requests.get('http://192.168.123.105:8000/db/', params=dic) //GET请求携带参数,不需要进行编码 In [10]: r.text Out[10]: u'GET OK' In [11]: r.status_code Out[11]: 200 In [12]: r.url Out[12]: u'http://192.168.123.105:8000/db/?ip=3.3.3.3&hostname=n3' In [11]: ua = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0'} In [12]: r = requests.get('https://www.qiushibaike.com/', headers=ua) //支持自定义User-Agent In [13]: r.status_code Out[13]: 200
2. POST请求
In [1]: import requests In [2]: import json In [3]: url = 'http://httpbin.org/post' In [4]: ua = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0'} In [5]: dic = {'key1': 'value1', 'key2': 'value2'} In [6]: r = requests.post(url, data=dic) //POST请求携带参数,不需要编码 In [7]: r.status_code Out[7]: 200 In [8]: json.loads(r.text) Out[8]: {u'args': {}, u'data': u'', u'files': {}, u'form': {u'key1': u'value1', u'key2': u'value2'}, u'headers': {u'Accept': u'*/*', u'Accept-Encoding': u'gzip, deflate', u'Connection': u'close', u'Content-Length': u'23', u'Content-Type': u'application/x-www-form-urlencoded', u'Host': u'httpbin.org', u'User-Agent': u'python-requests/2.18.4'}, u'json': None, u'origin': u'119.129.228.119', u'url': u'http://httpbin.org/post'} In [9]: r = requests.post(url, data=dic, headers=ua) //同样支持自定义User-Agent In [10]: r.status_code Out[10]: 200
3. 响应内容
Requests 会自动解码来自服务器的内容。大多数 unicode 字符集都能被无缝地解码
当你收到一个响应时,Requests 会猜测响应的编码方式,用于在你调用
Response.text
方法时对响应进行解码。Requests 首先在 HTTP 头部检测是否存在指定的编码方式,如果不存在,则会使用
charade 来尝试猜测编码方式
你可以找出 Requests 使用了什么编码,并且能够使用
r.encoding
属性来改变它
3.1 文本型相应内容(取文本)
In [14]: r = requests.get('https://github.com/timeline.json') In [15]: r.encoding Out[15]: 'utf-8' In [16]: r.text Out[16]: u'{"message":"Hello there, wayfaring stranger. If you\u2019re reading this then you probably didn\u2019t see our blog post a couple of years back announcing that this API would go away: http://git.io/17AROg Fear not, you should be able to get what you need from the shiny new Events API instead.","documentation_url":"https://developer.github.com/v3/activity/events/#list-public-events"}'
3.2 二进制型响应内容(取图片,文件)
In [17]: r.content Out[17]: '{"message":"Hello there, wayfaring stranger. If you\xe2\x80\x99re reading this then you probably didn\xe2\x80\x99t see our blog post a couple of years back announcing that this API would go away: http://git.io/17AROg Fear not, you should be able to get what you need from the shiny new Events API instead.","documentation_url":"https://developer.github.com/v3/activity/events/#list-public-events"}'
3.3 原始套接字响应内容
In [21]: r = requests.get('https://github.com/timeline.json', stream=True) In [22]: r.raw Out[22]: <urllib3.response.HTTPResponse at 0x2d380d0> In [23]: r.raw.read() Out[23]: '{"message":"Hello there, wayfaring stranger. If you\xe2\x80\x99re reading this then you probably didn\xe2\x80\x99t see our blog post a couple of years back announcing that this API would go away: http://git.io/17AROg Fear not, you should be able to get what you need from the shiny new Events API instead.","documentation_url":"https://developer.github.com/v3/activity/events/#list-public-events"}'
3.4 当Requests不能自动解码时,自定义编码来解码
[root@web spider]# cat demon1.py #!/usr/bin/env python import requests url = 'https://www.baidu.com/' r = requests.get(url, verify=False) print(r.encoding) //查看返回的文本的编码形式,在文本的头部有声明 print(r.text) [root@web spider]# python demon1.py ISO-8859-1 <!DOCTYPE html> <!--STATUS OK--><html> <head><meta http-equiv=content-type content=text/html;charset=utf-8><meta http-equiv=X-UA-Compatible content=IE=Edge><meta content=always name=referrer><link rel=stylesheet type=text/css href=https://ss1.bdstatic.com/5eN1bjq8AAUYm2zgoY3K/r/www/cache/bdorz/baidu.min.css><title>ç¾åº¦ä¸ä¸ï¼ä½ å°±ç¥é</title> //乱码
[root@web spider]# cat demon1.py #!/usr/bin/env python import requests url = 'https://www.baidu.com/' r = requests.get(url, verify=False) r.encoding = 'utf-8' //当requests模块不能正确对返回的内容解码时,需要定义r.encoding的值来解码,编码形式与返回的内容声明的编码形式一样 print(r.text) [root@web spider]# python demon1.py <!DOCTYPE html> <!--STATUS OK--><html> <head><meta http-equiv=content-type content=text/html;charset=utf-8><meta http-equiv=X-UA-Compatible content=IE=Edge><meta content=always name=referrer><link rel=stylesheet type=text/css href=https://ss1.bdstatic.com/5eN1bjq8AAUYm2zgoY3K/r/www/cache/bdorz/baidu.min.css><title>百度一下,你就知道</title>
4. 响应状态码
In [33]: r = requests.get('http://httpbin.org/get') In [34]: r.status_code Out[34]: 200 In [35]: r.status_code == requests.codes.ok //判断状态码是否为200 Out[35]: True In [36]: 301 == requests.codes.ok Out[36]: False In [37]: 302 == requests.codes.ok Out[37]: False
5. 响应头
In [48]: r = requests.get('https://www.github.com') In [49]: r.headers Out[49]: {'Status': '200 OK', 'Expect-CT': 'max-age=2592000, report-uri="https://api.github.com/_private/browser/errors"', 'X-Request-Id': 'fab96e29cdc0d636dd3ce5b51668717a', 'X-XSS-Protection': '1; mode=block', 'X-Content-Type-Options': 'nosniff', 'Content-Security-Policy': "default-src 'none'; base-uri 'self'; block-all-mixed-content; child-src render.githubusercontent.com; connect-src 'self' uploads.github.com status.github.com collector.githubapp.com api.github.com www.google-analytics.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com wss://live.github.com; font-src assets-cdn.github.com; form-action 'self' github.com gist.github.com; frame-ancestors 'none'; img-src 'self' data: assets-cdn.github.com identicons.github.com collector.githubapp.com github-cloud.s3.amazonaws.com *.githubusercontent.com; media-src 'none'; script-src assets-cdn.github.com; style-src 'unsafe-inline' assets-cdn.github.com", 'X-Runtime-rack': '0.040115', 'Transfer-Encoding': 'chunked', 'Set-Cookie': 'logged_in=no; domain=.github.com; path=/; expires=Thu, 31 Dec 2037 04:34:58 -0000; secure; HttpOnly, _gh_sess=eyJzZXNzaW9uX2lkIjoiNzczMjViNjAxNzdkYmEyM2EwZjQ2NDkzMDBmZGM1ZmYiLCJsYXN0X3JlYWRfZnJvbV9yZXBsaWNhcyI6MTUxNDY5NDg5ODYyNywiX2NzcmZfdG9rZW4iOiJpZFVNTGlERHBDT0M2MHI4MXAyY2N4dmtYU21lTlFJUHVoSW9rTng4eC9nPSJ9--196e7163a50c84523f9bd24e5b1f411c39df0455; path=/; secure; HttpOnly', 'Strict-Transport-Security': 'max-age=31536000; includeSubdomains; preload', 'Vary': 'X-PJAX, Accept-Encoding', 'Server': 'GitHub.com', 'X-GitHub-Request-Id': '8E25:166FF:3551BE5:4F1D6DD:5A4868F2', 'X-Runtime': '0.032924', 'X-UA-Compatible': 'IE=Edge,chrome=1', 'Cache-Control': 'no-cache', 'Date': 'Sun, 31 Dec 2017 04:34:58 GMT', 'X-Frame-Options': 'deny', 'Content-Type': 'text/html; charset=utf-8', 'Public-Key-Pins': 'max-age=0; pin-sha256="WoiWRyIOVNa9ihaBciRSC7XHjliYS9VwUGOIud4PB18="; pin-sha256="RRM1dGqnDFsCJXBTHky16vi1obOlCgFFn/yOhI/y+ho="; pin-sha256="k2v657xBsOVe1PQRwOsHsw3bsGT2VzIqz5K+59sNQws="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="IQBnNBEiFuhj+8x6X8XLgh01V9Ic5/V3IRQLNFFc7v4="; pin-sha256="iie1VXtL7HzAMF+/PVPR9xzT80kQxdZeJ+zduCB3uj0="; pin-sha256="LvRiGEjRqfzurezaWuj8Wie2gyHMrW5Q06LspMnox7A="; includeSubDomains', 'Content-Encoding': 'gzip'} In [51]: r.headers['Set-Cookie'] Out[51]: 'logged_in=no; domain=.github.com; path=/; expires=Thu, 31 Dec 2037 04:34:58 -0000; secure; HttpOnly, _gh_sess=eyJzZXNzaW9uX2lkIjoiNzczMjViNjAxNzdkYmEyM2EwZjQ2NDkzMDBmZGM1ZmYiLCJsYXN0X3JlYWRfZnJvbV9yZXBsaWNhcyI6MTUxNDY5NDg5ODYyNywiX2NzcmZfdG9rZW4iOiJpZFVNTGlERHBDT0M2MHI4MXAyY2N4dmtYU21lTlFJUHVoSW9rTng4eC9nPSJ9--196e7163a50c84523f9bd24e5b1f411c39df0455; path=/; secure; HttpOnly'
6. Cookie
cookie虽然不是字典类型,但是可以通过字典的方法进行取值 e.g. print(r.cookies['logged_in'])
cookie的五要素:
- name:对应的key值
- value:key对应的value
- domain:cookie所在的域,默认情况下时请求的域名
- path:cookie中的path能在域中进一步控制cookie的访问,当path=/,当前域的所有请求都可以访问这个cookie;如果path设为其他值,例如path=/test,那么只有www.domain.com/test下的请求才能访问这个cookie
- expires:cookie的过期时间,cookie作用于客户端(浏览器也是一个客户端),而session是作用于服务端,服务端的session为客户端生成一个cookie,session用于校验客户端携带的cookie访问的合法性
[root@web spider]# cat cookie1.py #!/usr/bin/env python import requests s = requests.session() s.get('https://www.github.com') print(s.cookies) print('#'*30) print(s.cookies.keys()) print('#'*30) print(s.cookies.values()) print('#'*30) for i in s.cookies: print(i.name, i.value, i.domain, i.path, i.expires) [root@web spider]# python cookie1.py <<class 'requests.cookies.RequestsCookieJar'>[<Cookie logged_in=no for .github.com/>, <Cookie _gh_sess=eyJzZXNzaW9uX2lkIjoiMTQ5ZTFhODdkN2MzYzNjYzZmODQ2MDdjMThlMzUwMjgiLCJsYXN0X3JlYWRfZnJvbV9yZXBsaWNhcyI6MTUxNjUxMDAzNDk4NiwiX2NzcmZfdG9rZW4iOiJtVmhsSDFJaURxUlRNN3dXSGxCZG5CU3RaRW5YZ2JNbkFlenRzeXFxY3pzPSJ9--b739efdc41973b7a3a976d5a8a0d9b7691ca68ab for github.com/>]> ############################## ['logged_in', '_gh_sess'] ############################## ['no','eyJzZXNzaW9uX2lkIjoiMTQ5ZTFhODdkN2MzYzNjYzZmODQ2MDdjMThlMzUwMjgiLCJsYXN0X3JlYWRfZnJvbV9yZXBsaWNhcyI6MTUxNjUxMDAzNDk4NiwiX2NzcmZfdG9rZW4iOiJtVmhsSDFJaURxUlRNN3dXSGxCZG5CU3RaRW5YZ2JNbkFlenRzeXFxY3pzPSJ9--b739efdc41973b7a3a976d5a8a0d9b7691ca68ab'] ############################## ('logged_in', 'no', '.github.com', '/', 2147662035) ('_gh_sess','eyJzZXNzaW9uX2lkIjoiMTQ5ZTFhODdkN2MzYzNjYzZmODQ2MDdjMThlMzUwMjgiLCJsYXN0X3JlYWRfZnJvbV9yZXBsaWNhcyI6MTUxNjUxMDAzNDk4NiwiX2NzcmZfdG9rZW4iOiJtVmhsSDFJaURxUlRNN3dXSGxCZG5CU3RaRW5YZ2JNbkFlenRzeXFxY3pzPSJ9--b739efdc41973b7a3a976d5a8a0d9b7691ca68ab', 'github.com', '/', None)
6.1 使用已知的cookie访问网站
[root@web spider]# cat cookie2.py #!/usr/bin/env python import requests url = 'http://httpbin.org/cookies' r = requests.get(url, cookies={'key1': 'value1', 'key2': 'value2'}) //get()提供了一个专门接收cookie的参数,参数接收一个字典 print(r.text) [root@web spider]# python cookie2.py { "cookies": { "key1": "value1", "key2": "value2" } }
7. 会话对象
会话对象让你能够跨请求保持某些参数。它也会在同一个 Session 实例发出的所有请求之间保持 cookie
In [110]: s = requests.session() //创建一个会话对象 In [111]: help(s.get) //get和post方法用法同requests.get/requests.post Help on method get in module requests.sessions: get(self, url, **kwargs) method of requests.sessions.Session instance Sends a GET request. Returns :class:`Response` object. :param url: URL for the new :class:`Request` object. :param \*\*kwargs: Optional arguments that ``request`` takes. :rtype: requests.Response In [112]: help(s.post) Help on method post in module requests.sessions: post(self, url, data=None, json=None, **kwargs) method of requests.sessions.Session instance Sends a POST request. Returns :class:`Response` object. :param url: URL for the new :class:`Request` object. :param data: (optional) Dictionary, bytes, or file-like object to send in the body of the :class:`Request`. :param json: (optional) json to send in the body of the :class:`Request`. :param \*\*kwargs: Optional arguments that ``request`` takes. :rtype: requests.Response In [113]: r = s.get('http://192.168.123.105:8000/admin') In [114]: r.status_code Out[114]: 200 In [115]: r.cookies Out[115]: <<class 'requests.cookies.RequestsCookieJar'>[Cookie(version=0, name='csrftoken', value='nDCM6HJnfOI10QazYD78vdEO2Gt2r6NO', port=None, port_specified=False, domain='192.168.123.105', domain_specified=False, domain_initial_dot=False, path='/', path_specified=True, secure=False, expires=1546137248, discard=False, comment=None, comment_url=None, rest={}, rfc2109=False)]>
8. 使用代理
[root@web spider]# cat daili1.py #!/usr/bin/env python import requests url = 'http://2017.ip138.com/ic.asp' r = requests.get(url) r.encoding = 'gb2312' print(r.text) proxies = { # 'https': 'http://122.72.18.35:80', 'http': 'http://125.88.177.128:3128' } r1 = requests.get(url, proxies=proxies) r1.encoding = 'gb2312' print(r1.text) [root@web spider]# python daili1.py <html> <head> <meta http-equiv="content-type" content="text/html; charset=gb2312"> <title> 您的IP地址 </title> </head> <body style="margin:0px"><center>您的IP是:[113.68.17.83] 来自:广东省广州市 电信</center></body></html> <html> <head> <meta http-equiv="content-type" content="text/html; charset=gb2312"> <title> 您的IP地址 </title> </head> <body style="margin:0px"><center>您的IP是:[119.129.229.185] 来自:广东省广州市 电信</center></body></html>