request

一.基本用法

1. 请求类型

    r = requests.post('http://httpbin.org/post')
    r = requests.put('http://httpbin.org/put')
    r = requests.delete('http://httpbin.org/delete')
    r = requests.head('http://httpbin.org/get')
    r = requests.options('http://httpbin.org/get')

2. GET请求

   • 参数

     1. params

         import requests
         data = {
             'name': 'germey',
             'age': 22
         }
         r = requests.get("http://httpbin.org/get", params=data)
         print(r.text)

     2. headers

         headers = {
             'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36'
         }
         r = requests.get("https://www.zhihu.com/explore", headers=headers)

         

   • json格式

      import requests 
      r = requests.get("http://httpbin.org/get")
      print(type(r.text))
      print(r.json())
      print(type(r.json()))

      

   • 抓取二进制数据

      import requests
      r = requests.get("https://github.com/favicon.ico")
      with open('favicon.ico', 'wb') as f:
          f.write(r.content)

3. POST请求

    import requests 
    data = {'name': 'germey', 'age': '22'}
    r = requests.post("http://httpbin.org/post", data=data)
    print(r.text)

    

4. 响应

   发送请求后，得到的自然就是响应。在上面的实例中，我们使用text和content获取了响应的内容。此外，还有很多属性和方法可以用来获取其他信息，比如状态码、响应头、Cookies等

    import requests 
    r = requests.get('http://www.jianshu.com')
    print(type(r.status_code), r.status_code)
    print(type(r.headers), r.headers)
    print(type(r.cookies), r.cookies)
    print(type(r.url), r.url)
    print(type(r.history), r.history)

    

二.高级用法

1. 文件上传

    

    import requests
    files = {'file': open('favicon.ico', 'rb')}
    r = requests.post("http://httpbin.org/post", files=files)
    print(r.text)

    

2. cookies

   获取Cookies

    import requests 
    r = requests.get("https://www.baidu.com")
    print(r.cookies)
    for key, value in r.cookies.items():
        print(key + '=' + value)

    <RequestsCookieJar[<Cookie BDORZ=27315 for .baidu.com/>, <Cookie __bsi=13533594356813414194_00_14_N_N_2_0303_C02F_N_N_N_0 for .www.baidu.com/>]>
    BDORZ=27315
    __bsi=13533594356813414194_00_14_N_N_2_0303_C02F_N_N_N_0

   cookies参数来设置

   不过这样就需要构造RequestsCookieJar对象，而且需要分割一下cookies

    import requests
     
    cookies = '...'
    jar = requests.cookies.RequestsCookieJar()
    headers = {
        'Host': 'www.zhihu.com',
        'User-Agent': '...'
    }
    for cookie in cookies.split(';'):
        key, value = cookie.split('=', 1)
        jar.set(key, value)
    r = requests.get("http://www.zhihu.com", cookies=jar, headers=headers)
    print(r.text)

3. 会话维持

    import requests 
    s = requests.Session()
    s.get('http://httpbin.org/cookies/set/number/123456789')
    r = s.get('http://httpbin.org/cookies')
    print(r.text)

    

4. SSL证书验证 requests还提供了证书验证的功能。当发送HTTP请求的时候，它会检查SSL证书，我们可以使用verify参数控制是否检查此证书。其实如果不加verify参数的话，默认是True，会自动验证

    import requests 
    response = requests.get('https://www.12306.cn')
    print(response.status_code)

    

5. 代理设置

    import requests 
    proxies = {
      "http": "http://10.10.1.10:3128",
      "https": "http://10.10.1.10:1080",
    } 
    requests.get("https://www.taobao.com", proxies=proxies)

   HTTP Basic Auth

    import requests 
    proxies = {
        "http": "http://user:password@10.10.1.10:3128/",
    }
    requests.get("https://www.taobao.com", proxies=proxies)

   SOCKS协议代理

    import requests 
    proxies = {
        'http': 'socks5://user:password@host:port',
        'https': 'socks5://user:password@host:port'
    }
    requests.get("https://www.taobao.com", proxies=proxies)

6. 超时设置

   import requests 
   r = requests.get("https://www.taobao.com", timeout = 1)
   print(r.status_code)
   

    

7. 身份认证

   import requests
   from requests.auth import HTTPBasicAuth 
   r = requests.get('http://localhost:5000', auth=HTTPBasicAuth('username', 'password'))
   print(r.status_code)
   

   简写 ​

   import requests 
   r = requests.get('http://localhost:5000', auth=('username', 'password'))
   print(r.status_code)
   

   OAuth认证

   pip3 install requests_oauthlib
   

    

   import requests
   from requests_oauthlib import OAuth1 
   url = 'https://api.twitter.com/1.1/account/verify_credentials.json'
   auth = OAuth1('YOUR_APP_KEY', 'YOUR_APP_SECRET',
   'USER_OAUTH_TOKEN', 'USER_OAUTH_TOKEN_SECRET')
   requests.get(url, auth=auth)
   

    

8. Prepared Request 前面介绍urllib时，我们可以将请求表示为数据结构，其中各个参数都可以通过一个Request对象来表示。这在requests里同样可以做到，这个数据结构就叫Prepared Request。

   from requests import Request, Session 
   url = 'http://httpbin.org/post'
   data = {
       'name': 'germey'
   }
   headers = {
       'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36'
   }
   s = Session()
   req = Request('POST', url, data=data, headers=headers)
   prepped = s.prepare_request(req)
   r = s.send(prepped)
   print(r.text)