python爬虫系列(2.2-requests库的高级使用)

一、设置代理ip

• 1、直接在请求的时候加上proxies就可以,注意我们一般会写上http和https的,这样当遇到http请求就会走http字典对应的代理

• 2、具体代码

  import requests
  
  if __name__ == "__main__":
      # 定义一个请求头(模拟浏览器)
      headers = {
          'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.162 Safari/537.36'}
  
      # 定义代理的字典(可以去http://www.xicidaili.com查找最新的代理)
      proxies = {
          'http': 'http://115.223.223.29:9000',
          'https': 'https://197.232.21.141:59075'
      }
      response = requests.get('http://httpbin.org/get', headers=headers, proxies=proxies)
      response.encoding = 'utf-8'
      print(response.text)
  

二、关于requests库操作cookie

• 1、从网站上获取cookie

  import requests
  
  if __name__ == "__main__":
      # 定义一个请求头(模拟浏览器)
      headers = {
          'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.162 Safari/537.36'}
      response = requests.get('https://www.baidu.com', headers=headers)
      # 迭代出全部的
      for k, v in response.cookies.items():
          print(k, '===', v)
      print(response.cookies)
      print(response.cookies.get_dict())
  

• 2、使用session会话存储当前网上的cookies

  import requests
  
  if __name__ == "__main__":
      url = 'https://www.lagou.com/jobs/positionAjax.json?needAddtionalResult=false'
      headers = {
          'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.162 Safari/537.36',
          'Referer': 'https://www.lagou.com/jobs/list_python?labelWords=&fromSearch=true&suginput='
      }
      data = {
          'first': 'true',
          'pn': '1',
          'kd': 'python',
      }
      # 使用session包装下
      session = requests.session()
      response = session.post(url=url, headers=headers, data=data)
      print(response.json())
  

三、不安全证书网站的请求(12306是最典型的)

• 1、只需要在请求的时候加上verify=False就可以

• 2、具体代码

  import requests
  
  if __name__ == "__main__":
      # 定义一个请求头(模拟浏览器)
      headers = {
          'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.162 Safari/537.36'}
      response = requests.get('http://www.12306.cn/mormhweb/', headers=headers, verify=False)
      response.encoding = 'utf-8'
      print(response.text)
  

四、更多关于博主文章请访问