数据爬虫---requests库

1、最基本的GET请求

import requests
response = requests.get("http://www.baidu.com/")

2、发送带header的请求

模拟浏览器，欺骗浏览器，获取和浏览器一致的内容。

url = "https://www.baidu.com/img/bd_logo1.png?where=super"
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36'}
requests.get(url, headers = headers)

3、发送带参数的请求

url问号后面的就是请求参数，也叫查询字符串。

kw = {"wd": "长城"}
response = requests.get("http://www.baidu.com/s?", params = kw, headers = headers)  #百度：长城

4、response的属性

print(type(response))
print('-'*100)
print('查看响应码:',response.status_code)  # 查看响应码，200即成功
print('-'*100)
print(type(response.text)) 
print('-'*100)
print(response.text)#查看响应内容  response.text 返回的是Unicode格式的数据
print('-'*100)
print(response.cookies) #response.content返回的字节流数据
print('-'*100)
print (response.encoding)# 查看响应头部字符编码，
print('-'*100)
print (response.url) # 查看完整url地址

response.apparent_encoding ## 查看编码

response.encoding = response.apparent_encoding
#response.encoding =‘utf-8’
response.text  #将响应内容中的乱码改为中文

5、poss请求

import requests

data = {'name': 'germey', 'age': '22'}
response = requests.post("http://httpbin.org/post", data=data)
print(response.text)

6、异常处理

import requests
from requests.exceptions import ReadTimeout, ConnectionError, RequestException
try:
    response = requests.get("http://httpbin.org/get", timeout = 0.5)
    print(response.status_code)
except ReadTimeout:
    print('Timeout')
except ConnectionError:
    print('Connection error')
except RequestException:
    print('Error')

7、requests.request

requests.request('GET','http://baidu.com')