python requests获取网页内容(urllib.request和requests的使用和区别)

urllib.request和requests的使用和区别 
请求网页内容可用urllib.request和requests函数,下面分别描述两种函数使用方法 

urllib.request的使用 

1.发起基础请求 

import urllib.request
req = urllib.request.Request(url)
response = urllib.request.urlopen(req)
print(response)

返回结果如下:  

返回的是一个object对象,如果想去获取内容还需进一步解析编码: 

import urllib.request
req = urllib.request.Request(url)
response = urllib.request.urlopen(req)
content  = response.read().decode("utf-8")
print(content)

返回结果如下:  

 这个时候就是网页实际html内容,但是我们在获取网页内容,例如写爬虫时往往需要带上header头信息. 
2.发起带header的请求: 

url = "https://movie.douban.com"
headers = {
    "User-Agent" : "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36"
}

req = urllib.request.Request(url=url,headers=headers)
response = urllib.request.urlopen(req)
content  = response.read().decode("utf-8")
print(content)

有时我们还需要更多的参数 3.发起带有更多参数的请求头(以供参考): 

url = "https://movie.douban.com"
headers = {
    "User-Agent" : "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36"
}
data = bytes(urllib.parse.urlencode({"hello":"world"}),encoding="utf-8")
req = urllib.request.Request(url=url,data=data,headers=headers,method="POST")
response = urllib.request.urlopen(req)
content  = response.read().decode("utf-8")
print(content)

 

requests的使用 

requests的使用看起来就比较简单方便点 1.发起请求: 

import requests
url = "https://movie.douban.com"
headers = {
    "User-Agent" : "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36"
}

req = requests.get(url=url, headers=headers)
print(req)

返回结果:  

返回的是一个状态码,这个时候requests提供一些方法来获取结果 1.req.text 得到的是str数据类型 

import requests
url = "https://movie.douban.com"
headers = {
    "User-Agent" : "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36"
}

req = requests.get(url=url, headers=headers)
print(req.text)

返回结果:  

2.req.content 得到的是Bytes类型,需要进行解码。 

import requests
url = "https://movie.douban.com"
headers = {
    "User-Agent" : "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36"
}

 

req = requests.get(url=url, headers=headers)
print(req.content)

返回结果:  

3.req.json 得到的是json数据 

import requests
url = "https://movie.douban.com"
headers = {
    "User-Agent" : "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36"
}

 

req = requests.get(url=url, headers=headers)
print(req.json)

返回结果: