python requests获取网页内容(urllib.request和requests的使用和区别)

urllib.request和requests的使用和区别

请求网页内容可用urllib.request和requests函数,下面分别描述两种函数使用方法

urllib.request的使用

1.发起基础请求

import urllib.request
req = urllib.request.Request(url)
response = urllib.request.urlopen(req)
print(response)

返回结果如下:

返回的是一个object对象,如果想去获取内容还需进一步解析编码:

import urllib.request
req = urllib.request.Request(url)
response = urllib.request.urlopen(req)
content  = response.read().decode("utf-8")
print(content)

返回结果如下:

这个时候就是网页实际html内容,但是我们在获取网页内容,例如写爬虫时往往需要带上header头信息.

2.发起带header的请求:

url = "https://movie.douban.com"
headers = {
    "User-Agent" : "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36"
}

req = urllib.request.Request(url=url,headers=headers)
response = urllib.request.urlopen(req)
content  = response.read().decode("utf-8")
print(content)

有时我们还需要更多的参数
3.发起带有更多参数的请求头(以供参考):

url = "https://movie.douban.com"
headers = {
    "User-Agent" : "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36"
}
data = bytes(urllib.parse.urlencode({"hello":"world"}),encoding="utf-8")
req = urllib.request.Request(url=url,data=data,headers=headers,method="POST")
response = urllib.request.urlopen(req)
content  = response.read().decode("utf-8")
print(content)

requests的使用

requests的使用看起来就比较简单方便点
1.发起请求:

import requests
url = "https://movie.douban.com"
headers = {
    "User-Agent" : "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36"
}

req = requests.get(url=url, headers=headers)
print(req)

返回结果:

返回的是一个状态码,这个时候requests提供一些方法来获取结果
1.req.text 得到的是str数据类型

import requests
url = "https://movie.douban.com"
headers = {
    "User-Agent" : "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36"
}

req = requests.get(url=url, headers=headers)
print(req.text)

返回结果:

2.req.content 得到的是Bytes类型,需要进行解码。

import requests
url = "https://movie.douban.com"
headers = {
    "User-Agent" : "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36"
}

req = requests.get(url=url, headers=headers)
print(req.content)

返回结果:

3.req.json 得到的是json数据

import requests
url = "https://movie.douban.com"
headers = {
    "User-Agent" : "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36"
}

req = requests.get(url=url, headers=headers)
print(req.json)

返回结果: