python网页请求_Python请求获取Web内容（urllib.请求和请求）,pythonrequests,网页内容,urllibrequest,的,使用,区别...

urllib.request和requests的使用和区别

请求网页内容可用urllib.request和requests函数,下面分别描述两种函数使用方法

urllib.request的使用

1.发起基础请求

import urllib.request

req = urllib.request.Request(url)

response = urllib.request.urlopen(req)

print(response)

返回结果如下:

返回的是一个object对象,如果想去获取内容还需进一步解析编码:

import urllib.request

req = urllib.request.Request(url)

response = urllib.request.urlopen(req)

content = response.read().decode("utf-8")

print(content)

返回结果如下:

这个时候就是网页实际html内容,但是我们在获取网页内容,例如写爬虫时往往需要带上header头信息.

2.发起带header的请求:

url = "https://movie.douban.com"

headers = {

"User-Agent" : "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36"

}

req = urllib.request.Request(url=url,headers=headers)

response = urllib.request.urlopen(req)

content = response.read().decode("utf-8")

print(content)

有时我们还需要更多的参数

3.发起带有更多参数的请求头(以供参考):

url = "https://movie.douban.com"

headers = {

"User-Agent" : "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36"

}

data = bytes(urllib.parse.urlencode({"hello":"world"}),encoding="utf-8")

req = urllib.request.Request(url=url,data=data,headers=headers,method="POST")

response = urllib.request.urlopen(req)

content = response.read().decode("utf-8")

print(content)

requests的使用

requests的使用看起来就比较简单方便点

1.发起请求:

import requests

url = "https://movie.douban.com"

headers = {

"User-Agent" : "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36"

}

req = requests.get(url=url, headers=headers)

print(req)

返回结果:

返回的是一个状态码,这个时候requests提供一些方法来获取结果

1.req.text 得到的是str数据类型

import requests

url = "https://movie.douban.com"

headers = {

"User-Agent" : "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36"

}

req = requests.get(url=url, headers=headers)

print(req.text)

返回结果:

2.req.content 得到的是Bytes类型,需要进行解码。

import requests

url = "https://movie.douban.com"

headers = {

"User-Agent" : "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36"

}

req = requests.get(url=url, headers=headers)

print(req.content)

返回结果:

3.req.json 得到的是json数据

import requests

url = "https://movie.douban.com"

headers = {

"User-Agent" : "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36"

}

req = requests.get(url=url, headers=headers)

print(req.json)

返回结果: