对CSDN首页进行一次简单爬取
知识点:
urllib主要包含四个模块:
1.request: 模拟发送请求
2.error: 处理异常
3.parse: 提供了URL的处理方法
4.robotparser: 用于识别网站的robot.txt文件在这里插入代码片
urlopen方法:
urlopen(url, data=None, [timeout,]*, cafile=None,
capath=Node, cadefault=False, context=None,
) # 默认使用GET方式请求
Request方法:
class urllib.request.Request(url, data=None, headers={},
origin_req_host=None, unverifiable=False, method=None
)
参数含义:
url:请求信息 必选参数
data: 附加数据 可选参数
headers: 请求头(能通过修改User-Agent让我们伪装成浏览器)
origin_req_host:请求方的host名称或IP地址
unverifiable: 权限验证
method: 用于指示请求的方式(GET/POST)
简单爬取
from urllib import request, parse, error
import socket
response = urllib.request.urlopen(
'https://www.csdn.net/?spm=1011.2124.3001.4476'
)
print(response.read().decode('utf-8'))
print(response.status)
print(response.getheaders())
print(response.getheader('Server'))
添加参数 使用请求测试网站进行测试
from urllib import request, parse, error
import socket
data = bytes(urllib.parse.urlencode({'word': 'hello'}), encoding='utf8')
try:
response = urllib.request.urlopen('https://httpbin.org/post', data=data, timeout=5)
except urllib.error.URLError as e:
if isinstance(e.reason, socket.timeout):
print("响应超时!")
else:
print(response.read())
使用Request方法进行访问
from urllib import request, parse, error
import socket
url = 'https://httpbin.org/post'
headers = {
'User-Agent': 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)'
}
dict = {
'name': 'sunqg'
}
data = bytes(parse.urlencode(dict), encoding='utf8')
req = request.Request(url=url, data=data, headers=headers, method='POST')
response = request.urlopen(req)
print(response.read().decode('utf8'))