python urllib爬虫_python+urllib爬虫

导入包

import urllib.request

import urllib.parse

获取get请求

response = urllib.request.urlopen("http://httpbin.org/get")

print(response.read().decode('utf-8'))

get超时(预备方案，问题最后解决使程序正常运行)

try:

response = urllib.request.urlopen("http://httpbin.org/get", timeout=0.01)

print(response.read().decode('utf-8'))

except urllib.error.URLError as e:

print("time out")

获取post请求

data = bytes(urllib.parse.urlencode({"hello":"world"}), encoding="utf-8")

response = urllib.request.urlopen("http://httpbin.org/post", data=data)

print(response.read().decode('utf-8'))

post伪装浏览器，解决418

1、User-Agent获取

随便打开一个网站(百度为例)，点F12(或者鼠标右键点击检查)，在点击Network

重新刷新网页(或者点击F5)，点击iconfont-9f2f4dde78.woff2

复制User-Agent

2、代码

url = "http://httpbin.org/post"

headers = {

"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.102 Safari/537.36"

}

data = bytes(urllib.parse.urlencode({'name':'lucy'}), encoding="utf-8")

req = urllib.request.Request(url=url,data=data,headers=headers,method="POST")

response = urllib.request.urlopen(req)

print(response.read().decode('utf-8'))