模拟浏览器是为了爬更多的内容,爬虫程序默认请求头用户代理payrhon-request
容易被识别为程序,所以需要更改user agent,模拟为浏览器
先找到一个任意网站,点F12->network->all
找到Headers栏下request headers,复制User-Agent内容
import urllib.request
url = "http://blog.csdn.net"
headers = ("User-Agent",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36")
opener = urllib.request.build_opener()
opener.addheaders = [headers]
data = opener.open(url).read()
fh = open("C:/Users/Nicht_sehen/Desktop/csdn.html","wb")
fh.write(data)
fh.close()