pythonurllib实战教学_Python——关于爬虫基础Urllib实战

今天来实际运行一下关于爬虫的一些小玩意

首先，我们打开IDLE，今天我们查询一下关于CSDN的标题名称。

>>> import urllib

>>> import urllib.request

>>> data=urllib.request.urlopen("https://www.csdn.net").read().decode("utf-8","ignore")

>>> len(data)

385957

>>> import re

>>> pat="

(.*?)"

>>> re.compile(pat,re.S).findall(data)

['CSDN-专业IT技术社区']

#爬到硬盘的文件中

>>> urllib.request.urlretrieve("http://www.jd.com",filename="本地路径(且要保存的文件名字")

>>> urllib.request.urlretrieve("https://www.csdn.net",filename="E:\\IDLE文件\\csdn.html")

#浏览器伪装

>>> opener=urllib.request.build_opener()

>>> UA=("user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.100 Safari/537.36")

>>> opener.addheaders=[UA]

>>> urllib.request.install_opener(opener)

>>> data=urllib.request.urlopen("https://www.csdn.net").read().decode("utf-8","ignore")

Traceback (most recent call last):

File "", line 1, in

data=urllib.request.urlopen("https://www.csdn.net").read().decode("utf-8","ignore")

File "D:\python\lib\urllib\request.py", line 222, in urlopen

return opener.open(url, data, timeout)

File "D:\python\lib\urllib\request.py", line 523, in open

req = meth(req)

File "D:\python\lib\urllib\request.py", line 1268, in do_request_

for name, value in self.parent.addheaders:

ValueError: too many values to unpack (expected 2)

上面当浏览器伪装时，为什么会错误呢，是要解包的值太多