BeautifulSoup学习笔记5

今天准备开始一个爬虫小项目，被虐的不要不要的

网上有好多Python爬取http://www.meizitu.com/网站图片的文章。
文章中大多用到了requests库。
自己在那儿调试，总是出错。

想着用urllib加bs4试一下吧

在Python的IDLE中urlopen()方法，连网页都打不开

>>> from urllib.request import urlopen
>>> from bs4 import BeautifulSoup
>>> html = urlopen('http://www.meizitu.com')
>>> bsobj = BeautifulSoup(html,'html.parser')
>>> bsobj

>>> 
>>> bsobj.find_all('ul')
[]
>>> 

换到了Anaconda，才能打开网页：

from bs4 import BeautifulSoup
from urllib.request import urlopen
pageurl = 'http://www.meizitu.com' 
r = urlopen(pageurl)
soup = BeautifulSoup(r, "html.parser")

ul_tag = soup.find('ul')

img_list = ul_tag.find_all('img')

for i in range(len(img_list)):
    img_url = img_list[i]['src']
    print(img_url)

代码运行结果，得到了网站首页大图的链接：

runfile('C:/Users/Administrator/Desktop/meizitu1.py', wdir='C:/Users/Administrator/Desktop')
http://mm.howkuai.com/wp-content/uploads/2017a/04/17/limg.jpg
http://mm.howkuai.com/wp-content/uploads/2017a/06/03/limg.jpg
http://mm.howkuai.com/wp-content/uploads/2017a/05/11/limg.jpg
http://mm.howkuai.com/wp-content/uploads/2017a/06/04/limg.jpg
http://mm.howkuai.com/wp-content/uploads/2017a/05/12/limg.jpg
http://mm.howkuai.com/wp-content/uploads/2017a/04/07/limg.jpg
http://mm.howkuai.com/wp-content/uploads/2017a/05/05/limg.jpg
http://mm.howkuai.com/wp-content/uploads/2017a/04/18/limg.jpg
http://mm.howkuai.com/wp-content/uploads/2017a/05/16/limg.jpg
http://mm.howkuai.com/wp-content/uploads/2017a/05/14/limg.jpg
http://mm.howkuai.com/wp-content/uploads/2017a/05/17/limg.jpg

遇到了一大堆问题。

首先，中文乱码了。
alt属性对应着图片名，这里出问题了：

img_list[2]
Out[3]: <img alt="ѧ½ãÒª±ÏÒµÁË£¬´øÄã×ß½ø¡®ÉñÃØ¡¯" height="64" src="http://mm.howkuai.com/wp-content/uploads/2017a/05/11/limg.jpg" width="64"/>

准备下载图片，出现了 “HTTPError: Forbidden”的报错。可能是网站不允许直接下载图片，requests库中有headers，可以模拟浏览器登录。bs4文档看完了好好研究下requests文档

for i in range(len(img_list)):
    img_url = img_list[i]['src']
#    print(img_url)

    path = 'F:\\meizitu\\'+str(i)+'.jpg' 
    print(path)
    urlretrieve(img_url ,path)

报错：

runfile('C:/Users/Administrator/Desktop/meizitu1.py', wdir='C:/Users/Administrator/Desktop')
F:\meizitu\0.jpg
Traceback (most recent call last):

  File "<ipython-input-4-c461b5c939e9>", line 1, in <module>
    runfile('C:/Users/Administrator/Desktop/meizitu1.py', wdir='C:/Users/Administrator/Desktop')

  File "C:\ProgramData\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 880, in runfile
    execfile(filename, namespace)

  File "C:\ProgramData\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 102, in execfile
    exec(compile(f.read(), filename, 'exec'), namespace)

  File "C:/Users/Administrator/Desktop/meizitu1.py", line 37, in <module>
    urlretrieve(img_url ,path)

  File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 248, in urlretrieve
    with contextlib.closing(urlopen(url, data)) as fp:

  File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 223, in urlopen
    return opener.open(url, data, timeout)

  File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 532, in open
    response = meth(req, response)

  File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 642, in http_response
    'http', request, response, code, msg, hdrs)

  File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 570, in error
    return self._call_chain(*args)

  File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 504, in _call_chain
    result = func(*args)

  File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 650, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)

HTTPError: Forbidden

我的天啊，认真看书啊，不然出了问题，都不知道该去哪儿找答案