最近学了学爬虫,由于平时笔者经常刷虎扑,于是决定实战一下,对虎扑上的美女图片进行爬取,特地来分享一下经验。
这次主要使用了request和BeautifulSoup两个库,使用urllib库对图片进行下载。
首先进入虎扑步行街,搜索关键字“秋名山论美”,得到如下网页:
首先f12打开网页检查,点击network然后刷新网页复制headers。根据观察发现网页的url中&page=1中的1为对应的页码,所以可以构建url列表页进行循环
for i in range(1,17): page_num = '&page=' i = str(i) url = 'https://my.hupu.com/search?q=%E3%80%90%E7%A7%8B%E5%90%8D%E5%B1%B1%E8%AE%BA%E7%BE%8E%E3%80%91'+page_num+i
之后右键每一期的标题,查看在网页源代码的位置。
使用Beautiful库中的selector选择器进行对网页url的提取
response = requests.get(url=url,headers=headers) soup = BeautifulSoup(response.content,'lxml') img_ = soup.select('.mytopic.topiclisttr tbody tr .p_title a') for _url in img_: img_url = _url['href'] url_list.append(img_url)
随后点击进入一期秋名山论美的网页,找到壁纸图片右键检查,找到图片的位置,同样使用selector选择器进行选择。
response = requests.get(url=url,headers=headers) soup = BeautifulSoup(response.content,'lxml') img_ = soup.select(".floor .floor-show .floor_box tbody tr td .quote-content img")创建文件夹,观察发现图片的url中?前面的是图片的url,所以使用split进行分割后提取之后下载和保存图片
if not os.path.exists(path):
os.mkdir(path)
os.chdir(path)
try: if '?' in img['data-original']: img_url = img['data-original'].split('?')[0] else: continue except KeyError: print(i) continue
try: content = urllib.request.urlopen(img_url) except urllib.error.HTTPError: continue content = content.read() with open(name, 'wb') as f: f.write(content) time.sleep(0.2)
最后附上源码进行学习
import requests import urllib.request import urllib.error import os import re import time from bs4 import BeautifulSoup headers = {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 UBrowser/6.2.3964.2 Safari/537.36' } url_list = [] for i in range(1,17): #第17页数字为汉字,共有15期,需要用number_i来命名 page_num = '&page=' i = str(i) url = 'https://my.hupu.com/search?q=%E3%80%90%E7%A7%8B%E5%90%8D%E5%B1%B1%E8%AE%BA%E7%BE%8E%E3%80%91'+page_num+i response = requests.get(url=url,headers=headers) soup = BeautifulSoup(response.content,'lxml') img_ = soup.select('.mytopic.topiclisttr tbody tr .p_title a') for _url in img_: img_url = _url['href'] url_list.append(img_url) # number_i=15 for url in url_list: response = requests.get(url=url,headers=headers) soup = BeautifulSoup(response.content,'lxml') img_ = soup.select(".floor .floor-show .floor_box tbody tr td .quote-content img") # ********************************************************************* title = soup.select(".subhead span")[0].string s = '' s = s.join(title) print(s) try: number = re.findall(r'第(\d+)期',s)[0] except IndexError: continue # number = str(number_i) # number_i-=1 path = 'E:\project\pachong'+'\\'+number # ************************************************************************ if not os.path.exists(path): os.mkdir(path) os.chdir(path) for i, img in enumerate(img_): i = str(i) print(img) try: if '?' in img['data-original']: img_url = img['data-original'].split('?')[0] else: continue except KeyError: print(i) continue print(i, img_url) name = number + '_' + i + '.' + 'jpg' try: content = urllib.request.urlopen(img_url) except urllib.error.HTTPError: continue content = content.read() with open(name, 'wb') as f: f.write(content) time.sleep(0.2) time.sleep(1)
![](https://i-blog.csdnimg.cn/blog_migrate/f0ca54e24d9817b71f6ab0bd0e5e4286.png)