1.在pycharm中新建空项目:名字为爬虫课程2
2.新建images文件夹,用于存放爬取到的图片
3.新建文件:爬取206页图片.py,并添加以下代码:
代码解释:
# 导入需要的库
import requests
from lxml import etree
headers = {
"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"
}
# url 是你需要爬取图片的网页
url = "https://pic.netbian.com/4kfengjing/index.html"
url_add = 1
for i in range(0, 205):
url_add += 1
# url返回的是一个字符串类型,这里把它转换为列表类型,因为网页中有206个网页的图片要爬,所以需要循环206次,以及更改206次网页地址
url_list = list(url)
# 因为网页地址是规则的,每次循环都在列表的倒数第五个元素上加一
url_add_list = url_list.insert(-5, '_'+str(url_add))
# 再把列表转换为字符串
url_str = ''.join(url_list)
# reponse是模拟发送请求
response = requests.get(url_str, headers)
# content是获取整个网页,字符集编码为gbk
content = response.content.decode('gbk')
html = etree.HTML(content)
# 以下代码分析网页的结构,找到要爬取图片的位置,然后写入images中
ul_list_xpath = html.xpath("//ul[@class='clearfix']")[0]
li_a_res_xpath = ul_list_xpath.xpath("./li/a[@target='_blank']")
for a in li_a_res_xpath[0:len(li_a_res_xpath) -1]:
img_res = a.xpath("./img")[0]
img_url = img_res.xpath("./@src")[0]
img_title = img_res.xpath("./@alt")[0]
filename = 'images/%s.jpg' % img_title
with open(filename,'wb') as f:
img = requests.get('https://pic.netbian.com' + img_url,headers).content
f.write(img)
print("图片[{}]下载并保存成功".format(filename))
4.执行代码,图片就爬取到了images文件夹中,我爬取的是放入images2文件中的,一共3889张图片