主要记录 使用requests和BueatifulSoup保存图片的的功能以及遇到的问题;正好我自己平时也会碰到需要保存图片的功能;
从浏览器检查中,可以看到图片在div的class为“f14”中;
具体步骤:
1.使用requests库get到对应url网页;
2.使用BeautifulSoup库解析http返回的结果;
3.筛选出对于的div或者其他标签,即目标所在的位置;
4.从div的img中,拿到src链接;
5.requests再次get到图片,保存二进制内容到文件中;
实现的具体代码:
# 1.导入第三方库import requests
from bs4 import BeautifulSoup
url = "https://****/6527882.html"
proxies = {
"http": None,
"https": None,
}
header = {
"Cache-Control": "no-cache",
# 通过浏览器控制台获取
"Cookie": "t=823e32b0fbc8ed9f50d5a84e5d1b4397; r=5685; t=d03bd513082df044d13290320ab2b6da; r=2046; Hm_lvt_a0c29956538209f8d51a5eede55c74f9=1678506921; statistics_clientid=me; Hm_lpvt_c13cf8e9faf62071ac13fd4eafaf1acf=1678507604; Hm_lpvt_a0c29956538209f8d51a5eede55c74f9=1678507604",
# 通过浏览器控制台获取
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36 115Browser/25.0.6.5'
}
requests.packages.urllib3.disable_warnings()
page_text = requests.get(
url, headers=header, verify=False, proxies=proxies).content
page_text.decode("utf-8", "ignore")
# print(page_text)# 2.发送请求获取网站的返回值(html)
res = requests.get(url)
# print(res.text)# 3.解析返回内容
soup = BeautifulSoup(res.text, "html.parser")
# 4.寻找目标标签
divs = soup.find_all("div", class_="f14")
i = 0
header = {
'User-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36 115Browser/25.0.6.5'
}
list_1=[]
for div in divs:
img = div.find_all("img")
list_1.append(img)
print("-------------------------------------")
for image in list_1:
for item in image:
print(item["src"])
pic = requests.get(item["src"], headers=header)
withopen("D:/index"+str(i)+".jpg", "wb+") as file:
file.write(pic.content)
i += 1
![](https://i-blog.csdnimg.cn/blog_migrate/295cb38bb19ebe2706aced192b0de42e.png)
遇到的问题:
1.代码中未设置headers,网站有反爬机制,返回了403;
2.安装requests库和Beautifulsoup4库,使用pip install不太容易装;
3.有些网页使用异步加载的方式,第一次的get请求,获取到的内容为JavaScript内容;需要是使用Selenium调用浏览器初步加载,然后再获取到图片相关的内容;
使用Selenium库的具体代码:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service as ChromeService
from webdriver_manager.chrome import ChromeDriverManager
import requests
from time import sleep
import urllib
url_1 = "https://***/meinv_t50/"
service = ChromeService(executable_path=ChromeDriverManager().install())
driver = webdriver.Chrome(service=service)
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--headless')
chrome_options.add_argument('--disable-gpu')
chrome_options.add_argument('--ignore-certificate-errors')
res = driver.get(url_1)
# requests库,构造会话
session = requests.Session()
# 获取cookies
cookies = driver.get_cookies()
# 填充for cookie in cookies:
session.cookies.set(cookie['name'], cookie['value'])
# 给网页留一点加载的时间,不然之后find不了元素
sleep(2)
detail = driver.find_elements(By.CLASS_NAME, "il_img")
src_img_list=[]
for i inrange(0, len(detail)):
print(i)
img_text = detail[i].find_element(By.TAG_NAME,"img")
url_img = img_text.get_attribute("src")
print(url_img)
src_img_list.append(url_img)
header = {
'User-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW 64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36 QIHU 360SE'
}
pic = requests.get(url_img, headers=header)
withopen("D:/index"+str(i)+".jpg","wb+") as file:
file.write(pic.content)
driver.quit()
![](https://i-blog.csdnimg.cn/blog_migrate/edeeba7d754863e157d9c447794232dd.png)