图像分类任务之爬取需要的训练图像数据集和去除不合格的图片
一、获取数据集
相信不少学习机器学习和深度学习的小伙伴都会遇到这样的困难:训练的数据集上哪找啊,初学深度学习的我也饱受摧残,所以不多废话,先丢几个提供数据集的连接,以后说不定能用上
ImageNet:https://image-net.org/download-images
Scikit-learn:http://scikit-learn.org/stable/datasets/index.html
UCI数据集:http://archive.ics.uci.edu/ml/
Kaggle数据集:https://www.kaggle.com/datasets
还有我最近发现的国内数据集网站:http://dataju.cn/Dataju/web/home
但是,我还是有一个问题,数据集好大,好难下载,能不能有方法只下载自己用得到的数据集,还真有-----爬虫,一下代码来自网络
# -*- coding: utf-8 -*-
import re
import requests
from urllib import error
from bs4 import BeautifulSoup
import os
num = 0
numPicture = 0
file = ''
List = []
def Find(url, A):
global List
print('正在检测图片总数,请稍等.....')
t = 0
i = 1
s = 0
while t < 1000:
Url = url + str(t)
try:
# 这里搞了下
Result = A.get(Url, timeout=7, allow_redirects=False)
except BaseException:
t = t + 60
continue
else:
result = Result.text
pic_url = re.findall('"objURL":"(.*?)",', result, re.S) # 先利用正则表达式找到图片url
s += len(pic_url)
if len(pic_url) == 0:
break
else:
List.append(pic_url)
t = t + 60
return s
def recommend(url):
Re = []
try:
html = requests.get(url, allow_redirects=False)
except error.HTTPError as e:
return
else:
html.encoding = 'utf-8'
bsObj = BeautifulSoup(html.text, 'html.parser')
div = bsObj.find('div', id='topRS')
if div is not None:
listA = div.findAll('a')
for i in listA:
if i is not None:
Re.append(i.get_text())
return Re
def dowmloadPicture(html, keyword):
global num
# t =0
pic_url = re.findall('"objURL":"(.*?)",', html, re.S) # 先利用正则表达式找到图片url
print('找到关键词:' + keyword + '的图片,即将开始下载图片...')
for each in pic_url:
print('正在下载第' + str(num + 1) + '张图片,图片地址:' + str(each))
try:
if each is not None:
pic = requests.get(each, timeout=7)
else:
continue
except BaseException:
print('错误,当前图片无法下载')
continue
else:
string = file + r'\\' + keyword + '_' + str(num) + '.jpg'
fp = open(string, 'wb')
fp.write(pic.content)
fp.close()
num += 1
if num >= numPicture:
return
if __name__ == '__main__': # 主函数入口
##############################
# 这里加了点
headers = {
'Accept-Language': 'zh-CN,zh;q=0.8,zh-TW;q=0.7,zh-HK;q=0.5,en-US;q=0.3,en;q=0.2',
'Connection': 'keep-alive',
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0',
'Upgrade-Insecure-Requests': '1'
}
A = requests.Session()
A.headers = headers
###############################
word = input("请输入搜索关键词(可以是人名,地名等): ")
# add = 'http://image.baidu.com/search/flip?tn=baiduimage&ie=utf-8&word=%E5%BC%A0%E5%A4%A9%E7%88%B1&pn=120'
url = 'https://image.baidu.com/search/flip?tn=baiduimage&ie=utf-8&word=' + word + '&pn='
# 这里搞了下
tot = Find(url, A)
Recommend = recommend(url) # 记录相关推荐
print('经过检测%s类图片共有%d张' % (word, tot))
numPicture = int(input('请输入想要下载的图片数量 '))
file = input('请建立一个存储图片的文件夹,输入文件夹名称即可')
y = os.path.exists(file)
if y == 1:
print('该文件已存在,请重新输入')
file = input('请建立一个存储图片的文件夹,)输入文件夹名称即可')
os.mkdir(file)
else:
os.mkdir(file)
t = 0
tmp = url
while t < numPicture:
try:
url = tmp + str(t)
# 这里搞了下
result = A.get(url, timeout=10, allow_redirects=False)
except error.HTTPError as e:
print('网络错误,请调整网络后重试')
t = t + 60
else:
dowmloadPicture(result.text, word)
t = t + 60
print('当前搜索结束,感谢使用')
print('猜你喜欢')
for re in Recommend:
print(re, end=' ')
运行效果
下载结果
二、去除不完整的图片
从网上爬取的图片,我们既不能保证图片格式的正确性(如PNG,GIF格式的图片直接改JPG后缀),也不能保证图片的完整性(如下载图片不完整),比如出现如下的状况
图一:图片格式强行转化的结果
图二:不完整的JPEG
经过大量的搜索,找到了较好的解决办法,参考博客https://blog.csdn.net/kingroc/article/details/90477650
import os
import shutil
from PIL import Image
DirList = [
'C://Users//94244//jupyterProject//dataset//pet'
]
def is_valid_jpg(jpg_file):
with open(jpg_file, 'rb') as f:
f.seek(-2, 2)
buf = f.read()
return buf == b'\xff\xd9'
def is_valid_png(png_file):
with open(png_file, 'rb') as f:
f.seek(-3, 2)
buf = f.read()
if buf == b'\x60\x82\x00':
return True
elif buf[1:] == b'\x60\x82':
return True
else:
return False
def is_valid_pic(pic_file):
if pic_file.endswith('jpeg'):
return is_valid_jpg(pic_file)
elif pic_file.endswith('png'):
return is_valid_png(pic_file)
else:
return False
for path in DirList:
for file in os.listdir(path):
pic_file = os.path.join(path, file)
if not is_valid_pic(pic_file):
try:
img = Image.open(pic_file)
img.load()
except Exception as e:
#print(e)
print(pic_file)
#shutil.copy(pic_file, '/home/king/Desktop/')
#os.remove(pic_file)
通过运行这段代码指定数据集路径,找出不合格的图片直接删除,这样就不会出现Tensorflow Invalid JPEG data or crop window, data size的问题啦
图三:查找结果