DeepAugment数据集增强算法的安装

DeepAugment数据集增强算法的安装

DeepAugment

DeepAugment是一个专注于数据扩充的AutoML工具。
它利用贝叶斯优化来发现针对您的图像数据集定制的数据增强策略。

DeepAugment的主要优点和特点是：
(1)降低CNN模型的错误率（在WRN-28-10上显示CIFAR10误差减少60％）
(2)通过自动化流程节省时间
(3)比谷歌之前的解决方案–AutoAugment 快50倍

官网地址:https://blog.insightdatascience.com/automl-for-data-augmentation-e87cf692c366
Github:https://github.com/barisozmen/deepaugment

PIP安装

环境配置

(1)确保python版本在3.6
(2)已安装tensorflow1.12.0版本
(3)已安装shapely和imgaug

安装Tensorflow1.12.0版本

(1)通过此pip命令直接安装

>pip install https://storage.googleapis.com/tensorflow/windows/cpu/tensorflow-1.12.0-cp36-cp36m-win_amd64.whl

这个方法可能导致后续使用TensorFlow模块时
弹出错误Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2的报错

解决方法:
首先，卸载原来版本的TensorFlow

pip uninstall tensorflow

然后根据第二步的方法找到支持cpu用AVX2编译的TensorFlow版本

(2)通过访问Tensorflow的Github寻找相应版本
windows:https://github.com/fo40225/tensorflow-windows-wheel
Mac&Linux:https://github.com/lakshayg/tensorflow-build

安装shapely和imgaug

再没有安装shapely和imgaug时如果直接安装deepaugment就会出现如下报错

Collecting Shapely (from imgaug==0.2.7->deepaugment)
  Using cached https://files.pythonhosted.org/packages/a2/fb/7a7af9ef7a35d16fa23b127abee272cfc483ca89029b73e92e93cdf36e6b/Shapely-1.6.4.post2.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\hjnui\AppData\Local\Temp\pip-install-_ec0vk4a\Shapely\setup.py", line 80, in <module>
        from shapely._buildcfg import geos_version_string, geos_version, \
      File "C:\Users\hjnui\AppData\Local\Temp\pip-install-_ec0vk4a\Shapely\shapely\_buildcfg.py", line 200, in <module>
        lgeos = CDLL("geos_c.dll")
      File "c:\users\hjnui\appdata\local\programs\python\python36\lib\ctypes\__init__.py", line 348, in __init__
        self._handle = _dlopen(self._name, mode)
    OSError: [WinError 126] 找不到指定的模块。

分析代码后发现是imgaug出现问题导致安装失败

解决方法:
(1)进入这个网址https://www.lfd.uci.edu/~gohlke/pythonlibs/#shapely 找到适合自己python版本的shapely下载到指定路径

然后通过pip安装Shapely

>python -m pip install Shapely-1.6.4.post1-cp36-cp36m-win_amd64.whl
Processing g:\python\pyhton36\scripts\shapely-1.6.4.post1-cp36-cp36m-win_amd64.whl
Installing collected packages: Shapely
Successfully installed Shapely-1.6.4.post1

(2)安装imgaug

>pip install imgaug
Collecting imgaug
  Using cached https://files.pythonhosted.org/packages/af/fc/c56a7da8c23122b7c5325b941850013880a7a93c21dc95e2b1ecd4750108/imgaug-0.2.7-py3-none-any.whl
Requirement already satisfied: Shapely in g:\python\pyhton36\lib\site-packages (from imgaug) (1.6.4.post1)
Requirement already satisfied: six in g:\python\pyhton36\lib\site-packages (from imgaug) (1.12.0)
Requirement already satisfied: Pillow in g:\python\pyhton36\lib\site-packages (from imgaug) (5.3.0)
Requirement already satisfied: scikit-image>=0.11.0 in g:\python\pyhton36\lib\site-packages (from imgaug) (0.14.1)
Requirement already satisfied: matplotlib in g:\python\pyhton36\lib\site-packages (from imgaug) (3.0.2)
Collecting imageio (from imgaug)
  Using cached https://files.pythonhosted.org/packages/28/b4/cbb592964dfd71a9de6a5b08f882fd334fb99ae09ddc82081dbb2f718c81/imageio-2.4.1.tar.gz
Requirement already satisfied: scipy in g:\python\pyhton36\lib\site-packages (from imgaug) (1.2.0)
Requirement already satisfied: numpy>=1.7.0 in g:\python\pyhton36\lib\site-packages (from imgaug) (1.15.4)
Requirement already satisfied: dask[array]>=0.9.0 in g:\python\pyhton36\lib\site-packages (from scikit-image>=0.11.0->imgaug) (1.0.0)
Requirement already satisfied: cloudpickle>=0.2.1 in g:\python\pyhton36\lib\site-packages (from scikit-image>=0.11.0->imgaug) (0.6.1)
Requirement already satisfied: networkx>=1.8 in g:\python\pyhton36\lib\site-packages (from scikit-image>=0.11.0->imgaug) (2.2)
Requirement already satisfied: PyWavelets>=0.4.0 in g:\python\pyhton36\lib\site-packages (from scikit-image>=0.11.0->imgaug) (1.0.1)
Requirement already satisfied: kiwisolver>=1.0.1 in g:\python\pyhton36\lib\site-packages (from matplotlib->imgaug) (1.0.1)
Requirement already satisfied: cycler>=0.10 in g:\python\pyhton36\lib\site-packages (from matplotlib->imgaug) (0.10.0)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in g:\python\pyhton36\lib\site-packages (from matplotlib->imgaug) (2.3.0)
Requirement already satisfied: python-dateutil>=2.1 in g:\python\pyhton36\lib\site-packages (from matplotlib->imgaug) (2.7.5)
Requirement already satisfied: toolz>=0.7.3; extra == "array" in g:\python\pyhton36\lib\site-packages (from dask[array]>=0.9.0->scikit-image>=0.11.0->imgaug) (0.9.0)
Requirement already satisfied: decorator>=4.3.0 in g:\python\pyhton36\lib\site-packages (from networkx>=1.8->scikit-image>=0.11.0->imgaug) (4.3.0)
Requirement already satisfied: setuptools in g:\python\pyhton36\lib\site-packages (from kiwisolver>=1.0.1->matplotlib->imgaug) (39.0.1)
Building wheels for collected packages: imageio
  Running setup.py bdist_wheel for imageio ... done
  Stored in directory: C:\Users\34905\AppData\Local\pip\Cache\wheels\e0\43\31\605de9372ceaf657f152d3d5e82f42cf265d81db8bbe63cde1
Successfully built imageio
Installing collected packages: imageio, imgaug

解决报错问题

安装DeepAugment

通过pip命令直接安装

pip install deepaugment

安装成功后显示

Collecting Shadowsocks.lnk
  Could not find a version that satisfies the requirement Shadowsocks.lnk (from versions: )
No matching distribution found for Shadowsocks.lnk

C:\Users\hjnui\Desktop>pip install Shapely-1.6.4.post1-cp36-cp36m-win_amd64.whl
Processing c:\users\hjnui\desktop\shapely-1.6.4.post1-cp36-cp36m-win_amd64.whl
Installing collected packages: Shapely
Successfully installed Shapely-1.6.4.post1

C:\Users\hjnui\Desktop>pip install imgaug
Collecting imgaug
  Using cached https://files.pythonhosted.org/packages/f2/e3/4e19b465d6662e841e2960d77aa187414315b463302142a001a389cdaf2a/imgaug-0.2.8-py2.py3-none-any.whl
Requirement already satisfied: numpy>=1.15.0 in c:\users\hjnui\appdata\local\programs\python\python36\lib\site-packages (from imgaug) (1.16.2)
Requirement already satisfied: Shapely in c:\users\hjnui\appdata\local\programs\python\python36\lib\site-packages (from imgaug) (1.6.4.post1)
Collecting Pillow (from imgaug)
  Using cached https://files.pythonhosted.org/packages/e6/37/fc5bde88462cf8a069e522dd6a0d259b69faba1326f0a3ce8b4e11d9f317/Pillow-6.0.0-cp36-cp36m-win_amd64.whl
Collecting opencv-python (from imgaug)
  Downloading https://files.pythonhosted.org/packages/96/30/99bd865802cd5f425c42efd2ee4e10bd3bc605640008f03e3c72a1dbe320/opencv_python-4.0.0.21-cp36-cp36m-win_amd64.whl (30.4MB)
    100% |████████████████████████████████| 30.4MB 328kB/s
Collecting matplotlib (from imgaug)
  Downloading https://files.pythonhosted.org/packages/21/4c/35fa1837a705f33621604a1967b1505bd3f695940fdf02fad77ef11de196/matplotlib-3.0.3-cp36-cp36m-win_amd64.whl (9.1MB)
    100% |████████████████████████████████| 9.1MB 3.4MB/s
Collecting scikit-image>=0.11.0 (from imgaug)
  Using cached https://files.pythonhosted.org/packages/24/d5/a016b686ba4030c7e6039033bef899ff8a790819479016b2365eb929eacc/scikit_image-0.15.0-cp36-none-win_amd64.whl
Collecting imageio (from imgaug)
  Using cached https://files.pythonhosted.org/packages/af/0a/943c965d372dae0b1f1482677d29030ab834351a61a9a632fd62f27f1523/imageio-2.5.0-py3-none-any.whl
Requirement already satisfied: six in c:\users\hjnui\appdata\local\programs\python\python36\lib\site-packages (from imgaug) (1.12.0)
Collecting scipy (from imgaug)
  Using cached https://files.pythonhosted.org/packages/b9/a2/62f77d2d3c42364d45ba714b4bdf7e1c4dfa67091dc9f614fa5a948b4fb4/scipy-1.2.1-cp36-cp36m-win_amd64.whl
Collecting kiwisolver>=1.0.1 (from matplotlib->imgaug)
  Using cached https://files.pythonhosted.org/packages/44/72/16630c3392eba03788ad87949390516bbc488e8e118047a3b824631d21a6/kiwisolver-1.0.1-cp36-none-win_amd64.whl
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in c:\users\hjnui\appdata\local\programs\python\python36\lib\site-packages (from matplotlib->imgaug) (2.3.1)
Collecting cycler>=0.10 (from matplotlib->imgaug)
  Using cached https://files.pythonhosted.org/packages/f7/d2/e07d3ebb2bd7af696440ce7e754c59dd546ffe1bbe732c8ab68b9c834e61/cycler-0.10.0-py2.py3-none-any.whl
Collecting python-dateutil>=2.1 (from matplotlib->imgaug)
  Using cached https://files.pythonhosted.org/packages/41/17/c62faccbfbd163c7f57f3844689e3a78bae1f403648a6afb1d0866d87fbb/python_dateutil-2.8.0-py2.py3-none-any.whl
Collecting PyWavelets>=0.4.0 (from scikit-image>=0.11.0->imgaug)
  Downloading https://files.pythonhosted.org/packages/1f/72/ff6ddab8fbfa9d301067ecf63c0662be07bff4fe074602346f4eba856ed8/PyWavelets-1.0.2-cp36-none-win_amd64.whl (4.2MB)
    100% |████████████████████████████████| 4.2MB 518kB/s
Collecting networkx>=2.0 (from scikit-image>=0.11.0->imgaug)
  Downloading https://files.pythonhosted.org/packages/f3/f4/7e20ef40b118478191cec0b58c3192f822cace858c19505c7670961b76b2/networkx-2.2.zip (1.7MB)
    100% |████████████████████████████████| 1.7MB 74kB/s
Requirement already satisfied: setuptools in c:\users\hjnui\appdata\local\programs\python\python36\lib\site-packages (from kiwisolver>=1.0.1->matplotlib->imgaug) (33.1.1)
Collecting decorator>=4.3.0 (from networkx>=2.0->scikit-image>=0.11.0->imgaug)
  Downloading https://files.pythonhosted.org/packages/5f/88/0075e461560a1e750a0dcbf77f1d9de775028c37a19a346a6c565a257399/decorator-4.4.0-py2.py3-none-any.whl
Building wheels for collected packages: networkx
  Building wheel for networkx (setup.py) ... done
  Stored in directory: C:\Users\hjnui\AppData\Local\pip\Cache\wheels\68\f8\29\b53346a112a07d30a5a84d53f19aeadaa1a474897c0423af91
Successfully built networkx
Installing collected packages: Pillow, opencv-python, kiwisolver, cycler, python-dateutil, matplotlib, PyWavelets, imageio, decorator, networkx, scikit-image, scipy, imgaug
Successfully installed Pillow-6.0.0 PyWavelets-1.0.2 cycler-0.10.0 decorator-4.4.0 imageio-2.5.0 imgaug-0.2.8 kiwisolver-1.0.1 matplotlib-3.0.3 networkx-2.2 opencv-python-4.0.0.21 python-dateutil-2.8.0 scikit-image-0.15.0 scipy-1.2.1

C:\Users\hjnui\Desktop>pip install deepaugment
Collecting deepaugment
  Using cached https://files.pythonhosted.org/packages/99/f9/40211d827039df475091639c6aded9a1786849f898b9c619e24c15efc82a/deepaugment-1.1.2-py2.py3-none-any.whl
Collecting click==7.0 (from deepaugment)
  Using cached https://files.pythonhosted.org/packages/fa/37/45185cb5abbc30d7257104c434fe0b07e5a195a6847506c074527aa599ec/Click-7.0-py2.py3-none-any.whl
Collecting scikit-optimize==0.5.2 (from deepaugment)
  Using cached https://files.pythonhosted.org/packages/f4/44/60f82c97d1caa98752c7da2c1681cab5c7a390a0fdd3a55fac672b321cac/scikit_optimize-0.5.2-py2.py3-none-any.whl
Collecting keras==2.2.4 (from deepaugment)
  Using cached https://files.pythonhosted.org/packages/5e/10/aa32dad071ce52b5502266b5c659451cfd6ffcbf14e6c8c4f16c0ff5aaab/Keras-2.2.4-py2.py3-none-any.whl
Collecting pandas==0.23.4 (from deepaugment)
  Using cached https://files.pythonhosted.org/packages/0e/67/def5bfaf4d3324fdb89048889ec523c0903c5efab1a64c8dbe0ac8eec13c/pandas-0.23.4-cp36-cp36m-win_amd64.whl
Collecting opencv-contrib-python (from deepaugment)
  Using cached https://files.pythonhosted.org/packages/28/ef/af0d5ebec5eaf69d19caf1be2c82c99b21d0a3314152dae251329d5c3f3e/opencv_contrib_python-4.0.0.21-cp36-cp36m-win_amd64.whl
Collecting matplotlib==3.0.2 (from deepaugment)
  Using cached https://files.pythonhosted.org/packages/b1/56/569c83515c10146fd0aa09e086816b12e301d0811048e3354a6e9b77ba9a/matplotlib-3.0.2-cp36-cp36m-win_amd64.whl
Collecting keras-applications==1.0.6 (from deepaugment)
  Using cached https://files.pythonhosted.org/packages/3f/c4/2ff40221029f7098d58f8d7fb99b97e8100f3293f9856f0fb5834bef100b/Keras_Applications-1.0.6-py2.py3-none-any.whl
Collecting setuptools==40.6.3 (from deepaugment)
  Using cached https://files.pythonhosted.org/packages/37/06/754589caf971b0d2d48f151c2586f62902d93dc908e2fd9b9b9f6aa3c9dd/setuptools-40.6.3-py2.py3-none-any.whl
Requirement already satisfied: tensorflow==1.12.0 in c:\users\hjnui\appdata\local\programs\python\python36\lib\site-packages (from deepaugment) (1.12.0)
Collecting numpy==1.15.4 (from deepaugment)
  Using cached https://files.pythonhosted.org/packages/51/70/7096a735b27359dbc0c380b23b9c9bd05fea62233f95849c43a6b02c5f40/numpy-1.15.4-cp36-none-win_amd64.whl
Collecting imgaug==0.2.7 (from deepaugment)
  Using cached https://files.pythonhosted.org/packages/af/fc/c56a7da8c23122b7c5325b941850013880a7a93c21dc95e2b1ecd4750108/imgaug-0.2.7-py3-none-any.whl
Requirement already satisfied: scipy>=0.14.0 in c:\users\hjnui\appdata\local\programs\python\python36\lib\site-packages (from scikit-optimize==0.5.2->deepaugment) (1.2.1)
Collecting scikit-learn>=0.19.1 (from scikit-optimize==0.5.2->deepaugment)
  Using cached https://files.pythonhosted.org/packages/e7/e4/56757c30853899077cf042f78fb981c0b60f7af4e7a54ca18328b80e0bff/scikit_learn-0.20.3-cp36-cp36m-win_amd64.whl
Requirement already satisfied: six>=1.9.0 in c:\users\hjnui\appdata\local\programs\python\python36\lib\site-packages (from keras==2.2.4->deepaugment) (1.12.0)
Requirement already satisfied: keras-preprocessing>=1.0.5 in c:\users\hjnui\appdata\local\programs\python\python36\lib\site-packages (from keras==2.2.4->deepaugment) (1.0.9)
Collecting pyyaml (from keras==2.2.4->deepaugment)
  Using cached https://files.pythonhosted.org/packages/72/da/386634cb17f33b12278af017ccb3314cede4806af70285f396b8f6af1203/PyYAML-5.1-cp36-cp36m-win_amd64.whl
Requirement already satisfied: h5py in c:\users\hjnui\appdata\local\programs\python\python36\lib\site-packages (from keras==2.2.4->deepaugment) (2.9.0)
Collecting pytz>=2011k (from pandas==0.23.4->deepaugment)
  Using cached https://files.pythonhosted.org/packages/61/28/1d3920e4d1d50b19bc5d24398a7cd85cc7b9a75a490570d5a30c57622d34/pytz-2018.9-py2.py3-none-any.whl
Requirement already satisfied: python-dateutil>=2.5.0 in c:\users\hjnui\appdata\local\programs\python\python36\lib\site-packages (from pandas==0.23.4->deepaugment) (2.8.0)
Requirement already satisfied: cycler>=0.10 in c:\users\hjnui\appdata\local\programs\python\python36\lib\site-packages (from matplotlib==3.0.2->deepaugment) (0.10.0)
Requirement already satisfied: kiwisolver>=1.0.1 in c:\users\hjnui\appdata\local\programs\python\python36\lib\site-packages (from matplotlib==3.0.2->deepaugment) (1.0.1)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in c:\users\hjnui\appdata\local\programs\python\python36\lib\site-packages (from matplotlib==3.0.2->deepaugment) (2.3.1)
Requirement already satisfied: termcolor>=1.1.0 in c:\users\hjnui\appdata\local\programs\python\python36\lib\site-packages (from tensorflow==1.12.0->deepaugment) (1.1.0)
Requirement already satisfied: astor>=0.6.0 in c:\users\hjnui\appdata\local\programs\python\python36\lib\site-packages (from tensorflow==1.12.0->deepaugment) (0.7.1)
Requirement already satisfied: absl-py>=0.1.6 in c:\users\hjnui\appdata\local\programs\python\python36\lib\site-packages (from tensorflow==1.12.0->deepaugment) (0.7.1)
Requirement already satisfied: protobuf>=3.6.1 in c:\users\hjnui\appdata\local\programs\python\python36\lib\site-packages (from tensorflow==1.12.0->deepaugment) (3.7.1)
Requirement already satisfied: tensorboard<1.13.0,>=1.12.0 in c:\users\hjnui\appdata\local\programs\python\python36\lib\site-packages (from tensorflow==1.12.0->deepaugment) (1.12.2)
Requirement already satisfied: gast>=0.2.0 in c:\users\hjnui\appdata\local\programs\python\python36\lib\site-packages (from tensorflow==1.12.0->deepaugment) (0.2.2)
Requirement already satisfied: grpcio>=1.8.6 in c:\users\hjnui\appdata\local\programs\python\python36\lib\site-packages (from tensorflow==1.12.0->deepaugment) (1.19.0)
Requirement already satisfied: wheel>=0.26 in c:\users\hjnui\appdata\local\programs\python\python36\lib\site-packages (from tensorflow==1.12.0->deepaugment) (0.33.1)
Requirement already satisfied: Shapely in c:\users\hjnui\appdata\local\programs\python\python36\lib\site-packages (from imgaug==0.2.7->deepaugment) (1.6.4.post1)
Requirement already satisfied: Pillow in c:\users\hjnui\appdata\local\programs\python\python36\lib\site-packages (from imgaug==0.2.7->deepaugment) (6.0.0)
Requirement already satisfied: scikit-image>=0.11.0 in c:\users\hjnui\appdata\local\programs\python\python36\lib\site-packages (from imgaug==0.2.7->deepaugment) (0.15.0)
Requirement already satisfied: imageio in c:\users\hjnui\appdata\local\programs\python\python36\lib\site-packages (from imgaug==0.2.7->deepaugment) (2.5.0)
Requirement already satisfied: werkzeug>=0.11.10 in c:\users\hjnui\appdata\local\programs\python\python36\lib\site-packages (from tensorboard<1.13.0,>=1.12.0->tensorflow==1.12.0->deepaugment) (0.15.2)
Requirement already satisfied: markdown>=2.6.8 in c:\users\hjnui\appdata\local\programs\python\python36\lib\site-packages (from tensorboard<1.13.0,>=1.12.0->tensorflow==1.12.0->deepaugment) (3.1)
Requirement already satisfied: PyWavelets>=0.4.0 in c:\users\hjnui\appdata\local\programs\python\python36\lib\site-packages (from scikit-image>=0.11.0->imgaug==0.2.7->deepaugment) (1.0.2)
Requirement already satisfied: networkx>=2.0 in c:\users\hjnui\appdata\local\programs\python\python36\lib\site-packages (from scikit-image>=0.11.0->imgaug==0.2.7->deepaugment) (2.2)
Requirement already satisfied: decorator>=4.3.0 in c:\users\hjnui\appdata\local\programs\python\python36\lib\site-packages (from networkx>=2.0->scikit-image>=0.11.0->imgaug==0.2.7->deepaugment) (4.4.0)
Installing collected packages: click, numpy, scikit-learn, scikit-optimize, keras-applications, pyyaml, keras, pytz, pandas, opencv-contrib-python, matplotlib, setuptools, imgaug, deepaugment
  Found existing installation: numpy 1.16.2
    Uninstalling numpy-1.16.2:
      Successfully uninstalled numpy-1.16.2
  Found existing installation: Keras-Applications 1.0.7
    Uninstalling Keras-Applications-1.0.7:
      Successfully uninstalled Keras-Applications-1.0.7
  Found existing installation: matplotlib 3.0.3
    Uninstalling matplotlib-3.0.3:
      Successfully uninstalled matplotlib-3.0.3
  Found existing installation: setuptools 33.1.1
    Uninstalling setuptools-33.1.1:
      Successfully uninstalled setuptools-33.1.1
  Found existing installation: imgaug 0.2.8
    Uninstalling imgaug-0.2.8:
      Successfully uninstalled imgaug-0.2.8
Successfully installed click-7.0 deepaugment-1.1.2 imgaug-0.2.7 keras-2.2.4 keras-applications-1.0.6 matplotlib-3.0.2 numpy-1.15.4 opencv-contrib-python-4.0.0.21 pandas-0.23.4 pytz-2018.9 pyyaml-5.1 scikit-learn-0.20.3 scikit-optimize-0.5.2 setuptools-40.6.3

测试DeepAugment

打开pychram 编写测试文件

from deepaugment.deepaugment import DeepAugment
deepaug = DeepAugment(images='E:\Test\_001a_0.png')
best_policies = deepaug.optimize(5)

import 语句显示正常 settings中含有deepaugment包
大功告成!