===
提示：文章写完后，目录可以自动生成，如何生成可参考右边的帮助文档
目标
目前正在学习hadoop及flink，想尝试用flink处理图片，但是能用的资料太少，于是打算把hadoop处理图片吃透再去探索flink处理图片，现在有个要求，要在hadoop上将mapreduce与opencv结合达到hadoop+opencv处理图片，网上相关资料很少加上自己很菜，所以走了很多弯路
提示：所用的系统Ubuntu16.04，hadoop集群安装了hadoop版本是3.2.1，jdk是1.8.0_271
一、hadoop如何在Linux环境下使用opencv4.5.1？
与windows不同，在Linux环境下使用opencv，尤其是编写mapreduce代码中使用opencv，需要将opencv的源码编译，安装到Linux上，然后将生成的libopencv_java451.so文件导入到$HADOOP_HOME/lib/native下
二、安装opencv4.5.1
1.安装所需依赖
代码如下：（当然，肯定是在hadoop已经搭建完成的情况下，所以安装jdk的部分不再赘述，我安装的是1.8版本）：
//下载apache ant1.9.15，这个是官网提到的需要安装的，很多安装opencv3.x的教程都没提到apache ant，我编译opencv4.5.1、安装后，发现没有生成libopencv_xxx.so。直到安装了apache ant以后才成功
iwget https://mirrors.tuna.tsinghua.edu.cn/apache//ant/binaries/apache-ant-1.9.15-bin.zip
//解压所得的apache-ant-1.9.15-bin.zip
unzip ~/apache-ant-1.9.7-bin.zip
//设置ANT_HOME，并添加$ANT_HOME/bin到PATH
export ANT_HOME=/home/ubuntu/Hadoop/opencv/ApacheAnt
export PATH=$PATH:$ANT_HOME/bin
//下载opencv4.5.1的源码，通过github或者官网下载都行，官网安装方式是推荐在github上下载，我是通过官网下载的，都行。
wget -O opencv.zip https://github.com/opencv/opencv/archive/master.zip
//解压opencv.zip，得到opencv.4.5.1，并在里面新建一个build文件夹并进入，待会的编译都要在build文件夹下
unzip opencv.zip
mkdir -p build && cd build
//安装编译所需要的依赖
sudo apt-get install cmake
sudo apt-get install build-essential libgtk2.0-dev libavcodec-dev libavformat-dev libjpeg.dev libtiff4.dev libswscale-dev libjasper-dev
//安装好依赖以后进行编译，这里是将opencv安装在build/install文件夹中，不在系统目录里。如果需要安装在/usr/local中，则需要把cmake那句中的./install 更改成为/usr/local即可。
cmake -D CMAKE_BUILD_TYPE=Release -D CMAKE_INSTALL_PREFIX=./install ..
//稍等,出现以下即为成功：
-- Configuring done
-- Generating done
-- Build files have been written to: 此处省略/opencv4.2.0/build
//往上翻，查看有关java的编译信息,看到两个YES就是ok的，我没有遇到过NO，如果出现NO应该是要检测jdk
-- Java:
-- ant: /usr/bin/ant (ver 1.10.5)
-- JNI: /usr/lib/jvm/java-8-openjdk-amd64/include /usr/lib/jvm/java-8-openjdk-amd64/include/linux /usr/lib/jvm/java-8-openjdk-amd64/include
-- Java wrappers: YES
-- Java tests: YES
//进行编译，make -j8，其中的8代表编译的并行度，看自己Ubuntu系统的cpu是几核的，我的是老机子，就设置了-j2，比较慢。
make -j2
//等待100%
[ 99%] Linking CXX executable ../../bin/opencv_test_video
[100%] Linking CXX executable ../../bin/opencv_perf_video
[100%] Linking CXX executable ../../bin/opencv_test_gapi
[100%] Built target opencv_test_video [100%] Built target
opencv_perf_video [100%] Built target opencv_test_gapi
//编译成功后，即可看到build出现了很多文件夹，然后进行安装，由于编译的时候设置了安装在build/install的信息，所以执行按照后，build下会多出一个install的文件夹
sudo make install
进入build/install就可以看到安装好的opencv4.5.1
我们所要的libopencv_java451.so文件就在
build/install/share/java/opencv4下（同时还可以看到opencv-451.jar）
接下来就是怎么与hadoop联系起来，在这里我是把整个opencv的文件夹导入到各个节点上，我没有尝试只拷贝libopencv_java451.so，哎，不懂。
然后将libopencv_java451.so，通过scp到各个节点，在这里我的hadoop集群安装的是hadoop3.2.1版本
最后，在各个节点的$HADOOP_HOME/lib/native下要保证都有libopencv_java451.so
在这里可以查看一下libopencv_java451.so文件是否可用
root@hadoop4:/home/ubuntu/Hadoop/hadoop-3.2.1/lib/native# ldd libopencv_java451.so
linux-vdso.so.1 => (0x00007ffe784fd000)
libopencv_ml.so.4.5 => /home/ubuntu/Hadoop/opencv/opencv-4.5.1/build/install/lib/libopencv_ml.so.4.5 (0x00007fc4df6c8000)
libopencv_photo.so.4.5 => /home/ubuntu/Hadoop/opencv/opencv-4.5.1/build/install/lib/libopencv_photo.so.4.5 (0x00007fc4df41f000)
libopencv_objdetect.so.4.5 => /home/ubuntu/Hadoop/opencv/opencv-4.5.1/build/install/lib/libopencv_objdetect.so.4.5 (0x00007fc4df1a6000)
libopencv_video.so.4.5 => /home/ubuntu/Hadoop/opencv/opencv-4.5.1/build/install/lib/libopencv_video.so.4.5 (0x00007fc4def2a000)
libopencv_videoio.so.4.5 => /home/ubuntu/Hadoop/opencv/opencv-4.5.1/build/install/lib/libopencv_videoio.so.4.5 (0x00007fc4decc2000)
libopencv_imgcodecs.so.4.5 => /home/ubuntu/Hadoop/opencv/opencv-4.5.1/build/install/lib/libopencv_imgcodecs.so.4.5 (0x00007fc4de7b4000)
libopencv_dnn.so.4.5 => /home/ubuntu/Hadoop/opencv/opencv-4.5.1/build/install/lib/libopencv_dnn.so.4.5 (0x00007fc4de168000)
libopencv_calib3d.so.4.5 => /home/ubuntu/Hadoop/opencv/opencv-4.5.1/build/install/lib/libopencv_calib3d.so.4.5 (0x00007fc4ddd7b000)
libopencv_features2d.so.4.5 => /home/ubuntu/Hadoop/opencv/opencv-4.5.1/build/install/lib/libopencv_features2d.so.4.5 (0x00007fc4ddaa9000)
libopencv_flann.so.4.5 => /home/ubuntu/Hadoop/opencv/opencv-4.5.1/build/install/lib/libopencv_flann.so.4.5 (0x00007fc4dd848000)
libopencv_imgproc.so.4.5 => /home/ubuntu/Hadoop/opencv/opencv-4.5.1/build/install/lib/libopencv_imgproc.so.4.5 (0x00007fc4dcfb9000)
libopencv_core.so.4.5 => /home/ubuntu/Hadoop/opencv/opencv-4.5.1/build/install/lib/libopencv_core.so.4.5 (0x00007fc4dc954000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fc4dc5d2000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fc4dc2c9000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fc4dc0b3000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc4dbce9000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fc4dbacc000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fc4db8c8000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fc4db6c0000)
libavcodec-ffmpeg.so.56 => /usr/lib/x86_64-linux-gnu/libavcodec-ffmpeg.so.56 (0x00007fc4da28c000)
libavformat-ffmpeg.so.56 => /usr/lib/x86_64-linux-gnu/libavformat-ffmpeg.so.56 (0x00007fc4d9e8d000)
libavutil-ffmpeg.so.54 => /usr/lib/x86_64-linux-gnu/libavutil-ffmpeg.so.54 (0x00007fc4d9c1e000)
libswscale-ffmpeg.so.3 => /usr/lib/x86_64-linux-gnu/libswscale-ffmpeg.so.3 (0x00007fc4d998f000)
libjpeg.so.8 => /usr/lib/x86_64-linux-gnu/libjpeg.so.8 (0x00007fc4d9736000)
libpng12.so.0 => /lib/x86_64-linux-gnu/libpng12.so.0 (0x00007fc4d9511000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fc4d92f7000)
/lib64/ld-linux-x86-64.so.2 (0x00007fc4dfca7000)
libswresample-ffmpeg.so.1 => /usr/lib/x86_64-linux-gnu/libswresample-ffmpeg.so.1 (0x00007fc4d90da000)
libva.so.1 => /usr/lib/x86_64-linux-gnu/libva.so.1 (0x00007fc4d8ebe000)
libzvbi.so.0 => /usr/lib/x86_64-linux-gnu/libzvbi.so.0 (0x00007fc4d8c33000)
libxvidcore.so.4 => /usr/lib/x86_64-linux-gnu/libxvidcore.so.4 (0x00007fc4d891f000)
libx265.so.79 => /usr/lib/x86_64-linux-gnu/libx265.so.79 (0x00007fc4d7d00000)
libx264.so.148 => /usr/lib/x86_64-linux-gnu/libx264.so.148 (0x00007fc4d795c000)
libwebp.so.5 => /usr/lib/x86_64-linux-gnu/libwebp.so.5 (0x00007fc4d7700000)
libwavpack.so.1 => /usr/lib/x86_64-linux-gnu/libwavpack.so.1 (0x00007fc4d74d7000)
libvpx.so.3 => /usr/lib/x86_64-linux-gnu/libvpx.so.3 (0x00007fc4d70b3000)
libvorbisenc.so.2 => /usr/lib/x86_64-linux-gnu/libvorbisenc.so.2 (0x00007fc4d6e0a000)
libvorbis.so.0 => /usr/lib/x86_64-linux-gnu/libvorbis.so.0 (0x00007fc4d6bdf000)
libtwolame.so.0 => /usr/lib/x86_64-linux-gnu/libtwolame.so.0 (0x00007fc4d69bc000)
libtheoraenc.so.1 => /usr/lib/x86_64-linux-gnu/libtheoraenc.so.1 (0x00007fc4d677d000)
libtheoradec.so.1 => /usr/lib/x86_64-linux-gnu/libtheoradec.so.1 (0x00007fc4d6563000)
libspeex.so.1 => /usr/lib/x86_64-linux-gnu/libspeex.so.1 (0x00007fc4d634a000)
libsnappy.so.1 => /usr/lib/x86_64-linux-gnu/libsnappy.so.1 (0x00007fc4d6142000)
libshine.so.3 => /usr/lib/x86_64-linux-gnu/libshine.so.3 (0x00007fc4d5f35000)
libschroedinger-1.0.so.0 => /usr/lib/x86_64-linux-gnu/libschroedinger-1.0.so.0 (0x00007fc4d5c60000)
libopus.so.0 => /usr/lib/x86_64-linux-gnu/libopus.so.0 (0x00007fc4d5a16000)
libopenjpeg.so.5 => /usr/lib/x86_64-linux-gnu/libopenjpeg.so.5 (0x00007fc4d57f3000)
libmp3lame.so.0 => /usr/lib/x86_64-linux-gnu/libmp3lame.so.0 (0x00007fc4d557e000)
libgsm.so.1 => /usr/lib/x86_64-linux-gnu/libgsm.so.1 (0x00007fc4d5370000)
libcrystalhd.so.3 => /usr/lib/x86_64-linux-gnu/libcrystalhd.so.3 (0x00007fc4d5155000)
liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007fc4d4f33000)
libssh-gcrypt.so.4 => /usr/lib/x86_64-linux-gnu/libssh-gcrypt.so.4 (0x00007fc4d4ce9000)
librtmp.so.1 => /usr/lib/x86_64-linux-gnu/librtmp.so.1 (0x00007fc4d4acd000)
libmodplug.so.1 => /usr/lib/x86_64-linux-gnu/libmodplug.so.1 (0x00007fc4d4742000)
libgme.so.0 => /usr/lib/x86_64-linux-gnu/libgme.so.0 (0x00007fc4d44f4000)
libbluray.so.1 => /usr/lib/x86_64-linux-gnu/libbluray.so.1 (0x00007fc4d42ab000)
libgnutls.so.30 => /usr/lib/x86_64-linux-gnu/libgnutls.so.30 (0x00007fc4d3f7b000)
libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007fc4d3d6b000)
libsoxr.so.0 => /usr/lib/x86_64-linux-gnu/libsoxr.so.0 (0x00007fc4d3b06000)
libnuma.so.1 => /usr/lib/x86_64-linux-gnu/libnuma.so.1 (0x00007fc4d38fb000)
libogg.so.0 => /usr/lib/x86_64-linux-gnu/libogg.so.0 (0x00007fc4d36f2000)
liborc-0.4.so.0 => /usr/lib/x86_64-linux-gnu/liborc-0.4.so.0 (0x00007fc4d3472000)
libgcrypt.so.20 => /lib/x86_64-linux-gnu/libgcrypt.so.20 (0x00007fc4d3191000)
libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007fc4d2f47000)
libhogweed.so.4 => /usr/lib/x86_64-linux-gnu/libhogweed.so.4 (0x00007fc4d2d14000)
libnettle.so.6 => /usr/lib/x86_64-linux-gnu/libnettle.so.6 (0x00007fc4d2ade000)
libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007fc4d285e000)
libxml2.so.2 => /usr/lib/x86_64-linux-gnu/libxml2.so.2 (0x00007fc4d24a3000)
libfontconfig.so.1 => /usr/lib/x86_64-linux-gnu/libfontconfig.so.1 (0x00007fc4d2260000)
libfreetype.so.6 => /usr/lib/x86_64-linux-gnu/libfreetype.so.6 (0x00007fc4d1fb6000)
libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007fc4d1d52000)
libidn.so.11 => /usr/lib/x86_64-linux-gnu/libidn.so.11 (0x00007fc4d1b1f000)
libtasn1.so.6 => /usr/lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007fc4d190c000)
libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00007fc4d16ea000)
libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007fc4d14d6000)
libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007fc4d1204000)
libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007fc4d0fd5000)
libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007fc4d0dd1000)
libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007fc4d0bc6000)
libicuuc.so.55 => /usr/lib/x86_64-linux-gnu/libicuuc.so.55 (0x00007fc4d0832000)
libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007fc4d0609000)
libffi.so.6 => /usr/lib/x86_64-linux-gnu/libffi.so.6 (0x00007fc4d0401000)
libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007fc4d01fd000)
libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007fc4cffe2000)
libicudata.so.55 => /usr/lib/x86_64-linux-gnu/libicudata.so.55 (0x00007fc4ce52b000)
没有出现not found的情况就是ok的，至于为什么要这么做，可以查看一下.so文件的作用，我不太懂。
将所有节点都拷贝了libopencv_java451.so以后就可以编写mapreduce的代码了，
2.编写mapreduce代码
在mapreduce的编写过程中，三个自定义类（Runner类，Mapper类，Reduce类）在一开始就要添加一句：
//以下语句十分重要，若是不填，则在Linux下会Error: org.opencv.core.Mat.n_Mat(III)J
System.loadLibrary("opencv_java451");
如果reduce类不添加System.loadLibrary(“opencv_java451”);会报错
即：
在windows下还是要安装opencv（这部分内容可以寻找其他资料，很多，包括引入jar包，导入xxx.dll，比在Linux环境下部署要简单的多），然后导入到IntelliJ IDEA 里才能顺利编写代码、如果不出意外的话，mapreduce代码编写完毕，打成jar包，传到集群上运行后即可。
在这里我是利用opencv将hdfs上的图片灰度化，并存在hdfs上
总结
很少有人用hadoop结合opencv去处理图片，以至于找不到什么hadoop+opencv的部署方法，我也是尝试了好久才将mapreduce与opencv联系起来，总而言之还是自己太菜，对很多东西还不理解，很多东西还没熟悉，以上说的很多东西如果有出错的地方，等我遇到问题解决后再改。
感谢几位大佬的参考链接：
1、ubuntu 编译opencv4.2 for java
2、centos 安装 opencv-3.1.0
3、以及官网链接：
1951
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
