Hadoop编译

最新推荐文章于 2024-04-12 04:46:23 发布
最新推荐文章于 2024-04-12 04:46:23 发布
阅读量1.1k 收藏 2
点赞数 1
分类专栏： 大数据 文章标签： hadoop maven 大数据
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/u011057409/article/details/122151987
版权

硬件环境

环境：同方龙芯两路服务器，操作系统 64 位 loongnix
下载重新编译需要的软件包

前期准备

  • 部署JAVA环境
    loongnix默认已经安装
[root@localhost hadoop-2.6.4-src]# java -version
openjdk version "1.8.0_202"
OpenJDK Runtime Environment (Loongson 8.1.2-loongson3a-Fedora) (build 1.8.0_202-b08)
OpenJDK 64-Bit Server VM (build 25.202-b08, mixed mode)
[root@localhost hadoop-2.6.4-src]#
  • 安装maven

yum install maven

[root@localhost hadoop-2.6.4-src]# mvn -version
Apache Maven 3.2.2 (NON-CANONICAL_2015-07-07T11:23:04_root; 2015-07-07T11:23:04+08:00)
Maven home: /usr/share/maven
Java version: 1.8.0_202, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.202-1.b08.8.1.2.fc21.loongson.mips64el/jre
Default locale: zh_CN, platform encoding: UTF-8
OS name: "linux", version: "3.10.0+", arch: "mips64el", family: "unix"
[root@localhost hadoop-2.6.4-src]#
  • 安装Protocol Buffer 2.5.0
    wget https://github.com/protocolbuffers/protobuf/archive/v2.5.0.zip
    需要访问googletest才能执行autogen.sh
    这是生成configure 的包
    https://pan.baidu.com/s/1pJlZubT
./autogen.sh
bash ./configure --prefix=$INSTALL_DIR
make
make check
make install

或使用yum源码安装

yum install protobuf-devel

验证：
查看是否安装成功：protoc --version
如果出现：libprotoc 2.5.0 则说明安装成功！
添加环境变量

[root@hadoop001 ~]# vi /etc/profile
export PROTOC_HOME=/home/hadoop/app/protobuf
export PATH=$PROTOC_HOME/bin:$PATH
[root@hadoop001 ~]# source /etc/profile
[root@hadoop001 ~]# protoc  --version
libprotoc 2.5.0  （安装成功）
  • 安装其他依赖包

yum install -y openssl openssl-devel svn ncurses-devel zlib-devel libtool
yum install -y snappy snappy-devel bzip2 bzip2-devel lzo lzo-devel lzop autoconf automake

  • 安装Findbugs（可选）

yum install findbugs
[root@localhost src]# findbugs -version
3.0.0
[root@localhost src]#

下载编译

  • 下载 hadoop2.6.4 的源码包
    http://mirrors.hust.edu.cn/apache/hadoop/common/

hadoop-2.6.4-src.tar.gz

  • 压解源码包

tar -xvzf hadoop-2.6.4-src.tar.gz

  • 编译

mvn clean package -Pdist,native -DskipTests -Dtar

还可以选择以下其他命令编译

mvn package -Pdist -DskipTests -Dtar //如果不需要native code、忽略测试用例和文档，可以用下面的命令创建二进制分发版
mvn package -Pdist,native,docs -DskipTests -Dtar //创建二进制分发版，带native code和文档
mvn package -Psrc -DskipTests //创建源码分发版
mvn package -Pdist,native,docs,src -DskipTests -Dtar //创建二进制带源码分发版，带native code和文档
mvn clean site; mvn site:stage -DstagingDirectory=/tmp/hadoop-site //创建本地版web页面，放在/tmp/hadoop-site

编译生成路径
hadoop-2.6.4-src/hadoop-dist/target/hadoop-2.6.4.tar.gz

  • 编译完成结果
main:
     [exec] $ tar cf hadoop-2.6.4.tar hadoop-2.6.4
     [exec] $ gzip -f hadoop-2.6.4.tar
     [exec] 
     [exec] Hadoop dist tar available at: /home/zhubo/disk_dir/hadoop/hadoop_src/hadoop-2.6.4-src/hadoop-dist/target/hadoop-2.6.4.tar.gz
     [exec] 
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ hadoop-dist ---
[INFO] Building jar: /home/zhubo/disk_dir/hadoop/hadoop_src/hadoop-2.6.4-src/hadoop-dist/target/hadoop-dist-2.6.4-javadoc.jar
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop Common ............................... SUCCESS [06:44 min]
[INFO] Apache Hadoop NFS .................................. SUCCESS [ 22.945 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [03:02 min]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [  0.156 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [09:43 min]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [01:02 min]
[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [ 51.817 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [ 14.303 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [  0.131 s]
[INFO] hadoop-yarn ........................................ SUCCESS [  0.123 s]
[INFO] hadoop-yarn-api .................................... SUCCESS [02:01 min]
[INFO] hadoop-yarn-common ................................. SUCCESS [02:14 min]
[INFO] hadoop-yarn-server ................................. SUCCESS [  0.156 s]
[INFO] hadoop-yarn-server-common .......................... SUCCESS [01:29 min]
[INFO] hadoop-yarn-server-nodemanager ..................... SUCCESS [01:54 min]
[INFO] hadoop-yarn-server-web-proxy ....................... SUCCESS [  8.869 s]
[INFO] hadoop-yarn-server-applicationhistoryservice ....... SUCCESS [ 21.212 s]
[INFO] hadoop-yarn-server-resourcemanager ................. SUCCESS [01:10 min]
[INFO] hadoop-yarn-server-tests ........................... SUCCESS [ 17.023 s]
[INFO] hadoop-yarn-client ................................. SUCCESS [ 22.395 s]
[INFO] hadoop-yarn-applications ........................... SUCCESS [  0.123 s]
[INFO] hadoop-yarn-applications-distributedshell .......... SUCCESS [  9.894 s]
[INFO] hadoop-yarn-applications-unmanaged-am-launcher ..... SUCCESS [  6.379 s]
[INFO] hadoop-yarn-site ................................... SUCCESS [  0.237 s]
[INFO] hadoop-yarn-registry ............................... SUCCESS [ 17.471 s]
[INFO] hadoop-yarn-project ................................ SUCCESS [ 13.062 s]
[INFO] hadoop-mapreduce-client ............................ SUCCESS [  0.225 s]
[INFO] hadoop-mapreduce-client-core ....................... SUCCESS [01:11 min]
[INFO] hadoop-mapreduce-client-common ..................... SUCCESS [ 56.526 s]
[INFO] hadoop-mapreduce-client-shuffle .................... SUCCESS [ 12.969 s]
[INFO] hadoop-mapreduce-client-app ........................ SUCCESS [ 33.396 s]
[INFO] hadoop-mapreduce-client-hs ......................... SUCCESS [ 27.378 s]
[INFO] hadoop-mapreduce-client-jobclient .................. SUCCESS [01:39 min]
[INFO] hadoop-mapreduce-client-hs-plugins ................. SUCCESS [  6.331 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [ 20.835 s]
[INFO] hadoop-mapreduce ................................... SUCCESS [ 14.185 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [ 31.983 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [ 48.137 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [  8.449 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [ 19.740 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [ 15.111 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [  9.635 s]
[INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [  9.216 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [ 10.886 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [ 27.424 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [ 16.461 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [05:13 min]
[INFO] Apache Hadoop Client ............................... SUCCESS [ 24.814 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [  0.357 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [ 14.325 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 24.274 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [  0.118 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [01:32 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 49:48 min
[INFO] Finished at: 2019-10-10T12:09:00+08:00
[INFO] Final Memory: 208M/2982M
[INFO] ------------------------------------------------------------------------

集群搭建

  • 设置主机名,三台主机分别设置
    echo mini >> /etc/hostname
    echo mini1 >> /etc/hostname
    echo mini2 >> /etc/hostname

设置域名，三台主机都设置

vim /etc/hosts
10.40.25.185 mini.com mini
10.40.25.186 mini1.com mini1
10.40.25.187 mini2.com mini2
  • 设置SSH免密码登录
    mini上面执行如下
ssh-keygen
cd /root/.ssh/
cat id_rsa.pub>> authorized_keys

验证，看是否可以免密登陆
ssh root@mini
分到其他主机

ssh-copy-id -i /root/.ssh/id_rsa.pub root@mini1
ssh-copy-id -i /root/.ssh/id_rsa.pub root@mini2

将上面编译生成的hadoop-2.6.4-src/hadoop-dist/target/hadoop-2.6.4.tar.gz拷贝出来到 /home/hadoop/apps

cd /home/hadoop/apps/

解压

tar xvzf hadoop-2.6.4.tar.gz

规划安装目录 /home/hadoop/apps/hadoop-2.6.4

export HADOOP_HOME=/home/hadoop/apps/hadoop-2.6.4
mkdir -p tmp、hdfs、hdfs/data、hdfs/name (必须手动创建，用于hdfs的节点信息，数据信息等存储)
修改配置文件
cd $HADOOP_HOME/etc/hadoop/

指定JAVA_HADOOP
vim hadoop_env.sh

export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.202-1.b08.8.1.2.fc21.loongson.mips64el

ip设置成master的地址或域名，tmp路径修改成自己对应路径
vim core-site.xml

<configuration>
<!-- 指定HADOOP所使用的文件系统schema（URI），HDFS的老大（NameNode）的地址 -->
<property>
<name>fs.defaultFS</name>
<value>hdfs://mini:9000</value>
</property>
<!-- 指定hadoop运行时产生文件的存储目录 -->
<property>
<name>hadoop.tmp.dir</name>
<value>/home/hadoop/apps/hadoop-2.6.4/tmp</value>
</property>
</configuration>

设置name目录和data目录，secondary ip设置成master的地址或域名
vim hdfs-site.xml

<configuration>
<!-- 指定HDFS副本的数量 -->
<property>
<name>dfs.namenode.name.dir</name>
<value>/home/hadoop/apps/hadoop-2.6.4/data/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/home/hadoop/apps/hadoop-2.6.4/data/data</value>
</property>

<property>
<name>dfs.replication</name>
<value>3</value>
</property>

<property>
<name>dfs.secondary.http.address</name>
<value>mini:50090</value>
</property>
</configuration>

cp mapred-site.xml.template mapred-site.xml
vim mapred-site.xml

<configuration>
<!-- 指定mr运行在yarn上 -->
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>

vim yarn-site.xml

<configuration>
<!-- 指定YARN的老大（ResourceManager）的地址 -->
<property>
<name>yarn.resourcemanager.hostname</name>
<value>mini</value>
</property>
<!-- reducer获取数据的方式 -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>

vim slaves

mini
mini1
mini2

启动集群

  • 将hadoop添加到环境变量
vim setenv.sh
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.202-1.b08.8.1.2.fc21.loongson.mips64el
export HADOOP_HOME=/home/hadoop/apps/hadoop-2.6.4
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

生效

source setenv.sh

初始化HDFS,格式化

hadoop namenode -format

启动HDFS

start-dfs.sh

启动YARN

start-yarn.sh

查看启动

[root@localhost hadoop-2.6.4]# jps
12997 Jps
11542 NameNode
11834 SecondaryNameNode
12251 ResourceManager
9773 DataNode
12350 NodeManager

验证

  • 上传文件到HDFS
    从本地上传一个文本文件到hdfs的/wordcount/input目录下
hadoop fs -mkdir -p /wordcount/input
hadoop fs -put /home/hadoop/test.txt  /wordcount/input
hadoop fs -ls /wordcount/input/
  • 运行一个mapreduce程序
    在HADOOP安装目录下，运行一个示例mr程序
cd hadoop-2.6.4/share/hadoop/mapreduce/
hadoop jar hadoop-mapreduce-examples-2.6.4.jar wordcount /wordcount/input  /wordcount/output
hadoop fs -ls /wordcount/output

查看生成文件

[root@mini mapreduce]# hadoop fs -ls /wordcount/output
-rw-r--r--   3 root supergroup          0 2019-10-12 15:46 /wordcount/output/_SUCCESS
-rw-r--r--   3 root supergroup        234 2019-10-12 15:46 /wordcount/output/part-r-00000
[root@mini mapreduce]#

查看执行结果

hadoop fs -cat /wordcount/output/part-r-00000

[root@mini mapreduce]# hadoop fs -cat /wordcount/output/part-r-00000
19/10/12 16:01:05 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
d	1
dddddddddddddddddddddddddddddfffffffffffffffffffffffffffffffffweeeeeeeeeeeeeeeeeeeeeeeeetrrrrrrrrrrrrrrrrrrrrrrrrrrrhhhhhhhhhhhhhhhhhhhhhhhqqqqqqqqqqqqqqqqqlkkkkkkkkkkkkkkkkhggggg	1
dfgd	1
ew	1
g	2
r	3
sdf	1
sdff	1
sfd	1
w	1
we	2
[root@mini mapreduce]#
  • 查看HDFS状态

hdfs dfsadmin –report

[root@mini mapreduce]# hdfs  dfsadmin  -report
19/10/12 15:48:57 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Configured Capacity: 18327826432 (17.07 GB)
Present Capacity: 9458700288 (8.81 GB)
DFS Remaining: 9458647040 (8.81 GB)
DFS Used: 53248 (52 KB)
DFS Used%: 0.00%
Under replicated blocks: 2
Blocks with corrupt replicas: 0
Missing blocks: 0

-------------------------------------------------
Live datanodes (1):

Name: 10.40.25.185:50010 (mini.com)
Hostname: mini.com
Decommission Status : Normal
Configured Capacity: 18327826432 (17.07 GB)
DFS Used: 53248 (52 KB)
Non DFS Used: 8869126144 (8.26 GB)
DFS Remaining: 9458647040 (8.81 GB)
DFS Used%: 0.00%
DFS Remaining%: 51.61%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Sat Oct 12 15:49:00 CST 2019
[root@mini mapreduce]#

编译问题总结

1.错误描述： Failed to find a viable JVM installation under JAVA_HOME

【解决方法】
进入到hadoop-2.6.4-src目录下的src目录

[root@localhost src]# cd /home/zhubo/disk_dir/hadoop/hadoop_src/hadoop-2.6.4-src/hadoop-common-project/hadoop-common/src
[root@localhost src]# ls
CMakeLists.txt  config.h.cmake  contrib  JNIFlags.cmake  main  site  test
[root@localhost src]# vim JNIFlags.cmake

在文件中添加以下代码（图中红色圈出的部分）

 ELSEIF (CMAKE_SYSTEM_PROCESSOR MATCHES "mips64")
        SET(_java_libarch "mips64el")

说明：
_java_libarch设置需要根据环境实际路径进行设置，如下：

[root@localhost disk_dir]# ls /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.202-1.b08.8.1.2.fc21.loongson.mips64el/jre/lib/mips64el/
hsdis-mips64el.so  libawt_headless.so  libdt_socket.so    libinstrument.so  libjaas_unix.so             libjavafx_font.so  libjdwp.so  libjsoundalsa.so  libmlib_image.so  libprism_common.so  libsctp.so          libverify.so
jli                libawt.so           libfontmanager.so  libj2gss.so       libjava_crw_demo.so         libjavafx_iio.so   libjpeg.so  libjsound.so      libnet.so         libprism_es2.so     libsplashscreen.so  libzip.so
jvm.cfg            libawt_xawt.so      libglass.so        libj2pcsc.so      libjavafx_font_freetype.so  libjava.so         libjsdt.so  liblcms.so        libnio.so         libprism_sw.so      libsunec.so         server
libattach.so       libdecora_sse.so    libhprof.so        libj2pkcs11.so    libjavafx_font_pango.so     libjawt.so         libjsig.so  libmanagement.so  libnpt.so         libsaproc.so        libunpack.so
[root@localhost disk_dir]#

CMAKE_SYSTEM_PROCESSOR 需要根据系统环境进行设置，使用如下命令进行查看：

grep -nr CMAKE_SYSTEM_PROCESSOR

执行过程如下

[root@localhost hadoop-2.6.4-src]# pwd
/home/zhubo/disk_dir/hadoop/hadoop_src/hadoop-2.6.4-src
[root@localhost hadoop-2.6.4-src]#
[root@localhost hadoop-2.6.4-src]# grep -nr CMAKE_SYSTEM_PROCESSOR
hadoop-common-project/hadoop-common/src/JNIFlags.cmake:25:    if (CMAKE_COMPILER_IS_GNUCC AND CMAKE_SYSTEM_PROCESSOR MATCHES ".*64")
hadoop-common-project/hadoop-common/src/JNIFlags.cmake:30:    if (CMAKE_SYSTEM_PROCESSOR STREQUAL "x86_64" OR CMAKE_SYSTEM_PROCESSOR STREQUAL "amd64")
hadoop-common-project/hadoop-common/src/JNIFlags.cmake:31:        # Set CMAKE_SYSTEM_PROCESSOR to ensure that find_package(JNI) will use
hadoop-common-project/hadoop-common/src/JNIFlags.cmake:33:        set(CMAKE_SYSTEM_PROCESSOR "i686")
hadoop-common-project/hadoop-common/src/JNIFlags.cmake:38:if (CMAKE_SYSTEM_PROCESSOR MATCHES "^arm" AND CMAKE_SYSTEM_NAME STREQUAL "Linux")
hadoop-common-project/hadoop-common/src/JNIFlags.cmake:66:endif (CMAKE_SYSTEM_PROCESSOR MATCHES "^arm" AND CMAKE_SYSTEM_NAME STREQUAL "Linux")
hadoop-common-project/hadoop-common/src/JNIFlags.cmake:75:    IF(CMAKE_SYSTEM_PROCESSOR MATCHES "^i.86$")
hadoop-common-project/hadoop-common/src/JNIFlags.cmake:77:    ELSEIF (CMAKE_SYSTEM_PROCESSOR STREQUAL "x86_64" OR CMAKE_SYSTEM_PROCESSOR STREQUAL "amd64")
hadoop-common-project/hadoop-common/src/JNIFlags.cmake:79:    ELSEIF (CMAKE_SYSTEM_PROCESSOR MATCHES "^arm")
hadoop-common-project/hadoop-common/src/JNIFlags.cmake:81:    ELSEIF (CMAKE_SYSTEM_PROCESSOR MATCHES "^(powerpc|ppc)64le")
hadoop-common-project/hadoop-common/src/JNIFlags.cmake:88:        SET(_java_libarch ${CMAKE_SYSTEM_PROCESSOR})
hadoop-common-project/hadoop-common/target/native/CMakeFiles/3.9.0/CMakeSystem.cmake:11:set(CMAKE_SYSTEM_PROCESSOR "mips64")

如下

IF("${CMAKE_SYSTEM}" MATCHES "Linux")
    #
    # Locate JNI_INCLUDE_DIRS and JNI_LIBRARIES.
    # Since we were invoked from Maven, we know that the JAVA_HOME environment
    # variable is valid.  So we ignore system paths here and just use JAVA_HOME.
    #
    FILE(TO_CMAKE_PATH "$ENV{JAVA_HOME}" _JAVA_HOME)
    IF(CMAKE_SYSTEM_PROCESSOR MATCHES "^i.86$")
        SET(_java_libarch "i386")
    ELSEIF (CMAKE_SYSTEM_PROCESSOR STREQUAL "x86_64" OR CMAKE_SYSTEM_PROCESSOR STREQUAL "amd64")
        SET(_java_libarch "amd64")
    ELSEIF (CMAKE_SYSTEM_PROCESSOR MATCHES "mips64")
        SET(_java_libarch "mips64el")
    ELSEIF (CMAKE_SYSTEM_PROCESSOR MATCHES "^arm")
        SET(_java_libarch "arm")
    ELSEIF (CMAKE_SYSTEM_PROCESSOR MATCHES "^(powerpc|ppc)64le")
        IF(EXISTS "${_JAVA_HOME}/jre/lib/ppc64le")
                SET(_java_libarch "ppc64le")
        ELSE()
                SET(_java_libarch "ppc64")
        ENDIF()
    ELSE()
        SET(_java_libarch ${CMAKE_SYSTEM_PROCESSOR})
    ENDIF()
    SET(_JDK_DIRS "${_JAVA_HOME}/jre/lib/${_java_libarch}/*"
                  "${_JAVA_HOME}/jre/lib/${_java_libarch}"
                  "${_JAVA_HOME}/jre/lib/*"
                  "${_JAVA_HOME}/jre/lib"
                  "${_JAVA_HOME}/lib/*"
                  "${_JAVA_HOME}/lib"
                  "${_JAVA_HOME}/include/*"
                  "${_JAVA_HOME}/include"
                  "${_JAVA_HOME}"

....

2、问题
[14:21:34@root hadoop_src]#protoc --version
protoc: error while loading shared libraries: libprotoc.so.8: cannot open shared object file: No such file or directory

解决办法

sudo ldconfig

3、问题

FailfindbugsXml.xml does not exist

是由于中途出错，再进行编译导致的错误，解决需要将出错的模块删除
解决：

mvn package -Pdist,native -DskipTests -Dtar -rf :hadoop-common

4
OpenJDK 64-Bit Server VM warning: You have loaded library /home/hadoop/apps/hadoop-2.6.4/lib/native/libhadoop.so which might have disabled stack guard. The VM will try to fix the stack guard now.
It’s highly recommended that you fix the library with 'execstack -c ', or link it with ‘-z noexecstack’.

5
WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable

6
Failed to execute goal org.apache.maven.plugins:maven-javadoc-plugin:2.8.1:j

解决
使用如下命令进行编译

mvn clean install -DskipTests -Dmaven.javadoc.skip=true

90geek
关注
专栏目录
CentOS 64位上编译 Hadoop 2.6.0
(Andrew)的专栏
03-19 8346
1.操作系统编译环境 yum install cmake lzo-devel zlib-devel gcc gcc-c++ autoconf automake libtool ncurses-devel openssl-devel libXtst 2.安装JDK 下载JDK1.7，注意只能用1.7，否则编译会出错 http://www.oracle.com/tech
Hadoop源码编译
weixin_40173707的博客
05-22 1555
准备前期工作 1、CentOS联网 使用能连接外网的CentOS系统虚拟机，使用root用户进行编译，避免出现文件夹权限问题。 2、Jar包准备 从官网下载以下安装包 （1）hadoop-3.1.3-src.tar.gz （2）jdk-8u212-linux-x64.tar.gz （3）apache-maven-3.6.3-bin.tar.gz （4）protobuf-2.5.0.tar.gz（序列化的框架） （5）cmake-3.13.1.tar.gz Jar包安装 1、上传软件包到指定的目录,例如 /o
Windows7-64编译hadoop-3.2.0
MoodStreet的博客
08-09 2614
以下的这一切操作都来源为了解决这个问题。。。。 1 文件准备 打开Hadoop src下的BUILDING.txt（多看看该文件），找到Building on Windows Jdk1.8  hadoop-3.2.0-src.tar https://hadoop.apache.org/release/3.2.0.html  apache-maven-3.6.1 http://maven.a...
hadoop操作命令报native错误
xichengxml的博客
01-27 263
utiWARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable 在配置文件hadoop-env.sh中最下面加以下两行即可 export HADOOP_HOME_WARN_SUPPRESS=1 expo...
hadoop 3.3.4 编译报错记录
等夏天
04-04 1345
在docker内编译，可能是centos的原因或者docker版本的原因，我用centos7.9安装docker23.x版本就可以完成编译。可能是protobuf安装的版本有问题，我是直接复制的别人编译hadoop3.1.x版本的需要yum的一些依赖包，但是观察hadoop3.3.4源码包中。缺少googletest的依赖，有梯子应该就直接从google下载了，但是这里没有梯子就手动从github下载。hadoop 3.3.4 注意。make check 通过。
HADOOP编译
lx940112的博客
05-20 1003
1.编译Hadoop的环境要求（摘抄自GitHub上Apache/Hadoop的BUILDING.txt）：* Unix系统* JDK 1.8* Maven 3.3或更高版本* ProtocolBuffer 2.5.0* CMake 3.1或更新（如果编译本机代码）* Zlib devel（如果编译本地代码）* Cyrus SASL devel（如果编译本机代码）* One of the comp...
hadoop的编译过程
浅谈
06-08 668
Hadoop的编译包括JDK的安装部署、maven的安装部署、findbugs的安装、Protobuff的安装、snappy的安装等等
1. Hadoop源代码编译
cl2010abc的专栏
05-19 732
安装
hadoop编译
01-01
### Hadoop编译知识点 #### 一、环境搭建与配置 **环境介绍：** - **虚拟机:** 使用VirtualBox作为虚拟机平台。 - **操作系统:** 64位 CentOS 6.4。 #### 二、软件准备与安装 **1. Apache Ant** - **目的:** ...
hadoop编译文档
01-25
### Hadoop编译指南 #### 一、概述 Hadoop是一款能够处理大数据的开源软件框架，主要用于分布式存储和处理大规模数据集。为了更好地理解和掌握Hadoop的内部机制，很多开发者会选择从源代码编译Hadoop。本文档将...
Hadoop2.7.3编译
08-01
Hadoop2.7.3编译,包括所有工具包,照着做就行,亲测可行
hadoop源码编译
12-15
WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 需要进行hadoop源码的编译
hadoop编译后的包
10-04
这个“hadoop编译后的包”特别之处在于它支持Snappy压缩，这是一个由Google开发的高效数据压缩库，通常用于提高数据存储和传输效率。 在Hadoop中，数据是以Block的形式存储在HDFS（Hadoop Distributed File System...
hadoop编译.zip
06-16
apache-ant-1.9.9-bin.tar.gz apache-maven-3.0.5-bin.tar.gz hadoop-2.7.2-src.tar.gz protobuf-2.5.0.tar.gz
hadoop3.3.0在centos7.7环境编译环境资源，附hadoop编译流程详细说明
04-30
hadoop编译流程详细说明 apache-ant-1.9.16-bin.tar.gz cmake-3.22.4.tar.gz protobuf-2.5.0.tar.gz snappy-1.1.4.tar.gz cyrus-sasl-2.1.26-23.el7.x86_64.rpm cyrus-sasl-devel-2.1.26-23.el7.x86_64.rpm cyrus-...
编译hadoop源码
安静的程序猿
09-21 1651
一、首先下载hadoop源码包 下载的方式还是比较多的 1.svn的方式 svn下载首先需要安装SVN，如果想尝试使用svn，可以参考源码管理、获取网络源码工具：TortoiseSVN使用手册，hadoop的svn下载地址点此 2.压缩包的方式 这里我们介绍一种压缩包的方式。 首先我们登录官网： http://hadoop.apache.org/，我们找到下面链接，点进去下载即可。如
【Hadoop】Hadoop 编译源码
最新发布
2401_84183033的博客
04-12 981
【代码】【Hadoop】Hadoop 编译源码。
搭建hadoop编译环境
04-10
搭建Hadoop编译环境的步骤如下[^1]： 1. 在Hadoop安装目录下，即`/opt/module/hadoop`路径下，修改以下四个配置文件：`core-site.xml`、`hdfs-site.xml`、`yarn-site.xml`、`mapred-site.xml`。根据你的需求进行...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值