hadoop的基础设施-protobuf-2.5.0编译和安装

本文链接：https://blog.csdn.net/zhuxuemin1991/article/details/125341850

引言

在安装编译Hadoop的时候需要提前安装protobuf-2.5.0的，而且版本不要高了或者低了，这个是因为hadoop的rpc通信里面是直接使用protobuf-2.5.0作为数据格式交换的，因为遇到很多次，每次都从头安装，这次记录下来。

编译时候的提示

我的编译版本源码是hadoop-2.7.3,如果环境中没有安装protobuf-2.5.0,在做源码编译的时候有如下提示：

[INFO] 
[INFO] --- hadoop-maven-plugins:2.7.3:protoc (compile-protoc) @ hadoop-common ---
[WARNING] [protoc, --version] failed: java.io.IOException: Cannot run program "protoc": error=2, 没有那个文件或目录
[ERROR] stdout: []

然后在我们的编译进程中是会中断的，具体表现如下：

[INFO] Executed tasks
[INFO] 
[INFO] --- hadoop-maven-plugins:2.7.3:protoc (compile-protoc) @ hadoop-common ---
[WARNING] [protoc, --version] failed: java.io.IOException: Cannot run program "protoc": error=2, 没有那个文件或目录
[ERROR] stdout: []
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop Main 2.7.3 ........................... SUCCESS [ 42.510 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 47.049 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [ 24.040 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [01:22 min]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 12.138 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 19.555 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [01:28 min]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [11:00 min]
[INFO] Apache Hadoop Auth ................................. SUCCESS [03:57 min]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 14.794 s]
[INFO] Apache Hadoop Common ............................... FAILURE [03:47 min]
[INFO] Apache Hadoop NFS .................................. SKIPPED
[INFO] Apache Hadoop KMS .................................. SKIPPED
[INFO] Apache Hadoop Common Project ....................... SUCCESS [  6.255 s]
......

当然，作为一名大数据开发工程师，除了知道有这回事，还要知道怎么来的。具体其实是在hadoop-common模块里面的pom.xml有定义,具体是在hadoop-2.7.3-src/hadoop-common-project/hadoop-common/pom.xml中，314-191行有对protoc的配置：

    <plugins>
      <plugin>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-maven-plugins</artifactId>
        <executions>
          <execution>
            <id>version-info</id>
            <phase>generate-resources</phase>
            <goals>
              <goal>version-info</goal>
            </goals>
            <configuration>
              <source>
                <directory>${basedir}/src/main</directory>
                <includes>
                  <include>java/**/*.java</include>
                  <include>proto/**/*.proto</include>
                </includes>
              </source>
            </configuration>
          </execution>
          <execution>
            <id>compile-protoc</id>
            <phase>generate-sources</phase>
            <goals>
              <goal>protoc</goal>
            </goals>
            <configuration>
              <protocVersion>${protobuf.version}</protocVersion>
              <protocCommand>${protoc.path}</protocCommand>
              <imports>
                <param>${basedir}/src/main/proto</param>
              </imports>
              <source>
                <directory>${basedir}/src/main/proto</directory>
                <includes>
                  <include>HAServiceProtocol.proto</include>
                  <include>IpcConnectionContext.proto</include>
                  <include>ProtocolInfo.proto</include>
                  <include>RpcHeader.proto</include>
                  <include>ZKFCProtocol.proto</include>
                  <include>ProtobufRpcEngine.proto</include>
                  <include>Security.proto</include>
                  <include>GetUserMappingsProtocol.proto</include>
                  <include>TraceAdmin.proto</include>
                  <include>RefreshAuthorizationPolicyProtocol.proto</include>
                  <include>RefreshUserMappingsProtocol.proto</include>
                  <include>RefreshCallQueueProtocol.proto</include>
                  <include>GenericRefreshProtocol.proto</include>
                </includes>
              </source>
              <output>${project.build.directory}/generated-sources/java</output>
            </configuration>
          </execution>
          <execution>
            <id>compile-test-protoc</id>
            <phase>generate-test-sources</phase>
            <goals>
              <goal>protoc</goal>
            </goals>
            <configuration>
              <protocVersion>${protobuf.version}</protocVersion>
              <protocCommand>${protoc.path}</protocCommand>
              <imports>
                <param>${basedir}/src/test/proto</param>
              </imports>
              <source>
                <directory>${basedir}/src/test/proto</directory>
                <includes>
                  <include>test.proto</include>
                  <include>test_rpc_service.proto</include>
                </includes>
              </source>
              <output>${project.build.directory}/generated-test-sources/java</output>
            </configuration>
          </execution>
        </executions>
      </plugin>

这一步的操作就是在maven编译的时候需要使用环境中的proto编译源码里面的proto文件，然后会生成rpc通信的源码。

proto编译

源码包是protobuf-2.5.0.tar.gz，需要翻墙下载，我用公司的网络是可以直接上谷歌的，不过百度一下也很多可以下载，我自己本机也有，如果实在不知道怎么找的话我单发也行

wget https://protobuf.googlecode.com/files/protobuf-2.5.0.tar.gz

源码上传到linux解压，编译

tar xvf protobuf-2.5.0.tar.gz
cd protobuf-2.5.0
./configure --prefix=/apps/svr/protobuf/
make
make install

验证

我把命令配置到我的环境变量中

export PATH=/apps/svr/protobuf/:$PATH

验证

[root@zhu-91-134 yum.repos.d]#  protoc --version
libprotoc 2.5.0

单独验证的话不需要所有都重头再来，可以单独验证common模块

cd hadoop-common-project/hadoop-common && mvn package -Pdist -DskipTests -Dtar

信息比较多，但是看到关键的成功就行

......
[INFO] Building jar: /root/hadoop-2.7.3-src/hadoop-common-project/hadoop-common/target/hadoop-common-2.7.3-javadoc.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:33 min
[INFO] Finished at: 2022-05-03T23:18:15+08:0

错误记录

error: cannot install `libprotoc.la' to a directory not ending in /usr/local/lib

这个错误其实是在我前面一次有配置过，需要执行一下make clean清理掉之前的配置