在大数据数仓实践过程中,hadoop技术栈充当了非常重要的环节,很多大数据平台,中台都是基于hive+hadoop+spark来搭建的,所以我们对于hadoop的编译,构建,安装是需要掌握的非常清楚。
而我们大数据的研发人员比较常用mac电脑来开发,mac 最新的 m1/m2 arm版本的系统在编译hive/hadoop的过程中会遇到protobuf编译失败的问题。
下面就是在编译过程中,遇到的问题,希望能给大家带来清晰的指导
在arm上编译protoc,出现异常
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.xolstice.maven.plugins:protobuf-maven-plugin:0.5.1:compile (default-cli) on project hadoop-hdfs: Missing:
[ERROR] ----------
[ERROR] 1) com.google.protobuf:protoc:exe:osx-aarch_64:3.7.1
[ERROR]
[ERROR] Try downloading the file manually from the project website.
[ERROR]
[ERROR] Then, install it using the command:
[ERROR] mvn install:install-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=3.7.1 -Dclassifier=osx-aarch_64 -Dpackaging=exe -Dfile=/path/to/file
[ERROR]
[ERROR] Alternatively, if you host your own repository you can deploy the file there:
[ERROR] mvn deploy:deploy-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=3.7.1 -Dclassifier=osx-aarch_64 -Dpackaging=exe -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]
[ERROR]
[ERROR] Path to dependency:
[ERROR] 1) org.apache.hadoop:hadoop-hdfs:jar:3.3.1
[ERROR] 2) com.google.protobuf:protoc:exe:osx-aarch_64:3.7.1
[ERROR]
[ERROR] ----------
[ERROR] 1 required artifact is missing.
[ERROR]
[ERROR] for artifact:
[ERROR] org.apache.hadoop:hadoop-hdfs:jar:3.3.1
[ERROR]
[ERROR] from the specified remote repositories:
[ERROR] aliyunmaven (https://maven.aliyun.com/repository/spring-plugin, releases=true, snapshots=true),
[ERROR] alimaven (http://maven.aliyun.com/nexus/content/groups/public/, releases=true, snapshots=false)
[ERROR]
于是去官方的issue上,找到了相关的问题,目前没有兼容m1芯片的版本,需要手动指定x86_64的版本
<plugin>
<groupId>org.xolstice.maven.plugins</groupId>
<artifactId>protobuf-maven-plugin</artifactId>
<version>${protobuf-maven-plugin.version}</version>
<extensions>true</extensions>
<configuration>
<protocArtifact>
com.google.protobuf:protoc:${hadoop.protobuf.version}:exe:${os.detected.classifier}
</protocArtifact>
<attachProtoSources>false</attachProtoSources>
</configuration>
<executions>
<execution>
<id>src-compile-protoc</id>
<phase>generate-sources</phase>
<goals>
<goal>compile</goal>
</goals>
<configuration>
<includeDependenciesInDescriptorSet>false</includeDependenciesInDescriptorSet>
<protoSourceRoot>${basedir}/src/main/proto</protoSourceRoot>
<outputDirectory>${project.build.directory}/generated-sources/java</outputDirectory>
<clearOutputDirectory>false</clearOutputDirectory>
<skip>true</skip>
</configuration>
</execution>
<execution>
<id>src-test-compile-protoc</id>
<phase>generate-test-sources</phase>
<goals>
<goal>test-compile</goal>
</goals>
<configuration>
<protoTestSourceRoot>${basedir}/src/test/proto</protoTestSourceRoot>
<outputDirectory>${project.build.directory}/generated-test-sources/java</outputDirectory>
<clearOutputDirectory>false</clearOutputDirectory>
<skip>true</skip>
</configuration>
</execution>
</executions>
</plugin>
修改插件的 ${os.detected.classifier} 值 为 osx-x86_64
<plugin>
<groupId>org.xolstice.maven.plugins</groupId>
<artifactId>protobuf-maven-plugin</artifactId>
<version>${protobuf-maven-plugin.version}</version>
<extensions>true</extensions>
<configuration>
<protocArtifact>
com.google.protobuf:protoc:${hadoop.protobuf.version}:exe:osx-x86_64
</protocArtifact>
<attachProtoSources>false</attachProtoSources>
</configuration>
<executions>
<execution>
<id>src-compile-protoc</id>
<phase>generate-sources</phase>
<goals>
<goal>compile</goal>
</goals>
<configuration>
<includeDependenciesInDescriptorSet>false</includeDependenciesInDescriptorSet>
<protoSourceRoot>${basedir}/src/main/proto</protoSourceRoot>
<outputDirectory>${project.build.directory}/generated-sources/java</outputDirectory>
<clearOutputDirectory>false</clearOutputDirectory>
<skip>true</skip>
</configuration>
</execution>
<execution>
<id>src-test-compile-protoc</id>
<phase>generate-test-sources</phase>
<goals>
<goal>test-compile</goal>
</goals>
<configuration>
<protoTestSourceRoot>${basedir}/src/test/proto</protoTestSourceRoot>
<outputDirectory>${project.build.directory}/generated-test-sources/java</outputDirectory>
<clearOutputDirectory>false</clearOutputDirectory>
<skip>true</skip>
</configuration>
</execution>
</executions>
</plugin>
希望对正在查看文章的您有所帮助,记得关注、评论、收藏,谢谢您