大数据平台架构实战（二）IntelliJ IDEA搭建hadoop

IntelliJ IDEA 构建工程

工具类开发，随便找，IntelliJ IDEA搭建Hadoop开发环境（下）_Captain.Y.的博客-CSDN博客

生成jar包

由于我们的开发环境和hadoop运行环境都在同一台机器，直接用maven工具就可以了

如果是其他情形 ，尽量保证jar包的依赖完整性。

pom文件

 <dependencies>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>2.10.2</version>
        </dependency>


        <!--单元测试依赖-->
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.12</version>

        </dependency>
        <dependency>
            <groupId>commons-httpclient</groupId>
            <artifactId>commons-httpclient</artifactId>
            <version>3.1</version>
        </dependency>
    </dependencies>

    <properties>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
    </properties>

打包完成后，就可以在hadoop平台直接运行了

 bin/hadoop jar /Users/xxxxxx/Project/hadoop_base/target/hadoop_base-1.0-SNAPSHOT.jar org.xxxx.algorithm.tools.WordCount /input /output/wc

遇到问题

/bin/bash: /bin/java: No such file or directory

22/06/01 15:26:32 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
22/06/01 15:26:33 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
22/06/01 15:26:33 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
22/06/01 15:26:33 INFO input.FileInputFormat: Total input files to process : 29
22/06/01 15:26:34 INFO mapreduce.JobSubmitter: number of splits:29
22/06/01 15:26:34 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1654067025193_0002
22/06/01 15:26:34 INFO conf.Configuration: resource-types.xml not found
22/06/01 15:26:34 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
22/06/01 15:26:34 INFO resource.ResourceUtils: Adding resource type - name = memory-mb, units = Mi, type = COUNTABLE
22/06/01 15:26:34 INFO resource.ResourceUtils: Adding resource type - name = vcores, units = , type = COUNTABLE
22/06/01 15:26:34 INFO impl.YarnClientImpl: Submitted application application_1654067025193_0002
22/06/01 15:26:34 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1654067025193_0002/
22/06/01 15:26:34 INFO mapreduce.Job: Running job: job_1654067025193_0002
22/06/01 15:26:36 INFO mapreduce.Job: Job job_1654067025193_0002 running in uber mode : false
22/06/01 15:26:36 INFO mapreduce.Job:  map 0% reduce 0%
22/06/01 15:26:36 INFO mapreduce.Job: Job job_1654067025193_0002 failed with state FAILED due to: Application application_1654067025193_0002 failed 2 times due to AM Container for appattempt_1654067025193_0002_000002 exited with  exitCode: 127
Failing this attempt.Diagnostics: [2022-06-01 15:26:36.146]Exception from container-launch.
Container id: container_1654067025193_0002_02_000001
Exit code: 127

[2022-06-01 15:26:36.147]Container exited with a non-zero exit code 127. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
/bin/bash: /bin/java: No such file or directory


[2022-06-01 15:26:36.147]Container exited with a non-zero exit code 127. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
/bin/bash: /bin/java: No such file or directory


For more detailed output, check the application tracking page: http://localhost:8088/cluster/app/application_1654067025193_0002 Then click on links to logs of each attempt.
. Failing the application.
22/06/01 15:26:36 INFO mapreduce.Job: Counters: 0

 解决方法：

修改 etc/hadoop/hadoop-env.sh 

export JAVA_HOME=/path/to/java/home