window下eclipse提交hadoop作业遇到的坑

1 用户权限问题，由于是windows用户，可能与远程集群用户不一致

解决方案：

首先集群端/usr/local/Cellar/hadoop/2.7.3/libexec/etc/hadoop/hdfs-site.xml下添加

 <property>
 <name>dfs.permissions</name>
 <value>false</value>
 </property>
 <property>
 <name>dfs.web.ugi</name>
 <value>jimmy,supergroup</value>    //这里jimmy为我当时安装hadoop时的操作系统用户
 </property>
 <property>
 <name>dfs.datanode.data.dir.perm</name>
 <value>777</value>
 </property>

其次在windows下配置提交任务的用户环境变量（系统环境变量里配置）：

HADOOP_USER_NAME

jimmy

参考：http://www.mobile-open.com/2016/922034.html ，这里为了简单我重新再window下添加用户jimmy

2 ExitCodeException exitCode=1: /bin/bash: line 0: fg: no job con 

参考：http://blog.csdn.net/ckfflyingdream/article/details/50112113（这篇文章提到下载hadoop包，并且在配置HADOOP_HOME,其实没必要，应为我没并不需要安装hadoop在windows下）

本质是linux下的环境变量符号与windows下不一样造成的

解决方法：

在windows项目（客户端）maven目录src/main/resources下mapred-site.xml里添加

<configuration>
<property>
<name>mapreduce.framework.name</name> //这里我的mapred是yarn管理，开始没配置，提交也有问题
<value>yarn</value>
</property> 
    <property>  
        <name>mapred.remote.os</name>
        <value>Linux</value>
        <description>Remote MapReduce framework's OS, can be either Linux or Windows</description> 
    </property>
    <property>
        <name>mapreduce.app-submission.cross-platform</name>
        <value>true</value>
    </property>

</configuration>

3.项目提交没运行，

检查windows项目maven目录src/main/resources下yarn-site.xml里有没有内容（开始我也没有内容，后来发下运行没反应，于是添加）：

首先：

<configuration>
<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>192.168.1.119:8032</value>  //hadoop集群的resourcemanager监控地址 如果不填可能有问题
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>192.168.1.119:8030</value>   //hadoop集群的resourcemanager的scheduler监控地址
</property>
</configuration>

其次：

设置maven打包后的jar文件地址到mapred.jar参数，否则会报classnotfound异常：

conf.set("mapred.jar","C:\\Users\\jimmy\\.m2\\repository\\com\\jimmy\\ibeifeng\\hdfs-api\\0.0.1-SNAPSHOT\\hdfs-api-0.0.1-SNAPSHOT.jar");

最后：

检查windows项目maven目录src/main/resources下core-site.xml里

<configuration>
 <property>
 <name>fs.defaultFS</name>
 <value>hdfs://192.168.1.119:9000</value>

</property> 

</configuration>

注释://在elipse配置插件hadoop的location时，对于hdfs和mapred端口都配置9000，因为这里我的mapred时运行在yarn上的

4 测试com.cloudy.lesson.jobs.wordcount.WordCountJob

args参数：

hdfs://192.168.1.119:9000/user/jimmy/input/wordcount.log

 hdfs://192.168.1.119:9000/user/jimmy/output

结果：

16/11/28 13:35:29 ERROR util.Shell: Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.               

 //这不是重要，因为我没配置hadoop路径在elipse里面
at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:379)
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:394)
at org.apache.hadoop.util.Shell.<clinit>(Shell.java:387)
at org.apache.hadoop.util.GenericOptionsParser.preProcessForWindows(GenericOptionsParser.java:440)
at org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:486)
at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:170)
at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:153)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:64)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at com.cloudy.lesson.jobs.wordcount.WordCountJob.main(WordCountJob.java:22)
16/11/28 13:35:29 INFO wordcount.WordCountJob: Program start on com.cloudy.lesson.jobs.wordcount.WordCountJob
args:hdfs://192.168.1.119:9000/user/jimmy/input/wordcount.log
16/11/28 13:35:30 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/11/28 13:35:34 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
16/11/28 13:35:34 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
16/11/28 13:35:34 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
16/11/28 13:35:35 INFO client.RMProxy: Connecting to ResourceManager at /192.168.1.119:8032
16/11/28 13:35:36 INFO input.FileInputFormat: Total input paths to process : 1
16/11/28 13:35:36 INFO mapreduce.JobSubmitter: number of splits:1
16/11/28 13:35:37 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1480308176838_0011
16/11/28 13:35:37 INFO impl.YarnClientImpl: Submitted application application_1480308176838_0011
16/11/28 13:35:37 INFO mapreduce.Job: The url to track the job: http://jimmys-MacBook-Air.local:8088/proxy/application_1480308176838_0011/
16/11/28 13:35:37 INFO mapreduce.Job: Running job: job_1480308176838_0011
16/11/28 13:35:44 INFO mapreduce.Job: Job job_1480308176838_0011 running in uber mode : false
16/11/28 13:35:44 INFO mapreduce.Job:  map 0% reduce 0%
16/11/28 13:35:49 INFO mapreduce.Job:  map 100% reduce 0%
16/11/28 13:35:55 INFO mapreduce.Job:  map 100% reduce 100%
16/11/28 13:35:55 INFO mapreduce.Job: Job job_1480308176838_0011 completed successfully
16/11/28 13:35:55 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=1041119
FILE: Number of bytes written=2321281
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=515600
HDFS: Number of bytes written=20334
HDFS: Number of read operations=6
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters 
Launched map tasks=1
Launched reduce tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=3213
Total time spent by all reduces in occupied slots (ms)=3283
Total time spent by all map tasks (ms)=3213
Total time spent by all reduce tasks (ms)=3283
Total vcore-milliseconds taken by all map tasks=3213
Total vcore-milliseconds taken by all reduce tasks=3283
Total megabyte-milliseconds taken by all map tasks=3290112
Total megabyte-milliseconds taken by all reduce tasks=3361792
Map-Reduce Framework
Map input records=1788
Map output records=89859
Map output bytes=861395
Map output materialized bytes=1041119
Input split bytes=121
Combine input records=0
Combine output records=0
Reduce input groups=1967
Reduce shuffle bytes=1041119
Reduce input records=89859
Reduce output records=1967
Spilled Records=179718
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=133
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0
Total committed heap usage (bytes)=303562752
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters 
Bytes Read=515479
File Output Format Counters 
Bytes Written=20334

代码共享：https://github.com/amerigroup/hadoop