hadoop2.7伪分布式模式运行C++程序

按上一篇步骤，伪分布式部署好之后，先执行一次hadoop自带的测试程序，确认是否可以正常运行

1、创建输入目录hadoop fs -mkdir /input

2、创建输入文件t，内容为1 2 4 5 6，上传到hdfs，hadoop fs -put t /input

3、进入hadoop安装目录，执行命令

hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar wordcount /input/t /output

第一次一直卡在 mapreduce.Job:  map 0% reduce 0%，磁盘占用率100%，后来查看logs/userlogs下面的日志，日志的目录结构为

usrlogs

 ----application_********

--------container_*******

------------stderr stdout syslog

其中stderr包含了关键错误，查看此文件，发现最后有一行

Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f9ad8fce000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)

确定是虚拟机分配的内存不足，将虚拟机内存由原来的1G调整为2G

再次执行测试命令

又出现以下错误

16/02/18 10:36:54 INFO client.RMProxy: Connecting to ResourceManager at hadoop1/188.168.1.180:8032
16/02/18 10:36:58 INFO input.FileInputFormat: Total input paths to process : 1
16/02/18 10:36:59 INFO mapreduce.JobSubmitter: number of splits:1
16/02/18 10:36:59 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1455762878791_0001
16/02/18 10:37:01 INFO impl.YarnClientImpl: Submitted application application_1455762878791_0001
16/02/18 10:37:01 INFO mapreduce.Job: The url to track the job: http://hadoop1:8088/proxy/application_1455762878791_0001/
16/02/18 10:37:01 INFO mapreduce.Job: Running job: job_1455762878791_0001
16/02/18 10:37:35 INFO mapreduce.Job: Job job_1455762878791_0001 running in uber mode : false
16/02/18 10:37:35 INFO mapreduce.Job:  map 0% reduce 0%
16/02/18 10:37:53 INFO mapreduce.Job:  map 100% reduce 0%
16/02/18 10:37:53 INFO mapreduce.Job: Task Id : attempt_1455762878791_0001_m_000000_0, Status : FAILED
Exception from container-launch.
Container id: container_1455762878791_0001_01_000002
Exit code: 1
Stack trace: ExitCodeException exitCode=1: 
        at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
        at org.apache.hadoop.util.Shell.run(Shell.java:456)
        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
        at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)




Container exited with a non-zero exit code 1

查看错误日志，一是也没有头绪，从网上查看其他人的配置过程，发现自己在上一篇的配置过程中，忘了修改yarn-env.sh，修改如下

# export JAVA_HOME=/home/y/libexec/jdk1.6.0/
export JAVA_HOME=/usr/java/jdk1.8.0_71

再次重启集群服务，执行测试命令，通过。

下面将自己用C++编写的WordCount程序上传到hdfs

hadoop fs -put wordcount /bin/wordcount

启动程序

hadoop pipes -D hadoop.pipes.java.recordreader=true -D hadoop.pipes.java.recordwriter=true -D mapred.job.name=wordcount -input /input/t -output output -program /bin/wordcount

得到如下错误

Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000f7780000, 40894464, 0) failed; error='Cannot allocate memory' (errno=12)

看来还是内存不足，继续加大内存到2.5G。再次执行测试，通过。