阿里云搭建单机Hadoop之YARN

YARN环境搭建

配置

路径 ：hadoop-2.6.0-cdh5.7.0/etc/hadoop
图示：

发现并没有下述的文件。所以需要进行对应的复制

$ cp mapred-site.xml.template mapred-site.xml

需要配置文件：
yarn-site.xml

<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>

mapred-site.xml

<property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
</property>

启动

$ sbin/start-yarn.sh

验证

jsp

或者直接web查看
http://localhost:8088/

停止 :

$ sbin/stop-yarn.sh

接下来是测试用例：

提交mr作业到yarn运行：
/usr/local/Hadoop/hadoop-2.6.0-cdh5.7.0/share/hadoop/mapreduce
hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar

$ hadoop jar hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar pi 2 3

结果：

Number of Maps  = 2
Samples per Map = 3
19/02/14 17:12:19 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Wrote input for Map #0
Wrote input for Map #1
Starting Job
19/02/14 17:12:21 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
19/02/14 17:12:22 INFO input.FileInputFormat: Total input paths to process : 2
19/02/14 17:12:22 INFO mapreduce.JobSubmitter: number of splits:2
19/02/14 17:12:22 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1550135276408_0001
19/02/14 17:12:22 INFO impl.YarnClientImpl: Submitted application application_1550135276408_0001
19/02/14 17:12:22 INFO mapreduce.Job: The url to track the job: http://iz2zeixqaqrvghplsi69imz:8088/proxy/application_1550135276408_0001/
19/02/14 17:12:22 INFO mapreduce.Job: Running job: job_1550135276408_0001
19/02/14 17:12:29 INFO mapreduce.Job: Job job_1550135276408_0001 running in uber mode : false
19/02/14 17:12:29 INFO mapreduce.Job:  map 0% reduce 0%
19/02/14 17:12:34 INFO mapreduce.Job:  map 50% reduce 0%
19/02/14 17:12:35 INFO mapreduce.Job:  map 100% reduce 0%
19/02/14 17:12:39 INFO mapreduce.Job:  map 100% reduce 100%
19/02/14 17:12:40 INFO mapreduce.Job: Job job_1550135276408_0001 completed successfully
19/02/14 17:12:40 INFO mapreduce.Job: Counters: 49
    File System Counters
        FILE: Number of bytes read=50
        FILE: Number of bytes written=335562
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=556
        HDFS: Number of bytes written=215
        HDFS: Number of read operations=11
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=3
    Job Counters
        Launched map tasks=2
        Launched reduce tasks=1
        Data-local map tasks=2
        Total time spent by all maps in occupied slots (ms)=5902
        Total time spent by all reduces in occupied slots (ms)=3001
        Total time spent by all map tasks (ms)=5902
        Total time spent by all reduce tasks (ms)=3001
        Total vcore-seconds taken by all map tasks=5902
        Total vcore-seconds taken by all reduce tasks=3001
        Total megabyte-seconds taken by all map tasks=6043648
        Total megabyte-seconds taken by all reduce tasks=3073024
    Map-Reduce Framework
        Map input records=2
        Map output records=4
        Map output bytes=36
        Map output materialized bytes=56
        Input split bytes=320
        Combine input records=0
        Combine output records=0
        Reduce input groups=2
        Reduce shuffle bytes=56
        Reduce input records=4
        Reduce output records=0
        Spilled Records=8
        Shuffled Maps =2
        Failed Shuffles=0
        Merged Map outputs=2
        GC time elapsed (ms)=213
        CPU time spent (ms)=1670
        Physical memory (bytes) snapshot=666251264
        Virtual memory (bytes) snapshot=8383307776
        Total committed heap usage (bytes)=559415296
    Shuffle Errors
        BAD_ID=0
        CONNECTION=0
        IO_ERROR=0
        WRONG_LENGTH=0
        WRONG_MAP=0
        WRONG_REDUCE=0
    File Input Format Counters
        Bytes Read=236
    File Output Format Counters
        Bytes Written=97
Job Finished in 18.998 seconds
Estimated value of Pi is 4.00000000000000000000

最后的Pi是4,，具体是因为啥，我暂时也不知道