hadoop之简单计算

配置主机名
    nn：vim core-site.xml

 
<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://hadoop001:9000</value>
    </property>
</configuration>

    dn:  vim slaves

[hadoop@hadoop001 hadoop]$ cat slaves
hadoop001

   snn:  vim hdfs-site.xml
   

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
    <property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>hadoop001:50090</value>
    </property>
    <property>
        <name>dfs.namenode.secondary.https-address</name>
        <value>hadoop001:50091</value>
    </property>
</configuration>

  启动

  [hadoop@hadoop001 hadoop]$ ../../sbin/start-dfs.sh

   MapReduce：计算
   Yarn: 资源和作业的调度 

[hadoop@hadoop001 hadoop]$ cat yarn-site.xml
<configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
</configuration>

  [hadoop@hadoop001 hadoop]$ cat mapred-site.xml

  <configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
  </configuration>

  

[hadoop@hadoop001 hadoop]$ jps
19264 DataNode
19169 NameNode
23764 Jps
19831 NodeManager
19736 ResourceManager
19453 SecondaryNameNode
[hadoop@hadoop001 hadoop]$ netstat -lntup | grep 19736
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 :::8032                     :::*                        LISTEN      19736/java          
tcp        0      0 :::8033                     :::*                        LISTEN      19736/java          
tcp        0      0 :::8088                     :::*                        LISTEN      19736/java          
tcp        0      0 :::8030                     :::*                        LISTEN      19736/java          
tcp        0      0 :::8031                     :::*                        LISTEN      19736/java          
[hadoop@hadoop001 hadoop]$ 

  简单计算

[hadoop@hadoop001 hadoop]$ cat 1.log
a
c
b
sina.com
[hadoop@hadoop001 hadoop]$ cat 2.log
a b c
www.sina.com
a c zzzz
[hadoop@hadoop001 hadoop]$ vim 1.log
[hadoop@hadoop001 hadoop]$ vim 2.log
[hadoop@hadoop001 hadoop]$ bin/hdfs dfs -mkdir /examples
19/07/08 20:36:35 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[hadoop@hadoop001 hadoop]$ bin/hdfs dfs -mkdir /examples/input
19/07/08 20:37:06 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[hadoop@hadoop001 hadoop]$ bin/hdfs dfs -put *.log /examples/input
19/07/08 20:38:05 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[hadoop@hadoop001 hadoop]$ bin/hdfs dfs -ls /examples/input
19/07/08 20:38:28 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 2 items
-rw-r--r--   1 hadoop supergroup         15 2019-07-08 20:38 /examples/input/1.log
-rw-r--r--   1 hadoop supergroup         28 2019-07-08 20:38 /examples/input/2.log
[hadoop@hadoop001 hadoop]$ bin/hadoop jar /home/hadoop/app/hadoop/share/hadoop/mapreduce2/hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar wordcount  /examples/input /examples/output
19/07/08 20:39:48 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/07/08 20:39:50 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
19/07/08 20:39:52 INFO input.FileInputFormat: Total input paths to process : 2
19/07/08 20:39:53 INFO mapreduce.JobSubmitter: number of splits:2
19/07/08 20:39:53 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1562568082175_0001
19/07/08 20:39:54 INFO impl.YarnClientImpl: Submitted application application_1562568082175_0001
19/07/08 20:39:54 INFO mapreduce.Job: The url to track the job: http://hadoop001:8088/proxy/application_1562568082175_0001/
19/07/08 20:39:54 INFO mapreduce.Job: Running job: job_1562568082175_0001
19/07/08 20:40:36 INFO mapreduce.Job: Job job_1562568082175_0001 running in uber mode : false
19/07/08 20:40:36 INFO mapreduce.Job:  map 0% reduce 0%
19/07/08 20:41:15 INFO mapreduce.Job:  map 50% reduce 0%
19/07/08 20:41:16 INFO mapreduce.Job:  map 100% reduce 0%
19/07/08 20:41:23 INFO mapreduce.Job:  map 100% reduce 100%

1562568082175为时间戳         开始时间   结束时间   完成状态  成功状态

计算结果

hadoop@hadoop001 hadoop]$ bin/hdfs dfs -ls /examples/output
19/07/08 21:36:07 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 2 items
-rw-r--r--   1 hadoop supergroup          0 2019-07-08 20:41 /examples/output/_SUCCESS
-rw-r--r--   1 hadoop supergroup         45 2019-07-08 20:41 /examples/output/part-r-00000
[hadoop@hadoop001 hadoop]$ bin/hdfs dfs -cat /examples/output/part-r-00000
19/07/08 21:37:03 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
a	3
b	2
c	3
sina.com	1
www.sina.com	1
zzzz	1