MapReduce之WordCount

1.hadoop fs常用命令

根目录就是core-site.xml中配置的hdfs://hadoop001:9000/

hadoop fs -mkdir /
hadoop fs -put
hadoop fs -get
hadoop fs -cat
hadoop fs -rm
hadoop fs -ls

[wzj@hadoop001 ~]$ hadoop fs -mkdir /a
19/12/02 14:02:39 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

[wzj@hadoop001 ~]$ hadoop fs -ls /
19/12/02 14:02:45 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 1 items
drwxr-xr-x   - wzj supergroup          0 2019-12-02 14:02 /a
[wzj@hadoop001 app]$ hadoop fs -ls hdfs://hadoop001:9000/
19/12/02 17:55:48 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 1 items
drwxr-xr-x   - wzj supergroup          0 2019-12-02 14:04 hdfs://hadoop001:9000/a

[wzj@hadoop001 ~]$ hadoop fs -put ce.log /a/
19/12/02 14:04:27 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[wzj@hadoop001 ~]$ hadoop fs -ls /a/
19/12/02 14:05:04 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 1 items
-rw-r--r--   1 wzj supergroup         14 2019-12-02 14:04 /a/ce.log
[wzj@hadoop001 ~]$ hadoop fs -cat /a/ce.log
19/12/02 14:05:23 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
start bigdata

2.使用MapReduce运行WordCount案例

2.1 创建文本文件

[wzj@hadoop001 data]$ vi name.log

wangzhen
wzj
ruoz
alibaba
zhengqi
caorui
bb
llmmzz
ll
zz
1
2
3
3
4
1
123
a
b
c
d
e
f
aaa
bbb

2.2文本文件put到hdfs上

[wzj@hadoop001 data]$ hdfs dfs -mkdir -p /wordcount/input
19/12/02 18:04:19 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[wzj@hadoop001 data]$ hdfs dfs -put name.log /wordcount/input
19/12/02 18:04:39 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[wzj@hadoop001 data]$ hdfs dfs -ls /wordcount/input/
19/12/02 18:04:51 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 1 items
-rw-r--r--   1 wzj supergroup         93 2019-12-02 18:04 /wordcount/input/name.log
[wzj@hadoop001 data]$

2.3查找统计的jar包

[wzj@hadoop001 hadoop]$ find ./ -name ‘example.jar’
./share/hadoop/mapreduce1/hadoop-examples-2.6.0-mr1-cdh5.16.2.jar
./share/hadoop/mapreduce2/sources/hadoop-mapreduce-examples-2.6.0-cdh5.16.2-test-sources.jar
./share/hadoop/mapreduce2/sources/hadoop-mapreduce-examples-2.6.0-cdh5.16.2-sources.jar
./share/hadoop/mapreduce2/hadoop-mapreduce-examples-2.6.0-cdh5.16.2.jar

2.4 运行jar包，对文件进行统计

[wzj@hadoop001 hadoop]$ hadoop jar share/hadoop/mapreduce2/hadoop-mapreduce-examples-2.6.0-cdh5.16.2.jar wordcount /wordcount/input /wordcount/output

2.5 查看结果

[wzj@hadoop001 hadoop]$ hdfs dfs -cat /wordcount/output/*
19/12/02 18:10:40 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
1	2
123	1
2	1
3	2
4	1
a	1
aaa	1
alibaba	1
b	1
bb	1
bbb	1
c	1
caorui	1
d	1
e	1
f	1
ll	1
llmmzz	1
ruoz	1
wangzhen	1
wzj	1
zhengqi	1
zz	1
[wzj@hadoop001 hadoop]$