利用Hadoop streaming 进行词频统计

创建一个文件夹

bin/hdfs dfs -mkdir /input

将要统计的文件上传到hadoop
bin/hadoop fs -put /test.txt /input

利用hadoop进行词频统计
bin/hadoop jar share/hadoop/tools/lib/Hadoop-streaming-2-9-2.jar –input /test.txt –output /user/results.txt –mapper /bin/cat -reducer /usr/bin/wc

删除results.txt文件

./bin/hadoop dfs -rmr /user/results.txt

查看results.txt文件目录

bin/hadoop dfs -ls /user/results.txt

查看统计结果

bin/hadoop dfs -ls /user/results.txt/part-0000

 

转载于:https://www.cnblogs.com/BigWatermelon/p/10844953.html