在IDEA中实现wordcount词频统计
需求分析:
1.Map过程:并行读取文本,对读取的单词进行map操作,每个词都以<key,value>形式生成
2.Reduce操作是对map的结果进行排序合并最后得出词频
1.添加maven依赖
注意hadoop-core依赖与hadoop-auth冲突,要求先注释掉。
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
<scope>test</scope>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.6.0</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>2.6.0</version>
</dependency>
<!-- <dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>