Hadoop IK分词 词频统计

    本文实验使用的是IK Analyzer作为分词器, Hadoop2.4作为计算框架的测试DEMO.

第一步: 搭建Ecelipse, 使用MAVEN构建.

pom.xml

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>com.nxcjh.iktest</groupId>
  <artifactId>iktest</artifactId>
  <version>0.0.1-SNAPSHOT</version>
  <packaging>jar</packaging>

  <name>iktest</name>
  <url>http://maven.apache.org</url>

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  </properties>

  <dependencies>
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-hdfs</artifactId>
    <version>2.4.0</version>
</dependency>
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-mapreduce-client-jobclient</artifactId>
    <version>2.4.0</version>
</dependency>
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-common</artifactId>
    <version>2.4.0</version>
</dependency>
<!-- https://mvnrepository.com/artifact/commons-cli/commons-cli -->
<dependency>
    <groupId>commons-cli</groupId>
    <artifactId>commons-cli</artifactId>
    <version>1.3.1</version>
</dependency>
<dependency>
    <groupId>com.nxcjh.ikanalyzer</groupId>
    <artifactId>ikanalyzer</artifactId>
    <version>1.1.0</version>
</dependency>
  </dependencies>
  <!-- 打依赖包 -->
 <build>
     <plugins>
          <plugin>
               <artifactId>maven-assembly-plugin</artifactId>
               <configuration>
                    <archive>
                         <manifest>
                              <mainClass>com.nxcjh.iktest.IKJob</mainClass>
                         </manifest>
                    </archive>
                    <descriptorRefs>
                         <descriptorRef>jar-with-dependencies</descriptorRef>
                    </descriptorRefs>
               </configuration>
               <executions>
                    <execution>
                         <id>make-assembly</id>
                         <phase>package</phase>
                         <goals>
                              <goal>single</goal>
                         </goals>
                    </execution>
               </executions>
          </plugin>
     </plugins>
</build>
</project>

代码:Mapper

package com.nxcjh.iktest;

import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.Reader;

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;
import org.wltea.analyzer.core.IKSegmenter;
import org.wltea.analyzer.core.Lexeme;

public class IKMapper extends Mapper<Object, Text, Text, IntWritable> {

	private final static IntWritable one = new IntWritable(1);
	private Text word = new Text();

	public void map(Object key, Text value, Context context)
			throws IOException, InterruptedException {

		byte[] bt = value.getBytes();
		InputStream ip = new ByteArrayInputStream(bt);
		Reader read = new InputStreamReader(ip);
		IKSegmenter iks = new IKSegmenter(read, true);
		Lexeme t;
		while ((t = iks.next()) != null) {
			word.set(t.getLexemeText());
			context.write(word, one);
		}
	}
}

Reducer

package com.nxcjh.iktest;

import java.io.IOException;

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;

public class IKReducer extends
Reducer<Text, IntWritable, Text, IntWritable> {

private IntWritable result = new IntWritable();

public void reduce(Text key, Iterable<IntWritable> values, Context context)
	throws IOException, InterruptedException {
	int sum = 0;
	for (IntWritable val : values) {
		sum += val.get();
	}
	result.set(sum);
	context.write(key, result);
	}
}

Job

package com.nxcjh.iktest;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.GenericOptionsParser;


public class IKJob {
	public static void main(String[] args) throws Exception {
	    Configuration conf = new Configuration();
	    String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();
	    if (otherArgs.length != 2) {
	      System.err.println("Usage: wordcount <in> <out>");
	      System.exit(2);
	    }
	    Job job = new Job(conf, "word count");
	    job.setJarByClass(IKJob.class);
	    job.setMapperClass(IKMapper.class);
	    job.setCombinerClass(IKReducer.class);
	    job.setReducerClass(IKReducer.class);
	    job.setOutputKeyClass(Text.class);
	    job.setOutputValueClass(IntWritable.class);
	    FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
	    FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
	    System.exit(job.waitForCompletion(true) ? 0 : 1);
	  }
}

注意: 因为无法找到IK Analyzer 的jar包, 所以手动本地添加到MAVEN库. 

mvn install:install-file -Dfile=IKAnalyzer2012_u6.jar -DgroupId=com.nxcjh.ikanalyzer -DartifactId=ikanalyzer -Dversion=1.1.0 -Dpackaging=jar

第二部: 打包构建jar包:

问题: 因为Hadoop集群中没有IK Analyzer 的jar包, 所以需要依赖第三方jar包, 有两种处理方式:

1. 在打包的时候把依赖包打进去, 缺点: 包占用空间较大.

在pom.xml 中添加:

<build>
     <plugins>
          <plugin>
               <artifactId>maven-assembly-plugin</artifactId>
               <configuration>
                    <archive>
                         <manifest>
                              <mainClass>com.nxcjh.iktest.IKJob</mainClass>
                         </manifest>
                    </archive>
                    <descriptorRefs>
                         <descriptorRef>jar-with-dependencies</descriptorRef>
                    </descriptorRefs>
               </configuration>
               <executions>
                    <execution>
                         <id>make-assembly</id>
                         <phase>package</phase>
                         <goals>
                              <goal>single</goal>
                         </goals>
                    </execution>
               </executions>
          </plugin>
     </plugins>
</build>

在工程目录中执行命令: mvn package

显示执行成功.

2. 在执行命令时添加依赖jar包.

在打包的时候不添加依赖jar包, 执行mvn命令: mvn clean install.

在执行hadoop命令如下:

hadoop jar [jar文件] [main主类] -libjars [依赖的jar包文件路径] [输入路径] [输出路径]

示例 : hadoop jar ikdemo-0.0.1-SNAPSHOT.jar com.mogujie.ikdemo.job.IKJob -libjars IKAnalyzer2012_u6.jar /input/street /output/street

第三步: 因为本文只是对词频进行统计, 没有涉及到排序, 所以可以使用sort进行排序:

1. 把结果下载到本地, 

2. 对结果进行排序,注意\t 需要使用$'\t'进行分隔.

sort -nrk2 -t$'\t' part-r-00000 > r1.

完成.