在CentOS 使用 ideaIC 开发工具 ideaIC
准备 idealC 安装包
ideaIC-2016.3.tar.gz 下载地址
上传到centos服务器上解压
tar -xzvf ideaIC-2016.2.3.tar.gz
进入bin目录打开idea.sh 进行安装
一切按照默认安装完成
创建项目 create new project
选择默认带的JDK 也可以 new sdk 选择本地安装的jdk也
开始编写 wordcount
创建项目 create new project
选择默认带的JDK 也可以 new sdk 选择本地安装的jdk也
下一步 template 不选 直接下一步 输入项目名称 WordCount 点击 Finish
写一个 main 方法测试一下 成功输出test...
开始编写 wordcount
导入hadoop包 (可以把所有jar放到一个文件夹引入)
File -> Project Structure(快捷键 Ctrl + Alt + Shift + s),点击Project Structure界面左侧的“Modules”显示下图界面。
选择 Dependencies 加 右边 +
选择 1 JARs or directories...
编写代码如下:
package z.test;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import java.io.IOException;
/**
* Created by z on .
*/
public class WordCount {
public static void main(String[] args) throws Exception{
if(args.length == 0){
System.out.println("args is null exit....");
System.exit(0);
}
Configuration configuration = new Configuration();
Job job = new Job(configuration);
job.setJarByClass(WordCount.class);
FileInputFormat.setInputPaths(job,new Path(args[0]));
FileOutputFormat.setOutputPath(job,new Path(args[1]));
job.setMapperClass(TestMap.class);
job.setReducerClass(TestReduce.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
job.waitForCompletion(true);
System.out.println("test....");
}
public static class TestMap extends Mapper
{
@Override
protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
String[] words = value.toString().split(" ");
for (String word : words){
context.write(new Text(word),new IntWritable(1));
}
}
}
public static class TestReduce extends Reducer
{
@Override
protected void reduce(Text key, Iterable
values, org.apache.hadoop.mapreduce.Reducer
.Context context) throws IOException, InterruptedException {
int count = 0;
for(IntWritable i : values){
count += i.get();
}
context.write(key,new IntWritable(count));
}
}
}
打包
File->project stucture
Artifacts->"+",选jar,选择from modules with dependencies,然后会有配置窗口出现,配置完成后,勾选Build on make >ok保存
启动hadop sbin/start-all.sh
执行 jar 包
hadoop jar WordCount.jar z.test.WordCount /in/words /out/wordcount
-------end----------
欢迎关注精彩黑科技
-------end----------
欢迎关注精彩黑科技