第一个hadoop程序


向hdfs:///dfs_operator.txt文件中写入10万次 test。很无聊啊,呵呵


不错,简单的程序没有出错,正常运行了

[lngz@LionmatoMacBook-Pro ~/hadoop]$ bin/hadoop jar ./dfs_operator.jar DFSOperator

[lngz@LionmatoMacBook-Pro ~/hadoop]$  bin/hadoop fs -ls /
Found 2 items
drwxr-xr-x   - lngz supergroup          0 2012-04-17 18:05 /Users
-rw-r--r--   1 lngz supergroup     800000 2012-04-18 16:53 /dfs_operator.txt
[lngz@LionmatoMacBook-Pro ~/hadoop]$  bin/hadoop fs -cat /dfs_operator.txt

-------------source-----------

import java.io.*;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.*;
public class DFSOperator {

 /**
  * @param args
  */
 public static void main(String[] args) {
  // TODO Auto-generated method stub
  Configuration conf = new Configuration();
    try {
    FileSystem fs = FileSystem.get(conf);
    Path f = new Path("hdfs:///dfs_operator.txt");
    FSDataOutputStream s = fs.create(f, true);
    int i=0;
    for (i=0; i<100000; ++i)
    s.writeChars("test");
    s.close();
    } catch (IOException e) {
    e.printStackTrace();
    }
 }
}

  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
Hadoop的第一个WordCount程序是一个非常简单的程序,它的主要目的是计算一个文本文件中每个单词出现的次数。 以下是一个基本的WordCount程序: 1. 创建一个Java项目并导入Hadoop库。 2. 创建一个Java类并实现以下Mapper和Reducer: Mapper类: ```java import java.io.IOException; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Mapper; public class WordCountMapper extends Mapper<LongWritable, Text, Text, LongWritable> { private final static LongWritable one = new LongWritable(1); private Text word = new Text(); @Override public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { String line = value.toString(); String[] words = line.split(" "); for (String w : words) { word.set(w); context.write(word, one); } } } ``` Reducer类: ```java import java.io.IOException; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Reducer; public class WordCountReducer extends Reducer<Text, LongWritable, Text, LongWritable> { @Override public void reduce(Text key, Iterable<LongWritable> values, Context context) throws IOException, InterruptedException { long sum = 0; for (LongWritable val : values) { sum += val.get(); } context.write(key, new LongWritable(sum)); } } ``` 3. 在应用程序的main()方法中,创建一个Job并设置Mapper和Reducer类: ```java import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; public class WordCount { public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); Job job = Job.getInstance(conf, "word count"); job.setJarByClass(WordCount.class); job.setMapperClass(WordCountMapper.class); job.setCombinerClass(WordCountReducer.class); job.setReducerClass(WordCountReducer.class); job.setMapOutputKeyClass(Text.class); job.setMapOutputValueClass(LongWritable.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(LongWritable.class); FileInputFormat.addInputPath(job, new Path(args[0])); FileOutputFormat.setOutputPath(job, new Path(args[1])); System.exit(job.waitForCompletion(true) ? 0 : 1); } } ``` 4. 在命令行中运行程序,并指定输入和输出文件路径: ```bash hadoop jar WordCount.jar WordCount /input /output ``` 其中,/input是输入文件路径,/output是输出文件路径。 这是最基本的WordCount程序,你可以根据需要进行修改和扩展。

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值