命令行编译运行WordCount.java

1.首先拷贝hadoop源码里的WordCount.java到一个目录wordcount下
 
2.在wordcount目录下新建目录bin准备存放class文件
 
3.编译WordCount.java文件(编译前先把java文件中的package包名删掉):
javac -classpath /usr/local/hadoop/share/hadoop/common/hadoop-common-2.6.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.6.0.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-cli-1.2.jar -d bin WordCount.java
 
4.制作jar包:  
 
jar -cvf WordCount.jar ./WordCount*.class
 
5.生成input在bin中执行命令
mkdir input
echo "echo of the rainbow" > ./input/file0
echo "the waiting game" > ./input/file1
 
6.在hadoop单机版运行WordCount.jar包
/usr/local/hadoop/bin/hadoop   jar   bin/WordCount.jar   WordCount   input   output
 
7.查看output结果(结果在output文件夹下的part-r-00000文件中)
 cat ./output/part-r-00000 或
 vim ./output/part-r-00000
 
8附WordCount.java源代码
/**
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership.  The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package org.apache.hadoop.examples;

import java.io.IOException;
import java.util.StringTokenizer;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.GenericOptionsParser;

public class WordCount {

  public static class TokenizerMapper 
       extends Mapper<Object, Text, Text, IntWritable>{
    
    private final static IntWritable one = new IntWritable(1);
    private Text word = new Text();
      
    public void map(Object key, Text value, Context context
                    ) throws IOException, InterruptedException {
      StringTokenizer itr = new StringTokenizer(value.toString());
      while (itr.hasMoreTokens()) {
        word.set(itr.nextToken());
        context.write(word, one);
      }
    }
  }
  
  public static class IntSumReducer 
       extends Reducer<Text,IntWritable,Text,IntWritable> {
    private IntWritable result = new IntWritable();

    public void reduce(Text key, Iterable<IntWritable> values, 
                       Context context
                       ) throws IOException, InterruptedException {
      int sum = 0;
      for (IntWritable val : values) {
        sum += val.get();
      }
      result.set(sum);
      context.write(key, result);
    }
  }

  public static void main(String[] args) throws Exception {
    Configuration conf = new Configuration();
    String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();
    if (otherArgs.length != 2) {
      System.err.println("Usage: wordcount <in> <out>");
      System.exit(2);
    }
    Job job = new Job(conf, "word count");
    job.setJarByClass(WordCount.class);
    job.setMapperClass(TokenizerMapper.class);
    job.setCombinerClass(IntSumReducer.class);
    job.setReducerClass(IntSumReducer.class);
    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(IntWritable.class);
    FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
    FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
    System.exit(job.waitForCompletion(true) ? 0 : 1);
  }
}

 

  • 0
    点赞
  • 5
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
要在Hadoop上运行Java程序,可以按照以下步骤进行操作: 1. 开发Java程序,并将其打包成可执行的JAR文件。 2. 在Hadoop集群上创建输入和输出目录。 3. 将JAR文件上传到Hadoop集群中。 4. 使用Hadoop命令行工具运行JAR文件,将输入目录和输出目录作为参数传递给程序。 下面是一个简单的示例,演示如何在Hadoop上运行Java程序: 1. 开发Java程序 编写一个简单的Java程序,用于统计文本文件中每个单词出现的次数。这里假设程序的名称为WordCount.java。 2. 打包JAR文件 使用Java编译编译程序,并将其打包成一个JAR文件。假设JAR文件的名称为wordcount.jar。 3. 创建输入和输出目录 在Hadoop集群上创建输入和输出目录。假设输入目录为/input,输出目录为/output。 4. 上传JAR文件 将JAR文件上传到Hadoop集群中。可以使用Hadoop命令行工具或FTP等工具进行上传。 5. 运行程序 使用Hadoop命令行工具运行JAR文件,并将输入目录和输出目录作为参数传递给程序。假设代码中的主类名为WordCountMain,运行命令如下: ``` hadoop jar wordcount.jar WordCountMain /input /output ``` 这将启动Hadoop作业,并将输入目录和输出目录作为参数传递给程序。程序将读取输入目录中的文件,并将结果写入输出目录中。 注意:Hadoop的具体命令和参数可能会因版本不同而有所变化。在运行程序之前,请查看相关文档,并确保您已正确配置Hadoop集群。

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值