认识wordcount（一）

最新推荐文章于 2021-07-25 21:29:32 发布
最新推荐文章于 2021-07-25 21:29:32 发布
阅读量162 收藏
点赞数
分类专栏： hadoop 文章标签： wordcount MapReduce
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/weixin_43855370/article/details/101721904
版权

认识wordcount（一）

在这里插入图片描述

0. 执行前准备

0.1 拷贝Hadoop二进制包中的可执行Jar包

将Hadoop二进制包中的可执行Jar包拷贝到工程下新建的lib文件夹，将他们都右键【build path】->【Add to build path】

0.2 设置日志配置文件

将Hadoop二进制包中的hadoop-2.7.3\etc\hadoop\log4j.properties文件拷贝到src下，才可以看到执行过程的日志信息

1. 本地提交

1.1 测试数据

1.1.1 word1.txt

hadoop
hive spark
zookeeper pig flink
hadoop hbase
spark
hdfs
namenode
hadoop sqoop hive hbase

1.1.2 word2.txt

datanode namenode pig hive
spark hbase hadoop
sqoop
zookeeper
mahout
flume
yarn mapreduce
impala storm kafka

1.1.3 word3.txt

hadoop spark
hive
kafka
flume pig flink
zookeeper

1.2 WordCount程序

1.2.1 Mapper
package wordcount;

import java.io.IOException;

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;

public class WordCountMap extends Mapper<LongWritable, Text, Text, IntWritable>{
	private Text word = new Text();
	private IntWritable one = new IntWritable(1);
	@Override
	protected void map(LongWritable key, Text value, Mapper<LongWritable, Text, Text, IntWritable>.Context context)
			throws IOException, InterruptedException {
		String line = value.toString();
		String[] words = line.split(" ");
		for(String str : words) {
			word.set(str);
			context.write(word, one);
		}
	}
}

默认的文件读取格式是TextInputFormat（由job.setInputFormat(输入类.class)设定），key为每行的首字符在文件中的字节数，value是一行的文本。

  1. 将一行文本从TextWritable转为String
  2. 因为文本中间是空格分隔，所以用空格分解文本
  3. 遍历获得的String数组，将文本和1组合成<文本,1>的二元组
  4. 形成中间数据传递给Reducer
1.2.2 Reducer
package wordcount;

import java.io.IOException;

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;

public class WordCountRecuce extends Reducer<Text, IntWritable, Text, IntWritable>{
	private IntWritable _sum = new IntWritable();
	@Override
	protected void reduce(Text key, Iterable<IntWritable> values,
			Reducer<Text, IntWritable, Text, IntWritable>.Context context) throws IOException, InterruptedException {
		int sum = 0;
		for (IntWritable value : values) {
			sum += value.get();
			_sum.set(sum);
		}
		context.write(key, _sum);
	}
}

Mapper阶段结束后，传给Reducer的是一个key和value集合的二元组<key,Iterable>

  1. 遍历Mapper传过来的IntWritable集合，将相同key值的value统计成总数
  2. 输出key和总数的二元组<key,_sum>
1.2.3 Driver
package wordcount;

import java.io.IOException;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;


public class WordCountDriver {
	public static void main(String[] args) throws ClassNotFoundException, InterruptedException, IOException {
		Configuration conf = new Configuration();
		conf.set("mapreduce.framework.name", "local");
		Path outfile = new Path("file:///D:/out");
		FileSystem fs = outfile.getFileSystem(conf);
		if(fs.exists(outfile)){
			fs.delete(outfile,true);
		}
		Job job = Job.getInstance(conf);
		job.setMapperClass(WordCountMap.class);
		job.setReducerClass(WordCountRecuce.class);
		job.setJobName("wordcount");
		job.setJarByClass(WordCountDriver.class);
		job.setMapOutputKeyClass(Text.class);
		job.setMapOutputValueClass(IntWritable.class);
		job.setOutputKeyClass(Text.class);
		job.setOutputValueClass(IntWritable.class);
		FileInputFormat.addInputPath(job, new Path("file:///D:/wordcount"));
		FileOutputFormat.setOutputPath(job, outfile);
		System.exit(job.waitForCompletion(true)?0:1);
	}
}
  1. 首先，获取一个Configuration对象
  2. 设置配置为本地运行
  3. 创建一个输出的路径
  4. 用Configuration对象获得Job实例
  5. 设置Job的MapperClass
  6. 设置Job的ReducerClass
  7. 设置Job的Name
  8. 设置打包的主类JarByClass
  9. 设置Mapper输出的key类型MapOutputKeyClass
  10. 设置Mapper输出的value类型MapOutputValueClass
  11. 设置Reducer输出的key类型OutputKeyClass
  12. 设置Reducer输出的value类型OutputValueClass
  13. 设置文件输入的路径setInputPaths
  14. 设置文件输出的路径setOutputPath
  15. 设置等待Job完成的方法waitForCompletion（必须设置，否则不继续执行）

1.3 执行程序

19/09/21 09:19:05 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
19/09/21 09:19:05 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
19/09/21 09:19:06 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
19/09/21 09:19:06 WARN mapreduce.JobResourceUploader: No job jar file set. User classes may not be found. See Job or Job#setJar(String).
19/09/21 09:19:07 INFO input.FileInputFormat: Total input paths to process : 3
19/09/21 09:19:07 INFO mapreduce.JobSubmitter: number of splits:3
19/09/21 09:19:08 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local291050806_0001
19/09/21 09:19:08 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
19/09/21 09:19:08 INFO mapreduce.Job: Running job: job_local291050806_0001
19/09/21 09:19:08 INFO mapred.LocalJobRunner: OutputCommitter set in config null
19/09/21 09:19:08 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
19/09/21 09:19:08 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
19/09/21 09:19:08 INFO mapred.LocalJobRunner: Waiting for map tasks
19/09/21 09:19:08 INFO mapred.LocalJobRunner: Starting task: attempt_local291050806_0001_m_000000_0
19/09/21 09:19:08 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
19/09/21 09:19:08 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree currently is supported only on Linux.
19/09/21 09:19:08 INFO mapred.Task: Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@1e9ffb4f
19/09/21 09:19:08 INFO mapred.MapTask: Processing split: file:/D:/wordcount/word2.txt:0+115
19/09/21 09:19:08 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
19/09/21 09:19:08 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
19/09/21 09:19:08 INFO mapred.MapTask: soft limit at 83886080
19/09/21 09:19:08 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
19/09/21 09:19:08 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
19/09/21 09:19:08 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTaskKaTeX parse error: Double subscript at position 523: …91050806_0001_m_̲000000_0 is don…MapOutputBuffer
19/09/21 09:19:08 INFO mapred.LocalJobRunner:
19/09/21 09:19:08 INFO mapred.MapTask: Starting flush of map output
19/09/21 09:19:08 INFO mapred.MapTask: Spilling map output
19/09/21 09:19:08 INFO mapred.MapTask: bufstart = 0; bufend = 155; bufvoid = 104857600
19/09/21 09:19:08 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214340(104857360); length = 57/6553600
19/09/21 09:19:09 INFO mapred.MapTask: Finished spill 0
19/09/21 09:19:09 INFO mapred.Task: Task:attempt_local291050806_0001_m_000001_0 is done. And is in the process of committing
19/09/21 09:19:09 INFO mapred.LocalJobRunner: map
19/09/21 09:19:09 INFO mapred.Task: Task ‘attempt_local291050806_0001_m_000001_0’ done.
19/09/21 09:19:09 INFO mapred.LocalJobRunner: Finishing task: attempt_local291050806_0001_m_000001_0
19/09/21 09:19:09 INFO mapred.LocalJobRunner: Starting task: attempt_local291050806_0001_m_000002_0
19/09/21 09:19:09 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
19/09/21 09:19:09 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree currently is supported only on Linux.
19/09/21 09:19:09 INFO mapreduce.Job: Job job_local291050806_0001 running in uber mode : false
19/09/21 09:19:09 INFO mapreduce.Job: map 100% reduce 0%
19/09/21 09:19:09 INFO mapred.Task: Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@5a3854ff
19/09/21 09:19:09 INFO mapred.MapTask: Processing split: file:/D:/wordcount/word3.txt:0+54
19/09/21 09:19:09 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
19/09/21 09:19:09 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
19/09/21 09:19:09 INFO mapred.MapTask: soft limit at 83886080
19/09/21 09:19:09 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
19/09/21 09:19:09 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
19/09/21 09:19:09 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
19/09/21 09:19:09 INFO mapred.LocalJobRunner:
19/09/21 09:19:09 INFO mapred.MapTask: Starting flush of map output
19/09/21 09:19:09 INFO mapred.MapTask: Spilling map output
19/09/21 09:19:09 INFO mapred.MapTask: bufstart = 0; bufend = 82; bufvoid = 104857600
19/09/21 09:19:09 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214368(104857472); length = 29/6553600
19/09/21 09:19:09 INFO mapred.MapTask: Finished spill 0
19/09/21 09:19:09 INFO mapred.Task: Task:attempt_local291050806_0001_m_000002_0 is done. And is in the process of committing
19/09/21 09:19:09 INFO mapred.LocalJobRunner: map
19/09/21 09:19:09 INFO mapred.Task: Task ‘attempt_local291050806_0001_m_000002_0’ done.
19/09/21 09:19:09 INFO mapred.LocalJobRunner: Finishing task: attempt_local291050806_0001_m_000002_0
19/09/21 09:19:09 INFO mapred.LocalJobRunner: map task executor complete.
19/09/21 09:19:09 INFO mapred.LocalJobRunner: Waiting for reduce tasks
19/09/21 09:19:09 INFO mapred.LocalJobRunner: Starting task: attempt_local291050806_0001_r_000000_0
19/09/21 09:19:09 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
19/09/21 09:19:09 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree currently is supported only on Linux.
19/09/21 09:19:09 INFO mapred.Task: Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@3fb1d90f
19/09/21 09:19:09 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@453cd5e7
19/09/21 09:19:09 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=1323407744, maxSingleShuffleLimit=330851936, mergeThreshold=873449152, ioSortFactor=10, memToMemMergeOutputsThreshold=10
19/09/21 09:19:09 INFO reduce.EventFetcher: attempt_local291050806_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
19/09/21 09:19:09 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local291050806_0001_m_000002_0 decomp: 100 len: 104 to MEMORY
19/09/21 09:19:09 INFO reduce.InMemoryMapOutput: Read 100 bytes from map-output for attempt_local291050806_0001_m_000002_0
19/09/21 09:19:09 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 100, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->100
19/09/21 09:19:09 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local291050806_0001_m_000001_0 decomp: 187 len: 191 to MEMORY
19/09/21 09:19:09 INFO reduce.InMemoryMapOutput: Read 187 bytes from map-output for attempt_local291050806_0001_m_000001_0
19/09/21 09:19:09 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 187, inMemoryMapOutputs.size() -> 2, commitMemory -> 100, usedMemory ->287
19/09/21 09:19:09 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local291050806_0001_m_000000_0 decomp: 207 len: 211 to MEMORY
19/09/21 09:19:09 INFO reduce.InMemoryMapOutput: Read 207 bytes from map-output for attempt_local291050806_0001_m_000000_0
19/09/21 09:19:09 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 207, inMemoryMapOutputs.size() -> 3, commitMemory -> 287, usedMemory ->494
19/09/21 09:19:09 INFO reduce.EventFetcher: EventFetcher is interrupted… Returning
19/09/21 09:19:09 INFO mapred.LocalJobRunner: 3 / 3 copied.
19/09/21 09:19:09 INFO reduce.MergeManagerImpl: finalMerge called with 3 in-memory map-outputs and 0 on-disk map-outputs
19/09/21 09:19:09 INFO mapred.Merger: Merging 3 sorted segments
19/09/21 09:19:09 INFO mapred.Merger: Down to the last merge-pass, with 3 segments left of total size: 467 bytes
19/09/21 09:19:09 INFO reduce.MergeManagerImpl: Merged 3 segments, 494 bytes to disk to satisfy reduce memory limit
19/09/21 09:19:09 INFO reduce.MergeManagerImpl: Merging 1 files, 494 bytes from disk
19/09/21 09:19:09 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
19/09/21 09:19:09 INFO mapred.Merger: Merging 1 sorted segments
19/09/21 09:19:09 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 479 bytes
19/09/21 09:19:09 INFO mapred.LocalJobRunner: 3 / 3 copied.
19/09/21 09:19:09 INFO Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
19/09/21 09:19:09 INFO mapred.Task: Task:attempt_local291050806_0001_r_000000_0 is done. And is in the process of committing
19/09/21 09:19:09 INFO mapred.LocalJobRunner: 3 / 3 copied.
19/09/21 09:19:09 INFO mapred.Task: Task attempt_local291050806_0001_r_000000_0 is allowed to commit now
19/09/21 09:19:09 INFO output.FileOutputCommitter: Saved output of task ‘attempt_local291050806_0001_r_000000_0’ to file:/D:/out/_temporary/0/task_local291050806_0001_r_000000
19/09/21 09:19:09 INFO mapred.LocalJobRunner: reduce > reduce
19/09/21 09:19:09 INFO mapred.Task: Task ‘attempt_local291050806_0001_r_000000_0’ done.
19/09/21 09:19:09 INFO mapred.LocalJobRunner: Finishing task: attempt_local291050806_0001_r_000000_0
19/09/21 09:19:09 INFO mapred.LocalJobRunner: reduce task executor complete.
19/09/21 09:19:10 INFO mapreduce.Job: map 100% reduce 100%
19/09/21 09:19:10 INFO mapreduce.Job: Job job_local291050806_0001 completed successfully
19/09/21 09:19:10 INFO mapreduce.Job: Counters: 35
File System Counters
FILE: Number of bytes read=4897
FILE: Number of bytes written=1165783
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=0
HDFS: Number of bytes written=0
HDFS: Number of read operations=0
HDFS: Number of large read operations=0
HDFS: Number of write operations=0
Map-Reduce Framework
Map input records=21
Map output records=39
Map output bytes=410
Map output materialized bytes=506
Input split bytes=279
Combine input records=0
Combine output records=0
Reduce input groups=18
Reduce shuffle bytes=506
Reduce input records=39
Reduce output records=18
Spilled Records=78
Shuffled Maps =3
Failed Shuffles=0
Merged Map outputs=3
GC time elapsed (ms)=0
Total committed heap usage (bytes)=1460142080
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=270
File Output Format Counters
Bytes Written=168

1.4 执行结果

datanode 1
flink 2
flume 2
hadoop 5
hbase 3
hdfs 1
hive 4
impala 1
kafka 2
mahout 1
mapreduce 1
namenode 2
pig 3
spark 4
sqoop 2
storm 1
yarn 1
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=270
File Output Format Counters
Bytes Written=168

1.4 执行结果

datanode 1
flink 2
flume 2
hadoop 5
hbase 3
hdfs 1
hive 4
impala 1
kafka 2
mahout 1
mapreduce 1
namenode 2
pig 3
spark 4
sqoop 2
storm 1
yarn 1
zookeeper 3

geekLinyi
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
在idea中编写wordcount程序
a1786742005的博客
02-25 4328
一、wordcount介绍 1、需求 统计给定文本文件中每一个单词出现的总次数，并输出。 2、输入数据 每个单词之间以"\t"分割 3、期望输出数据 hadoop 2 spark 1 sqoop 1 hbase 1 kafka 1 flume 1 mapreduce 1 二、wordcount 流程分析 按照 MapReduce 编程规范，我们需要分别编写 Mapper、Reducer、Driv...
如何理解wordcount
ITRunner123的博客
07-06 1178
1.
WordCount程序详解
MWhzx的专栏
11-01 7102
输入：hello hadoop hello word  下面是map函数 import java.io.IOException; import java.util.StringTokenizer; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.Text; import org.apache
理解WordCount
庐州小白的博客
10-12 2597
Apache Hadoop使用JAVA语言编程，对于使用java语言的程序员来说，这很友好。一般而言，一个hadoop程序主要分为两个部分：Map和Reduce。 我们可以观察一下其执行流： Map →\to→ Combiner →\to→ Sort →\to→ Partitioner →\to→ Reduce 下面我们以一个例子来阐述wordcount的执行流如何发生。 假设一个输入文本如下： ...
WordCount程序分析
tony的专栏
10-08 443
Hadoop的WordCount相当于java学习中的Hallo W
使用hadoop实现WordCount实验报告.docx
06-10
**WordCount应用** 是Hadoop平台上的一个基础示例，它统计输入文件中的每个单词出现的次数。例如，输入文件包含"Hello world"，则WordCount程序的输出应为"Hello 1"和"world 1"。 ### 二、实验原理与步骤 1. **...
Python入门Demo-WordCount
07-05
【Python入门Demo-WordCount】是一个适合初学者的Python编程示例，主要目的是介绍如何使用Python进行基础的数据处理，特别是文本分析中的词频统计。在这个Demo中，我们将学习到Python语言的一些基本语法、文件操作、...
大数据实验报告Windows环境下安装Spark及RDD编程和Spark编程实现wordcount.doc
12-13
《Windows环境下Spark...在实现WordCount的过程中，对RDD的各种操作有了直观的认识，提升了实际编程能力。同时，实验也暴露出版本匹配等潜在问题，提醒我们在今后的学习中要注重软件版本的适配，确保项目的稳定运行。
WordCount.rar_Java编程_Java_
08-12
【标题】"WordCount.rar" 是一个以Java编程语言实现的MapReduce项目，主要用于演示Hadoop框架中的WordCount...通过实践这个例子，可以让你对大数据处理有一个直观的认识，为进一步深入学习Hadoop和大数据分析打下基础。
浅谈Flink分布式运行时和数据流图的并行化
01-07
阅读完本文后，读者可以对Flink的分布式运行时有一个全面的认识。 1 Flink数据流图简介 1.1 Flink作业的逻辑视图 在大数据领域，词频统计（WordCount）程序就像是一个编程语言的HelloWorld程序，它展示了一个大数据...
wordcount运行分析
03-22
wordcount程序运行过程分析与应用。工作流程。。。。。。
（1）Hadoop 的第一个程序 WordCount 理解
BIG.KE的博客
02-24 96
Hadoop 的第一个程序 WordCount 理解 map and Reduce 相关概念 Map map 负责将自己区块数据， 做简单拆分， 成一个map， 这个map 是不去重的， 会在map 后面最加值， 让数据分组 比如两个 机器的两个map machine1： # 以下数据是machine1 hdfs 区块的数据 hello hello hello // 这是machine 1 的 context [ {"hello" : 1}, {"hello" : 1}, {"hello
打包编译hadoop之HelloWord（WordCount）
xuguokun1986的博客
10-07 2013
1、WordCount代码 package com.test; import java.io.IOException; import java.net.URI; import java.util.StringTokenizer; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileSyste
wordcount
weixin_43322061的博客
10-05 144
看了《hadoop应用开发技术详解》by刘刚中的入门wordcount 代码如下： mapper: reducer: wordmain: output为jar包传到$HADOOP_HOME下 jar wordcount.jar wordcount.wordmain /hellofile/hellofile* /hellofile/input 执行任务 遇到的问题： 1.java.io.IOExc...
【hadoop 学习笔记-1】wordcount 遇到的问题记录
金戈铁马
07-25 337
问题一： FileInputFormat.setInputPaths(xx,xx) FileOuputFormat.setOutputPath(xx, xx) 这两行报错，参数不匹配。 解决： 导入的类不对，IDEA默认导入的FileInputFormat和FileOuputFormat不对， 修改为 import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapred...
单纯linux系统下hadoop2.7.3 eclipse，记一次成功的运行wordcount的注意事项
weixin_34409703的博客
02-20 167
hadoop要正确安装好 hadoop eclipse plugin要对应相应的eclipse版本 define hadoop location mr master:9000 另一个9001  下面的是代理不用理会 bin/hdfs dfs -mkdir /user，在hdfs上创建user文件夹\ bin/hdfs dfs -put /usr/local/hadoop/READM...
实现wordcount功能
lizc_lizc的博客
08-25 646
新建一个maven工程 在pom.xml中添加相关依赖 &lt;?xml version="1.0" encoding="UTF-8"?&gt; &lt;project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:sche...
MapReduce中wordcount详细介绍
universe_ant的博客
09-22 9952
1、MapReduce编程模型 MapReduce采用分而治之的思想，把对大规模数据集的操作，分发给一个主节点管理下的各个分节点共同完成，然后通过整合各个节点的中间结果，得到最终结果。简单来说，MapReduce就是“任务的分解和结果的汇总”。 在Hadoop中，用于执行MapReduce任务的机器角色有两个：一个是JobTracker；另一个是TaskTracker。JobTracker用于
结对作业——WordCount进阶版
weixin_30871293的博客
10-08 141
结对作业——WordCount进阶版 标签（空格分隔）： 软件工程 Task1：Fork仓库的码云地址 博客作业要求地址：https://www.cnblogs.com/happyzm/p/9626779.html 结对项目码云地址：https://gitee.com/holmec/PairProject-Java/tree/master/ 结队伙伴：201621123033 姚雯婷 博客地址...
写一段wordcount的mapreduce
最新发布
05-31
好的，下面是一个简单的wordcount的mapreduce实现，您可以参考一下： Mapper代码： ``` public class WordCountMapper extends Mapper<LongWritable, Text, Text, IntWritable> { private final static IntWritable one = new IntWritable(1); private Text word = new Text(); public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { String line = value.toString(); StringTokenizer tokenizer = new StringTokenizer(line); while (tokenizer.hasMoreTokens()) { word.set(tokenizer.nextToken()); context.write(word, one); } } } ``` Reducer代码： ``` public class WordCountReducer extends Reducer<Text, IntWritable, Text, IntWritable> { public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException { int sum = 0; for (IntWritable val : values) { sum += val.get(); } context.write(key, new IntWritable(sum)); } } ``` Driver代码： ``` public class WordCount { public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); Job job = Job.getInstance(conf, "word count"); job.setJarByClass(WordCount.class); job.setMapperClass(WordCountMapper.class); job.setCombinerClass(WordCountReducer.class); job.setReducerClass(WordCountReducer.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); FileInputFormat.addInputPath(job, new Path(args[0])); FileOutputFormat.setOutputPath(job, new Path(args[1])); System.exit(job.waitForCompletion(true) ? 0 : 1); } } ``` 这个实现假设输入文件是一些文本文件，每个文件的每一行都是一些单词，输出是每个单词出现的次数。
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值