hadoop的Context简单使用

最新推荐文章于 2021-03-19 16:22:27 发布
最新推荐文章于 2021-03-19 16:22:27 发布
阅读量5.9k 收藏 1
点赞数
分类专栏： Bigdatda-Hadoop 文章标签： hadoop的Context简单使用
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/baolibin528/article/details/51008115
版权
hadoop的Context简单使用


作业输入： 
hadoop@hadoop:/home/hadoop/blb$ hdfs dfs -text /user/hadoop/libin/input/inputpath1.txt
hadoop  a
spark   a
hive    a
hbase   a
tachyon a
storm   a
redis   a
hadoop@hadoop:/home/hadoop/blb$ hdfs dfs -text /user/hadoop/libin/input/inputpath2.txt
hadoop  b
spark   b
kafka   b
tachyon b
oozie   b
flume   b
sqoop   b
solr    b
hadoop@hadoop:/home/hadoop/blb$


代码： 
import java.io.IOException;
import java.net.URI;


import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.JobID;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.TaskAttemptID;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.FileSplit;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
import org.apache.hadoop.util.GenericOptionsParser;


public class GetIDMapReduce {
	public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException {
		Configuration conf = new Configuration();
		String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();
		if(otherArgs.length!=2){
			System.err.println("Usage databaseV1 <inputpath> <outputpath>");
		}


		Job job = Job.getInstance(conf, GetIDMapReduce.class.getSimpleName() + "1");
		job.setJarByClass(GetIDMapReduce.class);
		job.setMapOutputKeyClass(Text.class);
		job.setMapOutputValueClass(Text.class);
		job.setOutputKeyClass(Text.class);
		job.setOutputValueClass(NullWritable.class);
		job.setMapperClass(MyMapper1.class);
		job.setNumReduceTasks(0);
		job.setInputFormatClass(TextInputFormat.class);
		job.setOutputFormatClass(TextOutputFormat.class);
		FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
		FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
		job.waitForCompletion(true);
	}
	public static class MyMapper1 extends Mapper<LongWritable, Text, Text, NullWritable>{
		@Override
		protected void map(LongWritable key, Text value, Mapper<LongWritable, Text, Text, NullWritable>.Context context)
				throws IOException, InterruptedException {
		
			FileSplit fileSplit=(FileSplit) context.getInputSplit();
			String pathname=fileSplit.getPath().getName();
			JobID jobid=context.getJobID();	//获取jobid
			LongWritable keyout=context.getCurrentKey(); //获取key偏移量
			Text valueout=context.getCurrentValue();  //获取一行的值
			String jobname=context.getJobName();	//获得job名字
			TaskAttemptID taskid=context.getTaskAttemptID();	//获得taskid
			float progress=context.getProgress();	//获取任务执行进度
			String jar = context.getJar();	//作业运行之前，往集群拷贝的作业资源jar
			//String status = context.getStatus();
			String user = context.getUser(); //获取当前用户
			//String[] fileTimestamps = context.getFileTimestamps();
			int numReduceTasks = context.getNumReduceTasks(); //获得reduce的数量
			//Path[] fileClassPaths = context.getFileClassPaths();
			Configuration configuration = context.getConfiguration();	//获得作业配置文件
			//RawComparator<?> groupingComparator = context.getGroupingComparator();
			boolean jobSetupCleanupNeeded = context.getJobSetupCleanupNeeded(); //Get whether job-setup and job-cleanup is needed for the job 
			int maxMapAttempts = context.getMaxMapAttempts(); //the max number of attempts per map task
			int maxReduceAttempts = context.getMaxReduceAttempts(); //he max number of attempts per reduce task.
			//@SuppressWarnings("deprecation")
			//Path[] localCacheFiles = context.getLocalCacheFiles();
			//OutputCommitter outputCommitter = context.getOutputCommitter();
			Path workingDirectory = context.getWorkingDirectory(); //工作目录
			boolean nextKeyValue = context.nextKeyValue();	//下一个键值对
			//URI[] cacheFiles = context.getCacheFiles();
			URI[] cacheArchives = context.getCacheArchives(); //Get cache archives set in the Configuration
			Path[] archiveClassPaths = context.getArchiveClassPaths();//Get the archive entries in classpath as an array of Path
			boolean profileEnabled = context.getProfileEnabled();//Get whether the task profiling is enabled.
			//String profileParams = context.getProfileParams();
			@SuppressWarnings("deprecation")
			boolean symlink = context.getSymlink();// Originally intended to check if symlinks should be used, but currently symlinks cannot be disabled
			//RawComparator<?> sortComparator = context.getSortComparator();
			//int hashCode = context.hashCode();
			context.write(new Text("===================================================================================="), NullWritable.get());
			context.write(new Text("pathname--"+pathname), NullWritable.get());
			context.write(new Text("jobid--"+jobid.toString()), NullWritable.get());
			context.write(new Text("keyout--"+keyout.toString()), NullWritable.get());
			context.write(new Text("keyout--"+valueout), NullWritable.get());
			context.write(new Text("jobname--"+jobname), NullWritable.get());
			context.write(new Text("taskid--"+taskid.toString()), NullWritable.get());
			context.write(new Text("progress--"+progress), NullWritable.get());
			context.write(new Text("jar--"+jar.toString()), NullWritable.get());
			//context.write(new Text("status--"+status), NullWritable.get());
			context.write(new Text("user--"+user), NullWritable.get());
			//context.write(new Text("fileTimestamps--"+fileTimestamps), NullWritable.get());
			context.write(new Text("numReduceTasks--"+numReduceTasks), NullWritable.get());
			//context.write(new Text("fileClassPaths--"+fileClassPaths), NullWritable.get());
			context.write(new Text("configuration--"+configuration), NullWritable.get());
			//context.write(new Text("groupingComparator--"+groupingComparator), NullWritable.get());
			context.write(new Text("jobSetupCleanupNeeded--"+jobSetupCleanupNeeded), NullWritable.get());
			context.write(new Text("maxMapAttempts--"+maxMapAttempts), NullWritable.get());
			context.write(new Text("maxReduceAttempts--"+maxReduceAttempts), NullWritable.get());
			//context.write(new Text("localCacheFiles--"+localCacheFiles), NullWritable.get());
			//context.write(new Text("outputCommitter--"+outputCommitter), NullWritable.get());
			context.write(new Text("workingDirectory--"+workingDirectory), NullWritable.get());
			context.write(new Text("nextKeyValue--"+nextKeyValue), NullWritable.get());
			//context.write(new Text("cacheFiles--"+cacheFiles), NullWritable.get());
			context.write(new Text("cacheArchives--"+cacheArchives), NullWritable.get());
			context.write(new Text("archiveClassPaths--"+archiveClassPaths), NullWritable.get());
			context.write(new Text("profileEnabled--"+profileEnabled), NullWritable.get());
			//context.write(new Text("profileParams--"+profileParams), NullWritable.get());
			context.write(new Text("symlink--"+symlink), NullWritable.get());
			//context.write(new Text("sortComparator--"+sortComparator), NullWritable.get());
			//context.write(new Text("hashCode--"+hashCode), NullWritable.get());
		}
	}
}


输出结果： 
-rw-r--r--   2 hadoop hadoop          0 2016-03-29 17:38 /user/hadoop/libin/out1/_SUCCESS
-rw-r--r--   2 hadoop hadoop       3058 2016-03-29 17:38 /user/hadoop/libin/out1/part-m-00000
-rw-r--r--   2 hadoop hadoop       3045 2016-03-29 17:38 /user/hadoop/libin/out1/part-m-00001


part-m-00000文件结果： 
====================================================================================
pathname--inputpath2.txt
jobid--job_1446086163035_21583
keyout--9
keyout--spark   b
jobname--GetIDMapReduce1
taskid--attempt_1446086163035_21583_m_000000_0
progress--0.13636364
jar--/tmp/hadoop-yarn/staging/hadoop/.staging/job_1446086163035_21583/job.jar
user--hadoop
numReduceTasks--0
configuration--Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml, job.xml
jobSetupCleanupNeeded--true
maxMapAttempts--4
maxReduceAttempts--4
workingDirectory--hdfs://hadoop:9000/user/hadoop
nextKeyValue--true
cacheArchives--null
archiveClassPaths--null
profileEnabled--false
symlink--true
====================================================================================
pathname--inputpath2.txt
jobid--job_1446086163035_21583
keyout--25
keyout--tachyon b
jobname--GetIDMapReduce1
taskid--attempt_1446086163035_21583_m_000000_0
progress--0.37878788
jar--/tmp/hadoop-yarn/staging/hadoop/.staging/job_1446086163035_21583/job.jar
user--hadoop
numReduceTasks--0
configuration--Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml, job.xml
jobSetupCleanupNeeded--true
maxMapAttempts--4
maxReduceAttempts--4
workingDirectory--hdfs://hadoop:9000/user/hadoop
nextKeyValue--true
cacheArchives--null
archiveClassPaths--null
profileEnabled--false
symlink--true
====================================================================================
pathname--inputpath2.txt
jobid--job_1446086163035_21583
keyout--43
keyout--flume   b
jobname--GetIDMapReduce1
taskid--attempt_1446086163035_21583_m_000000_0
progress--0.6515151
jar--/tmp/hadoop-yarn/staging/hadoop/.staging/job_1446086163035_21583/job.jar
user--hadoop
numReduceTasks--0
configuration--Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml, job.xml
jobSetupCleanupNeeded--true
maxMapAttempts--4
maxReduceAttempts--4
workingDirectory--hdfs://hadoop:9000/user/hadoop
nextKeyValue--true
cacheArchives--null
archiveClassPaths--null
profileEnabled--false
symlink--true
====================================================================================
pathname--inputpath2.txt
jobid--job_1446086163035_21583
keyout--59
keyout--solr    b
jobname--GetIDMapReduce1
taskid--attempt_1446086163035_21583_m_000000_0
progress--0.8939394
jar--/tmp/hadoop-yarn/staging/hadoop/.staging/job_1446086163035_21583/job.jar
user--hadoop
numReduceTasks--0
configuration--Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml, job.xml
jobSetupCleanupNeeded--true
maxMapAttempts--4
maxReduceAttempts--4
workingDirectory--hdfs://hadoop:9000/user/hadoop
nextKeyValue--true
cacheArchives--null
archiveClassPaths--null
profileEnabled--false
symlink--true
hadoop@hadoop:/home/hadoop/blb$


part-m-00001文件结果： 
====================================================================================
pathname--inputpath1.txt
jobid--job_1446086163035_21583
keyout--9
keyout--spark   a
jobname--GetIDMapReduce1
taskid--attempt_1446086163035_21583_m_000001_0
progress--0.15517241
jar--/tmp/hadoop-yarn/staging/hadoop/.staging/job_1446086163035_21583/job.jar
user--hadoop
numReduceTasks--0
configuration--Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml, job.xml
jobSetupCleanupNeeded--true
maxMapAttempts--4
maxReduceAttempts--4
workingDirectory--hdfs://hadoop:9000/user/hadoop
nextKeyValue--true
cacheArchives--null
archiveClassPaths--null
profileEnabled--false
symlink--true
====================================================================================
pathname--inputpath1.txt
jobid--job_1446086163035_21583
keyout--24
keyout--hbase   a
jobname--GetIDMapReduce1
taskid--attempt_1446086163035_21583_m_000001_0
progress--0.41379312
jar--/tmp/hadoop-yarn/staging/hadoop/.staging/job_1446086163035_21583/job.jar
user--hadoop
numReduceTasks--0
configuration--Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml, job.xml
jobSetupCleanupNeeded--true
maxMapAttempts--4
maxReduceAttempts--4
workingDirectory--hdfs://hadoop:9000/user/hadoop
nextKeyValue--true
cacheArchives--null
archiveClassPaths--null
profileEnabled--false
symlink--true
====================================================================================
pathname--inputpath1.txt
jobid--job_1446086163035_21583
keyout--42
keyout--storm   a
jobname--GetIDMapReduce1
taskid--attempt_1446086163035_21583_m_000001_0
progress--0.7241379
jar--/tmp/hadoop-yarn/staging/hadoop/.staging/job_1446086163035_21583/job.jar
user--hadoop
numReduceTasks--0
configuration--Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml, job.xml
jobSetupCleanupNeeded--true
maxMapAttempts--4
maxReduceAttempts--4
workingDirectory--hdfs://hadoop:9000/user/hadoop
nextKeyValue--true
cacheArchives--null
archiveClassPaths--null
profileEnabled--false
symlink--true
====================================================================================
pathname--inputpath1.txt
jobid--job_1446086163035_21583
keyout--58
keyout--
jobname--GetIDMapReduce1
taskid--attempt_1446086163035_21583_m_000001_0
progress--1.0
jar--/tmp/hadoop-yarn/staging/hadoop/.staging/job_1446086163035_21583/job.jar
user--hadoop
numReduceTasks--0
configuration--Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml, job.xml
jobSetupCleanupNeeded--true
maxMapAttempts--4
maxReduceAttempts--4
workingDirectory--hdfs://hadoop:9000/user/hadoop
nextKeyValue--false
cacheArchives--null
archiveClassPaths--null
profileEnabled--false
symlink--true


礼彬fly
关注
专栏目录
java hadoop context_Java JavaSparkContext.hadoopConfiguration方法代碼示例
weixin_42443651的博客
03-04 457
import org.apache.spark.api.java.JavaSparkContext; //導入方法依賴的package包/類public synchronized void start() { // 加鎖,單線程執行String id = getID();if (id != null) {log.info("Starting Batch Layer {}", id);}stream...
【人工智能】Hadoop和Spark使用教程
最新发布
大雨淅淅的博客
07-29 914
Hadoop和Spark是大数据处理领域的两个重要工具，它们各自具有独特的功能和优势，经常一起使用以构建高效的数据处理和分析系统。以下是一个关于Hadoop和Spark使用的基本教程，旨在提供从安装、配置到基本应用开发的概览。
Hadoop源代码分析（IDs类和Context类）
06-09
Hadoop源代码中的IDs类和Context类进行分析
java hadoop context,Java HookContext类代码示例
weixin_35414437的博客
03-19 159
import org.apache.hadoop.hive.ql.hooks.HookContext; //导入依赖的package包/类@Overridepublic void run(final HookContext hookContext) throws Exception {// clone to avoid concurrent accesstry {final HiveEventCo...
hadoop 中context.collect 和 context.write的区别
junming4的专栏
10-10 3008
学hadoop的的时候很纠结，很多源码都是写 context.collect 而不是context.write 百度了一下，得出一下以下结论: 老版本是：output.collect(key, result); // output’s type is: OutputCollector 新版本是：context.write(key, result); // output’s type is: C
hadoop中新版本的Context
bangjin4719的博客
04-28 118
在hadoop实战和hadoop权威指南中都使用了老版本的任务控制对象JobConf，并且需要包mapred包以及子包，在新版本中使用mapreduce包，用context代替了OutputCollect<k,v>和report使用job和configuration来控制任务，删除了JobConf类型。新版本在使用老版本的map和reduce类型可能会因为参数类型出错。 ...
Hadoop集群简单使用操作
weixin_43858651的博客
01-01 399
HDFS使用 1、查看集群状态 命令： hdfs dfsadmin –report web控制台查看HDFS集群信息，浏览器打开http://hadoop-node-01:50070/ 2、上传文件到HDFS ⦁ 查看HDFS中的目录信息 命令： hadoop fs –ls / ⦁ 上传文件 命令： hadoop fs -put ./ scala.txt to / ...
使用命令行编译打包运行自己的MapReduce程序 Hadoop2.6.0
02-04
在Hadoop 2.6.0中，运行一个简单的MapReduce示例——WordCount，至少需要以下三个JAR文件： - `$HADOOP_HOME/share/hadoop/common/hadoop-common-2.6.0.jar` - `$HADOOP_HOME/share/hadoop/mapreduce/hadoop-...
JAVA使用Apache Hadoop实现大规模数据处理.txt
08-17
本文将详细介绍如何使用Java编程语言结合Apache Hadoop实现一个简单的单词计数程序，该程序能够处理大规模的数据集。 #### 二、Hadoop概述 Hadoop主要由两个部分组成： - **HDFS (Hadoop Distributed File System)*...
hadoop使用mapreduce统计词频_Hadoop基础-08-MapReduce词频统计
weixin_29122543的博客
01-17 835
定义Mapper实现WordCountMapper extends Mapperpublic class Mapper {......}KEYIN : mapping 输入 key 的类型，即每行的偏移量offset(每行第一个字符在整个文本中的位置)，Long 类型，对应 Hadoop 中的 LongWritable 类型；VALUEIN : mapping 输入 value 的类型, 即其实就...
Hadoop中的Context使用
wjlwangluo的博客
08-04 3744
简要截取： 本篇博客以经典的wordcount程序为例来说明context的用法： 直接上代码：package MapReduce; import java.io.IOException; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FSDataInputStream; impo
hadoop context.write() 以多个字段为key
热门推荐
maixia24的专栏
10-22 1万+
context.write(字段1, value); context.write(字段2, value); 这样输出就是以字段1和字段2为key的数据了
hadoop开发必读：认识Context类的作用
xyjikl
05-13 1243
问题导读： 1.Context能干什么？ 2.你对Context类了解多少？ 3.Context在mapreduce中的作用是什么？ 本文实在能够阅读源码的基础上一个继续，如果你还不能阅读源码，请参考 从零教你如何获取hadoop2.4源码并使用eclipse关联hadoop2.4源码 http://www.aboutyun.com/thread-8211-1-1.html
mapreduce中的context类
Chunhong Song的专栏
12-30 1万+
在Mapper中的map、以及Reducer中的reduce都有一个Context的类型  1 public void map(Object key, Textvalue, Context context)  2                 throwsOException,InterruptedException{  3    StringTokenizer itr = new Str
Hadoop 中context.collect 和 context.write的区别
孙琨SealSun
04-28 2524
Mark写MapReduce时，发现很多源码都是写 context.collect 而不是context.write，查阅资料，得到以下结论 老版本是：output.collect(key, result); // output’s type is: OutputCollector新版本是：context.write(key, result); // output’s type is: Conte
Hadoop环境编程-Configuration类的使用
cjf_wei的博客
10-29 2727
程序开发中，经常需要将程序执行中的相关参数进行可配置化，以实现程序的灵活性。在Hadoop环境下编程，也有同样的需求。本文介绍在MapReduce编程中，怎样使用Configuration类读取相关配置。这些配置可能不仅仅在Job配置时需要，有些配置还要在Map或者Reduce编程间传递。
context.write
hechenghai
06-16 8311
今天因为Context.write的一个问题调试了半天， 对于hadoop的context.write来讲，必须保证有值写入，不能是null，如果是null会让map输出和reduce的输入不匹配而造成程序运行错误。
2016/09/21 context.getConfiguration().get()
weixin_30347335的博客
09-21 692
查看api:http://hadoop.apache.org/docs/stable/api/ public String get(String name) Get the value of the name property, null if no such property exists. If the key is deprecated, it returns the value of ...
Hadoop 全局变量与数据传递
05-25 9166
首先明确：  1.Hadoop不支持全局变量，也不建议使用全局变量。    我的理解是，这是因为hadoop具有map类和reducer类，并且不同的task一般执行的是不同的map或reduce。所以全局变量是无法传递的。（但是一般情况下，我们也许会需要一个对于所有map和reduce都能访问的全局变量），暂时我知道的解决方法如下： 2.如果Mapper类和Reducer类都是主类的内
hadoop如何使用Java api编程
03-29
Hadoop是一个分布式处理框架，它提供了Java API来编写分布式程序。下面是使用Java API编程的步骤： 1. 引入Hadoop依赖 在Java项目中，需要引入Hadoop的依赖。可以使用Maven或手动导入jar包。 2. 配置Hadoop环境 ...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值