Hadoop MapReduce 任务发布

最新推荐文章于 2024-04-17 19:02:39 发布
最新推荐文章于 2024-04-17 19:02:39 发布
阅读量532 收藏 1
点赞数
分类专栏： Hadoop
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/F_TimeOk/article/details/80904127
版权

MapReduce 任务发布

预埋数据0.0

[root@CentOS ~]# vim words 
When I am down
You raise me up
I believe I can fly

上传到hdfs中

[root@CentOS ~]# hdfs dfs -mkdir /demo
[root@CentOS ~]# hdfs dfs -put words /demo/words

1）jar包发布

代码中添加

job.setJarByClass(CustomJobSubmitter.class);

具体如下

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;

import java.io.IOException;

public class CustomJobSubmitter {
    public static void main(String[] args) throws Exception {
        //1.创建job
        Configuration conf = new Configuration();
        Job job = Job.getInstance(conf);

        job.setJarByClass(CustomJobSubmitter.class);

        //2.设置数据的格式类型 决定读入和写出的数据的方式
        job.setInputFormatClass(TextInputFormat.class);
        job.setOutputFormatClass(TextOutputFormat.class);

        //3.设置数据的读入和写入路径
        Path src = new Path("/demo/words");
        Path res = new Path("/demo/res");

        FileSystem fileSystem = FileSystem.get(conf);

        if (fileSystem.exists(res)) {
            fileSystem.delete(res, true);
        }
        TextInputFormat.addInputPath(job, src);
        TextOutputFormat.setOutputPath(job, res);

        //4.设置数据的处理规则
        job.setMapperClass(WordsMapper.class);
        job.setReducerClass(WordsReducer.class);

        //5.设置Mapper和Reducer的key,value类型
        job.setMapOutputKeyClass(Text.class);
        job.setMapOutputValueClass(IntWritable.class);

        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);

        //6.提交任务job
        job.waitForCompletion(true);

    }

    public static class WordsMapper extends Mapper<LongWritable, Text, Text, IntWritable> {
        @Override
        protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
            String[] tokens = value.toString().split(" ");
            for (String token : tokens) {
                context.write(new Text(token), new IntWritable(1));
            }
        }
    }

    public static class WordsReducer extends Reducer<Text, IntWritable, Text, IntWritable> {
        @Override
        protected void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
            int total = 0;
            for (IntWritable value : values) {
                total += value.get();
            }
            context.write(new Text(key), new IntWritable(total));
        }
    }
}
使用maven打包项目

将构建成功生成的jar上传至linux系统中（这里使用lrzsz软件）

[root@CentOS ~]# rz

[root@CentOS ~]# hadoop jar job_submit-1.0-SNAPSHOT.jar com.fdh.submit.CustomJobSubmitter
18/05/25 22:33:33 INFO client.RMProxy: Connecting to ResourceManager at CentOS/192.168.199.128:8032
18/05/25 22:33:34 WARN mapreduce.JobSubmitter: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
18/05/25 22:33:34 INFO input.FileInputFormat: Total input paths to process : 1
18/05/25 22:33:34 INFO mapreduce.JobSubmitter: number of splits:1
18/05/25 22:33:35 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1527258142264_0002
18/05/25 22:33:36 INFO impl.YarnClientImpl: Submitted application application_1527258142264_0002
18/05/25 22:33:36 INFO mapreduce.Job: The url to track the job: http://CentOS:8088/proxy/application_1527258142264_0002/
18/05/25 22:33:36 INFO mapreduce.Job: Running job: job_1527258142264_0002
18/05/25 22:33:56 INFO mapreduce.Job: Job job_1527258142264_0002 running in uber mode : false
18/05/25 22:33:56 INFO mapreduce.Job:  map 0% reduce 0%
18/05/25 22:34:14 INFO mapreduce.Job:  map 100% reduce 0%
18/05/25 22:34:27 INFO mapreduce.Job:  map 100% reduce 100%
18/05/25 22:34:28 INFO mapreduce.Job: Job job_1527258142264_0002 completed successfully
18/05/25 22:34:29 INFO mapreduce.Job: Counters: 49
	File System Counters
		FILE: Number of bytes read=135
		FILE: Number of bytes written=212181
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=145
		HDFS: Number of bytes written=69
		HDFS: Number of read operations=6
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=2
	Job Counters 
		Launched map tasks=1
		Launched reduce tasks=1
		Data-local map tasks=1
		Total time spent by all maps in occupied slots (ms)=16336
		Total time spent by all reduces in occupied slots (ms)=8814
		Total time spent by all map tasks (ms)=16336
		Total time spent by all reduce tasks (ms)=8814
		Total vcore-seconds taken by all map tasks=16336
		Total vcore-seconds taken by all reduce tasks=8814
		Total megabyte-seconds taken by all map tasks=16728064
		Total megabyte-seconds taken by all reduce tasks=9025536
	Map-Reduce Framework
		Map input records=3
		Map output records=13
		Map output bytes=103
		Map output materialized bytes=135
		Input split bytes=94
		Combine input records=0
		Combine output records=0
		Reduce input groups=11
		Reduce shuffle bytes=135
		Reduce input records=13
		Reduce output records=11
		Spilled Records=26
		Shuffled Maps =1
		Failed Shuffles=0
		Merged Map outputs=1
		GC time elapsed (ms)=515
		CPU time spent (ms)=2280
		Physical memory (bytes) snapshot=263921664
		Virtual memory (bytes) snapshot=1687027712
		Total committed heap usage (bytes)=129011712
	Shuffle Errors
		BAD_ID=0
		CONNECTION=0
		IO_ERROR=0
		WRONG_LENGTH=0
		WRONG_MAP=0
		WRONG_REDUCE=0
	File Input Format Counters 
		Bytes Read=51
	File Output Format Counters 
		Bytes Written=69

查看map reduce计算结果

[root@CentOS ~]# hdfs dfs -cat /demo/res/part-r-00000
I	3
When	1
You	1
am	1
believe	1
can	1
down	1
fly	1
me	1
raise	1
up	1

进阶版

设置启动入口，这样当我们发布任务时只需要将jar上传到linux系统中即可

[root@CentOS ~]# cat Start.java 
public class Start {
    public static void main(String[] args) throws Exception{
        Runtime.getRuntime().exec("hadoop jar job_submit-1.0-SNAPSHOT.jar com.fdh.submit.CustomJobSubmitter");
    }
}

[root@CentOS ~]# javac Start.java 
[root@CentOS ~]# java Start

此时查看map reduce的结果时一样的

[root@CentOS ~]# hdfs dfs -cat /demo/res/part-r-00000
I	3
When	1
You	1
am	1
believe	1
can	1
down	1
fly	1
me	1
raise	1
up	1

2）跨平台发布任务

拷贝hadoop的四个配置文件到项目目录下

[root@CentOS ~]# sz /usr/hadoop-2.6.0/etc/hadoop/core-site.xml 
[root@CentOS ~]# sz /usr/hadoop-2.6.0/etc/hadoop/hdfs-site.xml 
[root@CentOS ~]# sz /usr/hadoop-2.6.0/etc/hadoop/mapred-site.xml
[root@CentOS ~]# sz /usr/hadoop-2.6.0/etc/hadoop/yarn-site.xml

修改mapred-site.xml添加如下配置

<property>
    <name>mapreduce.app-submission.cross-platform</name>
    <value>true</value>
</property>

使用maven的package插件打包，查看jar路径

添加配置信息后重新package

 

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;

import java.io.IOException;

public class CustomJobSubmitter {
    public static void main(String[] args) throws Exception {
        //1.创建job
        Configuration conf = new Configuration();

        conf.addResource("core-site.xml");
        conf.addResource("hdfs-site.xml");
        conf.addResource("mapred-site.xml");
        conf.addResource("yarn-site.xml");

        conf.set("mapreduce.job.jar", "file:///E:/ideaWorkspace/hadoop/job_submit/target/job_submit-1.0-SNAPSHOT.jar");

        Job job = Job.getInstance(conf);

        job.setJarByClass(CustomJobSubmitter.class);

        //2.设置数据的格式类型 决定读入和写出的数据的方式
        job.setInputFormatClass(TextInputFormat.class);
        job.setOutputFormatClass(TextOutputFormat.class);

        //3.设置数据的读入和写入路径
        Path src = new Path("hdfs://CentOS:9000/demo/words");
        Path res = new Path("hdfs://CentOS:9000/demo/res");

        FileSystem fileSystem = FileSystem.get(conf);

        Path res1 = new Path("/demo/res");
        if (fileSystem.exists(res1)) {
            fileSystem.delete(res1, true);
        }
        TextInputFormat.addInputPath(job, src);
        TextOutputFormat.setOutputPath(job, res);

        //4.设置数据的处理规则
        job.setMapperClass(WordsMapper.class);
        job.setReducerClass(WordsReducer.class);

        //5.设置Mapper和Reducer的key,value类型
        job.setMapOutputKeyClass(Text.class);
        job.setMapOutputValueClass(IntWritable.class);

        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);

        //6.提交任务job
        job.waitForCompletion(true);

    }

    public static class WordsMapper extends Mapper<LongWritable, Text, Text, IntWritable> {
        @Override
        protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
            String[] tokens = value.toString().split(" ");
            for (String token : tokens) {
                context.write(new Text(token), new IntWritable(1));
            }
        }
    }

    public static class WordsReducer extends Reducer<Text, IntWritable, Text, IntWritable> {
        @Override
        protected void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
            int total = 0;
            for (IntWritable value : values) {
                total += value.get();
            }
            context.write(new Text(key), new IntWritable(total));
        }
    }
}

3）本地仿真（不依赖真实的yarn集群，测试阶段使用）

目录结构

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;

import java.io.IOException;

public class CustomJobSubmitter {
    public static void main(String[] args) throws Exception {
        //1.创建job
        Configuration conf = new Configuration();

        Job job = Job.getInstance(conf);

        //2.设置数据的格式类型 决定读入和写出的数据的方式
        job.setInputFormatClass(TextInputFormat.class);
        job.setOutputFormatClass(TextOutputFormat.class);

        //3.设置数据的读入和写入路径
        Path src = new Path("file:///E:/hadoop_test/demo/words");
        Path res = new Path("file:///E:/hadoop_test/demo/res");

        FileSystem fileSystem = FileSystem.get(conf);

        if (fileSystem.exists(res)) {
            fileSystem.delete(res, true);
        }
        TextInputFormat.addInputPath(job, src);
        TextOutputFormat.setOutputPath(job, res);

        //4.设置数据的处理规则
        job.setMapperClass(WordsMapper.class);
        job.setReducerClass(WordsReducer.class);

        //5.设置Mapper和Reducer的key,value类型
        job.setMapOutputKeyClass(Text.class);
        job.setMapOutputValueClass(IntWritable.class);

        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);

        //6.提交任务job
        job.waitForCompletion(true);

    }

    public static class WordsMapper extends Mapper<LongWritable, Text, Text, IntWritable> {
        @Override
        protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
            String[] tokens = value.toString().split(" ");
            for (String token : tokens) {
                context.write(new Text(token), new IntWritable(1));
            }
        }
    }

    public static class WordsReducer extends Reducer<Text, IntWritable, Text, IntWritable> {
        @Override
        protected void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
            int total = 0;
            for (IntWritable value : values) {
                total += value.get();
            }
            context.write(new Text(key), new IntWritable(total));
        }
    }
}

出现以下异常

将错误的哪一行 修改为return true即可

别等时光非礼了梦想z
关注
专栏目录
Hadoop之MapReduce相关内容
派大星的博客
08-21 440
MapReduce
三十三、发布MapReduce程序在集群上运行之wordcount案例实施
09-17 534
环境准备： Hadoop2.6.0 IDEA maven3.5.4 MapReduce在本地已经可以正常运行，具体见： MapReduce之wordcount案例(环境搭建及案例实施)https://blog.csdn.net/m0_54925305/article/details/120155693 案例实施： 1.在原pom.xml依赖中添加集群依赖： <build> <plugins> <plugin>
五、MapReduce 发布服务
aijing7224的博客
01-04 142
是一个并行计算框架（计算的数据源比较广泛-HDFS、RDBMS、NoSQL），Hadoop的 MR模块充分利用了HDFS中所有数据节点（datanode）所在机器的内存、CUP以及少量磁盘完成对大数据集的分布式计算。MapReduce将计算分为两个阶段： 通过将一个大的计算任务分割成若干个小任务（计算目标数据集的分割），每一个小任务会分配给所有的计算节点（datanode所在物理机器...
hadoop 控制map任务数详解
spark的专栏
01-13 1074
但是通过这种方式设置map的个数，并不是每次都有效的。原因是mapred.map.tasks只是一个hadoop的参考数值，最终map的个数，还取决于其他的因素。 为了方便介绍，先来看几个名词： block_size : hdfs的文件块大小，默认为64M，可以通过参数dfs.block.size设置 total_size : 输入文件整体的大小 input_file_num : 输入文件的个数
Hadoop中Map任务的执行框架
xhh198781的专栏
12-14 4264
在前面的一片博文中，我重点讲述了Task被TaskTracker调度执行的原理及过程，但是在详细的介绍执行Task的过程细节之前，我想先来认真的讨论一下Map Task和Reduce Task的执行框架。当然本文主要集中在Map Task上，至于Reduce Task的相关内容，我会在下一篇博文中谈到。在这篇文章中，我将尽量给出一个最抽象的Map模型      在Hadoop的MapRduce中
Hadoop MapReduce实现tfidf源码
05-18
在大数据处理领域，Hadoop MapReduce是一种广泛应用的分布式计算框架，它使得在大规模数据集上进行并行计算成为可能。本篇文章将详细讲解如何利用Hadoop MapReduce实现TF-IDF（Term Frequency-Inverse Document ...
python hadoop mapreduce 相似用户|mapreduce.rar
12-01
在大数据处理领域，Python、Hadoop MapReduce是两个非常重要的工具。本文将深入探讨如何使用Python来编写Hadoop MapReduce程序，以实现微博关注者之间的相似用户分析。这个任务的关键在于理解并应用分布式计算原理，...
Hadoop MapReduce实战手册(完整版)
08-28
《Hadoop MapReduce实战手册》是一本专注于大数据处理技术的专著，主要针对Apache Hadoop中的MapReduce框架进行了深入的探讨。MapReduce是Hadoop生态系统中的核心组件之一，用于处理和生成大规模数据集。该书旨在...
Hadoop mapreduce实现wordcount
01-28
【标题】Hadoop MapReduce 实现 WordCount MapReduce 是 Apache Hadoop 的核心组件之一，它为大数据处理提供了一个分布式计算框架。WordCount 是 MapReduce 框架中经典的入门示例，它统计文本文件中每个单词出现的...
大数据 hadoop mapreduce 词频统计
10-23
【大数据Hadoop MapReduce词频统计】 大数据处理是现代信息技术领域的一个重要概念，它涉及到海量数据的存储、管理和分析。Hadoop是Apache软件基金会开发的一个开源框架，专门用于处理和存储大规模数据集。Hadoop的...
hadoop2任务提交过程
milyhj123的博客
06-22 765
1.在hadoop任意节点上通过 bin/hadoop jar命令开始任务；RunJar进程启动，相当于一个客户端client，计算输入分片。RunJar内置有一个Cluster对象，通它过可以向ResourceManager进行rpc通信； 2、客户端向ResourceManager申请作业ID，并且把作业资源文件包括MapReduce程序打包的jar文件、配置信息和客户端计算的输入划分信息存
Mapreduce中的MapTask工作机制（Hadoop）
最新发布
喻师傅的学习笔记
04-17 1515
MapTask工作机制
Strata+Hadoop World 北京大会日程发布
OReillyData
06-28 262
8月3-6号， Strata+ Hadoop World 北京大会----期待与您相聚北京。
hadoop 2.7.0版本发布
www.hohode.com
07-09 489
http://dongxicheng.org/mapreduce-nextgen/hadoop-2-7-0-release/
hadoop执行mapreduce任务，能够map，不能reduce，Shuffle阶段报错
Leo's Blog
05-15 7092
第一次运行： [root@sjfx jar]# hadoop jar /home/tangzw/jar/GameLoginLogAnalyzeA.jar /tangzw/input /tangzw/output 08/01/25 07:13:43 INFO client.RMProxy: Connecting to ResourceManager at sjfx/192.1
hadoop+eclipse运行设置报错
With_me
09-20 575
hadoop:2.6.4     eclipse:mars2    1、我想尝试一下zai hadoop上跑分布式缓存文件的测试代码。main方法设置如下（我有提前将hadoop上的主要配置文件hdfs-site.xml、core-site.xml等加入到classpath中，其实也可以直接在代码汇总设置conf.set(key,value)）： Configuration conf
Hadoop集群模式下运行Mapreduce任务
兮小熙的博客
04-19 4522
写了一个Hadoop权威指南中MapReduce处理天气数据的Demo一.MapReduce执行过程map前 map后 mapreduce流程图二.编写Mapper和Reducer类MaxTemperatureMapper MaxTemperatureReducer//mapper import java.io.IOException; import org.apache.hadoop.io.
在wind7中运行启动hadoop出现错误 java.lang.UnsatisfiedLinkError
热门推荐
w3045872817的博客
06-23 1万+
在wind7中运行启动hadoop出现错误： 2017-06-23 20:10:08,526 WARN [main] util.NativeCodeLoader (NativeCodeLoader.java:(62)) - Unable to load native-hadoop library for your platform... using builtin-java clas
Hadoop MapReduce实战指南
"Hadoop MapReduce Cookbook 是一本针对初学者的指南，通过实例解析大型和复杂数据集的Hadoop MapReduce分析方法。由Srinath Perera和Thilina Gunarathne撰写，由Packt Publishing出版。" 在大数据处理领域，Hadoop...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值