MapReduce--OutputFormat详解以及实现自定义OutputFormat

最新推荐文章于 2024-05-16 17:17:19 发布
最新推荐文章于 2024-05-16 17:17:19 发布
阅读量519 收藏
点赞数
分类专栏： Hadoop 文章标签： mapreduce hadoop java
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/qq_41301707/article/details/110857016
版权

      MapReduce--OutputFormat详解以及实现自定义OutputFormat

 

1. OutputFormat源代码解析

  • MapReuce OutputFormat 输出一般输出到：文件里面或者数据库中，今天就对常用的OutputFormat来分析一下
  • OutputFormat中的源代码
/** 
 * <code>OutputFormat</code> describes the output-specification for a 
 * Map-Reduce job.
 *
 * <p>The Map-Reduce framework relies on the <code>OutputFormat</code> of the
 * job to:<p>
 * <ol>
 *   <li>
 *   Validate the output-specification of the job. For e.g. check that the 
 *   output directory doesn't already exist. 
 *   <li>
 *   Provide the {@link RecordWriter} implementation to be used to write out
 *   the output files of the job. Output files are stored in a 
 *   {@link FileSystem}.
 *   </li>
 * </ol>
 * 
 * @see RecordWriter
 */
@InterfaceAudience.Public
@InterfaceStability.Stable
public abstract class OutputFormat<K, V> {

  /** 
   * Get the {@link RecordWriter} for the given task.
   *
   * @param context the information about the current task.
   * @return a {@link RecordWriter} to write the output for the job.
   * @throws IOException
   */
  public abstract RecordWriter<K, V> 
    getRecordWriter(TaskAttemptContext context
                    ) throws IOException, InterruptedException;

  /** 
   * Check for validity of the output-specification for the job.
   *  
   * <p>This is to validate the output specification for the job when it is
   * a job is submitted.  Typically checks that it does not already exist,
   * throwing an exception when it already exists, so that output is not
   * overwritten.</p>
   *
   * @param context information about the job
   * @throws IOException when output should not be attempted
   */
  public abstract void checkOutputSpecs(JobContext context
                                        ) throws IOException, 
                                                 InterruptedException;

  /**
   * Get the output committer for this output format. This is responsible
   * for ensuring the output is committed correctly.
   * @param context the task context
   * @return an output committer
   * @throws IOException
   * @throws InterruptedException
   */
  public abstract 
  OutputCommitter getOutputCommitter(TaskAttemptContext context
                                     ) throws IOException, InterruptedException;
}
  • 如需要实现OutputFormat则需要实现RecordWriter
  • OutputFormat 里面的checkOutputSpecs函数是检查输出路径，检查输出路径是否设置以及输出路径是否存在
  • RecordWriter abstract class
/**
 * <code>RecordWriter</code> writes the output &lt;key, value&gt; pairs 
 * to an output file.
 
 * <p><code>RecordWriter</code> implementations write the job outputs to the
 * {@link FileSystem}.
 * 
 * @see OutputFormat
 */
@InterfaceAudience.Public
@InterfaceStability.Stable
public abstract class RecordWriter<K, V> {
  /** 
   * Writes a key/value pair.
   *
   * @param key the key to write.
   * @param value the value to write.
   * @throws IOException
   */      
  public abstract void write(K key, V value
                             ) throws IOException, InterruptedException;

  /** 
   * Close this <code>RecordWriter</code> to future operations.
   * 
   * @param context the context of the task
   * @throws IOException
   */ 
  public abstract void close(TaskAttemptContext context
                             ) throws IOException, InterruptedException;
  • RecordWriter中只要实现write、close就可以实现自定义的RecordWriter
  • 下面就来操作试试，自定义实现输出为文件、数据库

 

2 自定义实现输出文件

2.1 需求

计算每个域名的访问量总和
1. 把www.baidu.com域名的MapReduce数据结果输出到www.baidu.com.log文件里面
2. 把www.qq.com域名的MapReduce数据结果输出到www.qq.com.log文件里面

2.2 数据

域名，访问次数
www.baidu.com,10
www.qq.com,9
www.baidu.com,7
www.qq.com,10
www.qq.com,23
www.baidu.com,6
www.qq.com,12
www.qq.com,24
www.baidu.com,9

2.3 Code

2.3.1 MyFileOutputFormatDriver Code

package com.xk.bigata.hadoop.mapreduce.outputformat;

import com.xk.bigata.hadoop.utils.FileUtils;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

import java.io.IOException;

public class MyFileOutputFormatDriver {

    public static void main(String[] args) throws Exception {

        String input = "mapreduce-basic/data/domain.data";
        String output = "mapreduce-basic/out";

        // 1 创建 MapReduce job
        Configuration conf = new Configuration();
        Job job = Job.getInstance(conf);

        // 删除输出路径
        FileUtils.deleteFile(job.getConfiguration(), output);

        // 2 设置运行主类
        job.setJarByClass(MyFileOutputFormatDriver.class);

        // 3 设置Map和Reduce运行的类
        job.setMapperClass(MyMapper.class);
        job.setReducerClass(MyReducer.class);

        // 4 设置Map 输出的 KEY 和 VALUE 数据类型
        job.setMapOutputKeyClass(Text.class);
        job.setMapOutputValueClass(IntWritable.class);

        // 5 设置Reduce 输出 KEY 和 VALUE 数据类型
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);

        // 6 设置输入和输出路径
        FileInputFormat.setInputPaths(job, new Path(input));
        FileOutputFormat.setOutputPath(job, new Path(output));

        // 设定之定义的output
        job.setOutputFormatClass(MyFileOutputFormat.class);

        // 7 提交job
        boolean result = job.waitForCompletion(true);
        System.exit(result ? 0 : 1);

    }

    public static class MyMapper extends Mapper<LongWritable, Text, Text, IntWritable> {

        @Override
        protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
            String[] spilts = value.toString().split(",");
            context.write(new Text(spilts[0]), new IntWritable(Integer.parseInt(spilts[1])));
        }
    }

    public static class MyReducer extends Reducer<Text, IntWritable, Text, IntWritable> {
        @Override
        protected void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
            int sum = 0;
            for (IntWritable value : values) {
                sum += value.get();
            }
            context.write(key, new IntWritable(sum));
        }
    }
}

2.3.2 MyFileOutputFormat Code

  • 由于写入的是文件，所以直接继承FileOutputFormat class就可以了，可以不用自己再写 checkOutputSpecs、getOutputCommitter这两个函数的实现了
package com.xk.bigata.hadoop.mapreduce.outputformat;

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.RecordWriter;
import org.apache.hadoop.mapreduce.TaskAttemptContext;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

import java.io.IOException;

public class MyFileOutputFormat extends FileOutputFormat<Text, IntWritable> {

    @Override
    public RecordWriter<Text, IntWritable> getRecordWriter(TaskAttemptContext job) throws IOException, InterruptedException {
        return new MyRecordWriter(job);
    }
}

2.3.3 MyRecordWriter Code

package com.xk.bigata.hadoop.mapreduce.outputformat;

import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IOUtils;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.RecordWriter;
import org.apache.hadoop.mapreduce.TaskAttemptContext;

import java.io.IOException;

public class MyRecordWriter extends RecordWriter<Text, IntWritable> {

    FileSystem fs = null;
    FSDataOutputStream baiduOut = null;
    FSDataOutputStream qqOut = null;

    public MyRecordWriter(TaskAttemptContext job) {
        try {
            fs = FileSystem.get(job.getConfiguration());
            baiduOut = fs.create(new Path("mapreduce-basic/out/www.baidu.com.log"));
            qqOut = fs.create(new Path("mapreduce-basic/out/www.qq.com.log"));
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

    @Override
    public void write(Text key, IntWritable value) throws IOException, InterruptedException {
        String domain = key.toString();
        if (domain.equals("www.baidu.com")) {
            baiduOut.write((key.toString() + "\t" + value.toString()).getBytes());
        } else if (domain.equals("www.qq.com")) {
            qqOut.write((key.toString() + "\t" + value.toString()).getBytes());
        }
    }

    @Override
    public void close(TaskAttemptContext context) throws IOException, InterruptedException {
        if (null != baiduOut) {
            IOUtils.closeStream(baiduOut);
        }
        if (null != qqOut) {
            IOUtils.closeStream(qqOut);
        }
        if (null != fs) {
            fs.close();
        }
    }
}

2.4 结果

2.4.1 www.baidu.com.log

www.baidu.com	32

2.4.2 www.qq.com.log

www.qq.com	78

MyFileOutputFormatDriver Code

3 DBOutputFormat

3.1 需求

使用MapReduce进行词频统计
把MapReduce 计算结果输出到MySQL里面

3.2 数据

hadoop,spark,flink
hbase,hadoop,spark,flink
spark
hadoop
hadoop,spark,flink
hbase,hadoop,spark,flink
spark
hadoop
hbase,hadoop,spark,flink

3.3 DDL

CREATE TABLE `wc` (
 `word` varchar(100) DEFAULT NULL,
 `cnt` int(11) DEFAULT NULL
);

3.4 Code

3.4.1 MysqlWordCountDoamin Code

package com.xk.bigata.hadoop.mapreduce.domain;

import org.apache.hadoop.io.Writable;
import org.apache.hadoop.mapreduce.lib.db.DBWritable;

import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;

public class MysqlWordCountDoamin implements Writable, DBWritable {

    private String word;

    private int cnt;

    public MysqlWordCountDoamin() {
    }

    public MysqlWordCountDoamin(String word, int cnt) {
        this.word = word;
        this.cnt = cnt;
    }

    @Override
    public String toString() {
        return "MysqlWordCountDoamin{" +
                "word='" + word + '\'' +
                ", cnt=" + cnt +
                '}';
    }

    public String getWord() {
        return word;
    }

    public void setWord(String word) {
        this.word = word;
    }

    public int getCnt() {
        return cnt;
    }

    public void setCnt(int cnt) {
        this.cnt = cnt;
    }

    @Override
    public void write(DataOutput out) throws IOException {
        out.writeUTF(word);
        out.writeInt(cnt);
    }

    @Override
    public void readFields(DataInput in) throws IOException {
        word = in.readUTF();
        cnt = in.readInt();
    }

    @Override
    public void write(PreparedStatement statement) throws SQLException {
        statement.setString(1, word);
        statement.setInt(2, cnt);
    }

    @Override
    public void readFields(ResultSet resultSet) throws SQLException {
        word = resultSet.getString(1);
        cnt = resultSet.getInt(2);
    }
}

3.4.2 MysqlDBOutputFormat Code

package com.xk.bigata.hadoop.mapreduce.outputformat;

import com.xk.bigata.hadoop.mapreduce.domain.MysqlWordCountDoamin;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.db.DBConfiguration;
import org.apache.hadoop.mapreduce.lib.db.DBOutputFormat;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;

import java.io.IOException;

public class MysqlDBOutputFormat {

    public static void main(String[] args) throws Exception {
        String input = "mapreduce-basic/data/wc.txt";

        // 1 创建 MapReduce job
        Configuration conf = new Configuration();
        // 设置JDBC连接
        DBConfiguration.configureDB(conf,
                "com.mysql.jdbc.Driver",
                "jdbc:mysql://bigdatatest01:3306/bigdata",
                "root",
                "Jgw@31500");
        Job job = Job.getInstance(conf);

        // 2 设置运行主类
        job.setJarByClass(MysqlDBOutputFormat.class);

        // 3 设置Map和Reduce运行的类
        job.setMapperClass(MyMapper.class);
        job.setReducerClass(MyReducer.class);

        // 4 设置Map 输出的 KEY 和 VALUE 数据类型
        job.setMapOutputKeyClass(Text.class);
        job.setMapOutputValueClass(IntWritable.class);

        // 5 设置Reduce 输出 KEY 和 VALUE 数据类型
        job.setOutputKeyClass(NullWritable.class);
        job.setOutputValueClass(MysqlWordCountDoamin.class);

        // 设置输出类
        job.setOutputFormatClass(DBOutputFormat.class);
        // 6 设置输入和输出路径
        FileInputFormat.setInputPaths(job, new Path(input));
        DBOutputFormat.setOutput(job, "wc", "word", "cnt");
        // 7 提交job
        boolean result = job.waitForCompletion(true);
        System.exit(result ? 0 : 1);
    }

    public static class MyMapper extends Mapper<LongWritable, Text, Text, IntWritable> {
        IntWritable ONE = new IntWritable(1);

        @Override
        protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
            String[] spilts = value.toString().split(",");
            for (String word : spilts) {
                context.write(new Text(word), ONE);
            }
        }
    }

    public static class MyReducer extends Reducer<Text, IntWritable, NullWritable, MysqlWordCountDoamin> {

        @Override
        protected void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
            int count = 0;
            for (IntWritable value : values) {
                count += value.get();
            }
            MysqlWordCountDoamin mysqlWordCountDoamin = new MysqlWordCountDoamin(key.toString(), count);
            context.write(NullWritable.get(), mysqlWordCountDoamin);
        }
    }
}

3.5 报错

java.lang.Exception: java.lang.ClassCastException: org.apache.hadoop.io.NullWritable cannot be cast to org.apache.hadoop.mapreduce.lib.db.DBWritable
	at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:489)
	at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:556)
Caused by: java.lang.ClassCastException: org.apache.hadoop.io.NullWritable cannot be cast to org.apache.hadoop.mapreduce.lib.db.DBWritable
	at org.apache.hadoop.mapreduce.lib.db.DBOutputFormat$DBRecordWriter.write(DBOutputFormat.java:66)
	at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:558)
	at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
	at org.apache.hadoop.mapreduce.lib.reduce.WrappedReducer$Context.write(WrappedReducer.java:105)
	at com.xk.bigata.hadoop.mapreduce.outputformat.MysqlDBOutputFormat$MyReducer.reduce(MysqlDBOutputFormat.java:81)
	at com.xk.bigata.hadoop.mapreduce.outputformat.MysqlDBOutputFormat$MyReducer.reduce(MysqlDBOutputFormat.java:72)
	at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
	at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)
	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
	at org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:346)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

解决方案

  • 是由于Reduce输出的Key设置成NullWritable，无法write数据
  • 只要把Reduce里面的key改成自定义的数据类型，value改成NullWritable
package com.xk.bigata.hadoop.mapreduce.outputformat;

import com.xk.bigata.hadoop.mapreduce.domain.MysqlWordCountDoamin;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.db.DBConfiguration;
import org.apache.hadoop.mapreduce.lib.db.DBOutputFormat;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;

import java.io.IOException;

public class MysqlDBOutputFormat {

    public static void main(String[] args) throws Exception {
        String input = "mapreduce-basic/data/wc.txt";

        // 1 创建 MapReduce job
        Configuration conf = new Configuration();
        // 设置JDBC连接
        DBConfiguration.configureDB(conf,
                "com.mysql.jdbc.Driver",
                "jdbc:mysql://bigdatatest01:3306/bigdata",
                "root",
                "Jgw@31500");
        Job job = Job.getInstance(conf);

        // 2 设置运行主类
        job.setJarByClass(MysqlDBOutputFormat.class);

        // 3 设置Map和Reduce运行的类
        job.setMapperClass(MyMapper.class);
        job.setReducerClass(MyReducer.class);

        // 4 设置Map 输出的 KEY 和 VALUE 数据类型
        job.setMapOutputKeyClass(Text.class);
        job.setMapOutputValueClass(IntWritable.class);

        // 5 设置Reduce 输出 KEY 和 VALUE 数据类型
        job.setOutputKeyClass(MysqlWordCountDoamin.class);
        job.setOutputValueClass(NullWritable.class);

        // 设置输出类
        job.setOutputFormatClass(DBOutputFormat.class);
        // 6 设置输入和输出路径
        FileInputFormat.setInputPaths(job, new Path(input));
        DBOutputFormat.setOutput(job, "wc", "word", "cnt");
        // 7 提交job
        boolean result = job.waitForCompletion(true);
        System.exit(result ? 0 : 1);
    }

    public static class MyMapper extends Mapper<LongWritable, Text, Text, IntWritable> {
        IntWritable ONE = new IntWritable(1);

        @Override
        protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
            String[] spilts = value.toString().split(",");
            for (String word : spilts) {
                context.write(new Text(word), ONE);
            }
        }
    }

    public static class MyReducer extends Reducer<Text, IntWritable, MysqlWordCountDoamin, NullWritable> {

        @Override
        protected void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
            int count = 0;
            for (IntWritable value : values) {
                count += value.get();
            }
            MysqlWordCountDoamin mysqlWordCountDoamin = new MysqlWordCountDoamin(key.toString(), count);
            context.write(mysqlWordCountDoamin, NullWritable.get());
        }
    }
}

3.6 结果

word,cnt
flink,5
hadoop,7
hbase,3
spark,7

MysqlDBOutputFormat Code

 

XK&RM
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
java 中自定义OutputFormat的实例详解
08-29
主要介绍了java 中 自定义OutputFormat的实例详解的相关资料,这里提供实例帮助大家学习理解这部分内容，希望通过本文能帮助到大家，需要的朋友可以参考下
用MapReduce实现KMeans算法
01-18
《使用MapReduce实现KMeans算法详解》 KMeans算法是一种广泛应用的无监督学习方法，用于数据聚类。在大数据处理的背景下，传统的单机实现往往无法应对海量数据，因此，结合分布式计算框架MapReduce实现KMeans算法就...
在MapReduce中使用压缩
andyguan01_2的博客
04-23 1508
要想压缩MapReduce作业的输出，应在作业配置过程中将mapreduce.output.fileoutputformat.compress属性设为true，将mapreduce.output.fileoutputformat.compress.codec属性设置为打算使用的压缩codec的类名。另一种方案是在FileOutputFormat中使用更便捷的方法设置这些属性，如以下范例，对查找最高...
自定义OutputFormat案例实操
最新发布
weixin_63257947的博客
05-16 715
我们可以看到这里的需求跟分区差不多，可能会想到用Partition分区去做，但是不行，在Hadoop的MapReduce框架中，OutputFormat接口和Partitioner类在数据处理和结果输出方面扮演着不同的角色。在这个实现中，我们可以根据某些条件（如键的特定值或值的特定属性）将数据写入不同的文件。过滤输入的log日志，包含atguigu的网站输出到e:/atguigu.log，不包含atguigu的网站输出到e:/other.log。
Java大数据学习09--Mapreduce数据压缩介绍和配置方法
stalin_的博客
06-03 480
一、概述： 数据压缩是mapreduce的一种优化策略：通过压缩编码对mapper或者reducer的输出进行压缩，以减少磁盘IO，提高MR程序运行速度（但相应增加了cpu运算负担） 二、基本原则： 运算密集型的job，少用压缩 IO密集型的job，多用压缩 注： 1、 Mapreduce支持将map输出的结果或者reduce输出的结果进行压缩，以减少网络IO或最终输出数据的体积 2、...
MapReduce快速入门系列(12) | MapReduce之OutputFormat
不温卜火
04-29 5354
前面我们讲解了MapTask,ReduceTask和MapReduce运行机制。，那么这篇文章博主继续为大家讲解OutputFormat数据输出。 目录一. OutputFormat接口实现类1.1 文本输出TextOutputFormat1.2 SequenceFileOutputFormat1.3 ==自定义OutputFormat==二. 自定义OutputFormat的使用场景和步骤2...
Hadoop_MapReduce_OutputFormat工作原理
热门推荐
BLU_111的博客
05-04 4万+
Hadoop_MapReduce_OutputFormat工作原理 OutputFormat 的作用： 校验job中指定输出路径是否存在 将结果写入输出文件 OutputFormat是一个抽象类，有以下几个子类： DBOutputFormat FileOutputFormat FilterOutputFormat NullOutputFormat FileOutputFormat有以下几个...
KNN mapreduce实现
01-12
《KNN MapReduce实现详解》 KNN，即K最近邻算法，是机器学习领域中最基础的分类与回归方法之一。它依据“物以类聚”的原则，通过寻找样本集中与待预测样本最接近的K个邻居，根据这些邻居的类别进行投票，从而决定待...
Hadoop-Programs:Hadoop-MapReduce
05-08
【Hadoop MapReduce详解】 Hadoop MapReduce是Apache Hadoop项目的核心组件之一，它是一种分布式计算框架，专门设计用于处理和存储海量数据。这个框架基于Google的MapReduce模型，允许在大规模集群上运行数据密集型...
hadoop MapReduce的输出压缩算法的设置（四种方法）
统木木的博客
09-04 2727
MapReduce的过程中，一个job结束之后，会输出处理完毕的数据集，也就是reduce的处理结果。我们可以将这些结果采用指定的压缩算法输出到指定的文件夹中。 map端数据的压缩：要求是reduce从map端进行数据拉取的时候，传输速度要快，此时适合选择snappy数据压缩算法。 reduce端数据压缩：reduce端数据最终输出到HDFS上进行数据存储，要求是数据的占用空间要小，所以，可以...
MapReduce作业提交源码分析
Java大数据
08-28 275
MapReduce原理图 job提交初步源码中主要方法提炼 boolean res = job.waitForCompletion(true); waitForCompletion{ submit(){ connect(); // 其实就是为了拿到一个cluster（Local YARN） submitJobInternal(){ ...
OutputFormat 源码详解
说文科技，做有态度的研究。
12-20 699
OutputFormat源码详解 1. 源码 package org.apache.hadoop.mapreduce; import java.io.IOException; import org.apache.hadoop.classification.InterfaceAudience; import org.apache.hadoop.classification.InterfaceSt...
hadoop-MapReduce-OutputFormat
liu_1221的博客
09-14 504
——尚硅谷课程笔记 OutputFormat接口实现类 OutputFormat是MapReduce输出的基类，所有实现MapReduce输出都实现了 OutputFormat接口。下面几种常见的OutputFormat实现类。 1．文本输出TextOutputFormat 默认的输出格式是TextOutputFormat，它把每条记录写为文本行。它的键和值可以是任意类型，...
Spark系列--OutputFormat 详解
大数据技术分享
08-23 1839
前言 本文主要内容 什么是OutputFormat及其运行机制？ 如何自定义自己的OutputFormat？ 实战自定义mysql OutputFormat。 一丶什么是OutputFormat？ 定义了 spark 的输出规则的类。这也许会让你想到 Hadoop Mapreduce 的 OutputFormat，没错，其实他们是一个东西，嗯，完全一样...
java outputformat_java 中自定义OutputFormat的实例详解
weixin_39605296的博客
02-16 344
java 中 自定义outputformat的实例详解实例代码：package com.ccse.hadoop.outputformat;import java.io.ioexception;import java.net.uri;import java.net.urisyntaxexception;import java.util.stringtokenizer;import org.apach...
Hadoop之OutputFormat数据输出详解
努力努力再努力L
06-10 1724
Hadoop之OutputFormat数据输出详解 目录 OutputFormat接口实现类 自定义OutputFormat 1. OutputFormat接口实现类 OutputFormat是MapReduce输出的基类，所有实现MapReduce输出都实现了 OutputFormat接口。下面我们介绍几种常见的OutputFormat实现类。 文本输出TextOutputForma...
Hadoop2.7.3 mapreduce（二）类型匹配异常解决方案及源码分析
李润泽的博客
07-16 1344
我们在运行mapreduce时，有时候会出现类型匹配异常。 java.lang.Exception: java.io.IOException: Type mismatch in key from map: expected org.apache.hadoop.io.LongWritable, received org.apache.hadoop.io.Text at org.apache.ha
十三、MapReduce中的OutputFormat
象在舞的技术专栏
08-25 941
既然有InputFormat，那么自然就会有OutputFormat，本文主要介绍MapReduce中的OutputFormat。关注专栏《from zero to hero（Hadoop篇）》查看相关系列的文章~ ...
自定义OutPutFormat
jackydai987的专栏
03-06 6619
昨天学习了自定义InputFormat,今天又去看了看TextOutFormat.java的源码。一样比较简单。现在我们来实现自定义的OutPutFormat先上代码：public class MyOutputFormat extends FileOutputFormat{ //可以看到这里继承了FileOutputFormat，这个类一样可以自己改写。 protected static class MyRecordWriter extends RecordWriter { //这里实现了自己的
hadoop-mapreduce-client-core
11-18
hadoop-mapreduce-client-core模块提供了与MapReduce相关的类和方法，方便开发者实现自定义的Map和Reduce任务。 具体来说，hadoop-mapreduce-client-core模块包含了以下重要组件和功能： 1. Job：Job表示一个...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值