MapReduce hdfs 写入 hbase 表

最新推荐文章于 2021-08-12 21:35:17 发布
最新推荐文章于 2021-08-12 21:35:17 发布
阅读量266 收藏
点赞数
分类专栏： MapReduce hbase
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/servletwjx/article/details/88847412
版权

文件内容如下：

入hbase 表 

依赖jar 包

<?xml version="1.0" encoding="UTF-8"?>

<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>sitech-e3base</groupId>
    <artifactId>sitech-e3base</artifactId>
    <packaging>pom</packaging>
    <version>1.0-cdh5</version>
    <modules>

    </modules>


    <!--<inceptionYear>2014</inceptionYear>-->
    <!--<modules>-->
    <!--<module>hbase</module>-->
    <!--<module>hive</module>-->
    <!--<module>oozie</module>-->
    <!--<module>storm</module>-->
    <!--<module>spark</module>-->
    <!--<module>mapreduce</module>-->
    <!--<module>kafka</module>-->
    <!--<module>impala</module>-->
    <!--<modupateoracle.testtest</module>-->
    <!--<module>hiho</module>-->
    <!--<module>cb2.0</module>-->
    <!--<module>mysql-hbase</module>-->
    <!--</modules>-->

    <properties>
        <hadoop.version>2.6.0-cdh5.5.0</hadoop.version>
        <hbase.version>1.0.0-cdh5.5.0</hbase.version>
        <hive.version>1.1.0-cdh5.5.0</hive.version>
        <spark.version>1.5.0-cdh5.5.0</spark.version>
        <oozie.version>4.1.0-cdh5.5.0</oozie.version>
        <pig.version>0.12.0-cdh5.5.0</pig.version>

        <storm.version>0.9.3</storm.version>

        <scala.version>2.10.4</scala.version>
        <scala.binary.version>2.10</scala.binary.version>
        <mockito.version>1.8.1</mockito.version>
        <mysql.connector.version>5.1.13</mysql.connector.version>
        <jackson.version>1.4.0</jackson.version>
        <commons-net.version>1.4.1</commons-net.version>
        <salesforce.version>27.0.0</salesforce.version>
        <salesforce.partner.version>27.0.0</salesforce.partner.version>
        <junit.version>4.5</junit.version>
        <avro.version>1.4.1</avro.version>

    </properties>

    <repositories>

        <repository>
            <id>scala-tools.org</id>
            <name>Scala-Tools Maven2 Repository</name>
            <url>http://scala-tools.org/repo-releases</url>
        </repository>

        <repository>
            <id>cloudera</id>
            <url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
        </repository>
    </repositories>


    <pluginRepositories>
        <pluginRepository>
            <id>scala-tools.org</id>
            <name>Scala-Tools Maven2 Repository</name>
            <url>http://scala-tools.org/repo-releases</url>
        </pluginRepository>
    </pluginRepositories>

    <dependencies>
        <!--  kudu  -->
        <dependency>
            <groupId>org.apache.kudu</groupId>
            <artifactId>kudu-client</artifactId>
            <version>1.6.0</version>
        </dependency>

        <!--  hdfs输出格式parquent  -->
        <dependency>
            <groupId>org.apache.parquet</groupId>
            <artifactId>parquet-common</artifactId>
            <version>1.8.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.parquet</groupId>
            <artifactId>parquet-encoding</artifactId>
            <version>1.8.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.parquet</groupId>
            <artifactId>parquet-column</artifactId>
            <version>1.8.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.parquet</groupId>
            <artifactId>parquet-hadoop</artifactId>
            <version>1.8.1</version>
        </dependency>


        <dependency>
            <groupId>org.apache.thrift</groupId>
            <artifactId>libthrift</artifactId>
            <version>0.9.0</version>
            <type>pom</type>
        </dependency>

        <!-- https://mvnrepository.com/artifact/com.jcraft/jsch -->
        <dependency>
            <groupId>com.jcraft</groupId>
            <artifactId>jsch</artifactId>
            <version>0.1.51</version>
        </dependency>

        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>${scala.version}</version>
        </dependency>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>3.8.1</version>
            <scope>test</scope>
        </dependency>


        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>${hadoop.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-hdfs</artifactId>
            <version>${hadoop.version}</version>
        </dependency>


        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-yarn-common</artifactId>
            <version>${hadoop.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-yarn-client</artifactId>
            <version>${hadoop.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-yarn-server-resourcemanager</artifactId>
            <version>${hadoop.version}</version>
        </dependency>


        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-mapreduce-client-jobclient</artifactId>
            <version>${hadoop.version}</version>
        </dependency>

        <dependency>
            <groupId>com.hadoop.gplcompression</groupId>
            <artifactId>hadoop-lzo</artifactId>
            <version>cdh4-0.4.15-gplextras</version>
        </dependency>


        <dependency>
            <groupId>org.apache.hbase</groupId>
            <artifactId>hbase-common</artifactId>
            <version>${hbase.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.hbase</groupId>
            <artifactId>hbase-client</artifactId>
            <version>${hbase.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.hbase</groupId>
            <artifactId>hbase-protocol</artifactId>
            <version>${hbase.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hbase</groupId>
            <artifactId>hbase-server</artifactId>
            <version>${hbase.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hbase</groupId>
            <artifactId>hbase-thrift</artifactId>
            <version>${hbase.version}</version>
        </dependency>


        <dependency>
            <groupId>org.apache.hive</groupId>
            <artifactId>hive-common</artifactId>
            <version>${hive.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hive</groupId>
            <artifactId>hive-metastore</artifactId>
            <version>${hive.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hive</groupId>
            <artifactId>hive-serde</artifactId>
            <version>${hive.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hive</groupId>
            <artifactId>hive-service</artifactId>
            <version>${hive.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hive</groupId>
            <artifactId>hive-exec</artifactId>
            <version>${hive.version}</version>
        </dependency>


        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_${scala.binary.version}</artifactId>
            <version>${spark.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-hive_${scala.binary.version}</artifactId>
            <version>${spark.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming_${scala.binary.version}</artifactId>
            <version>${spark.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming-twitter_${scala.binary.version}</artifactId>
            <version>${spark.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming-kafka_${scala.binary.version}</artifactId>
            <version>${spark.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming-flume_${scala.binary.version}</artifactId>
            <version>${spark.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming-zeromq_${scala.binary.version}</artifactId>
            <version>${spark.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming-mqtt_${scala.binary.version}</artifactId>
            <version>${spark.version}</version>
        </dependency>

        <dependency>
            <groupId>ch.ethz.ganymed</groupId>
            <artifactId>ganymed-ssh2</artifactId>
            <version>build210</version>
        </dependency>
        <!--
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-hive-thriftserver_${scala.binary.version}</artifactId>
            <version>${spark.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-mllib_${scala.binary.version}</artifactId>
            <version>${spark.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-bagel_${scala.binary.version}</artifactId>
            <version>${spark.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-yarn_${scala.binary.version}</artifactId>
            <version>${spark.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-graphx_${scala.binary.version}</artifactId>
            <version>${spark.version}</version>
        </dependency>
        -->

        <dependency>
            <groupId>org.apache.storm</groupId>
            <artifactId>storm-core</artifactId>
            <version>${storm.version}</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.storm</groupId>
            <artifactId>storm-hdfs</artifactId>
            <version>${storm.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.storm</groupId>
            <artifactId>storm-hbase</artifactId>
            <version>${storm.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.storm</groupId>
            <artifactId>storm-kafka</artifactId>
            <version>${storm.version}</version>
        </dependency>

        <dependency>
            <groupId>com.101tec</groupId>
            <artifactId>zkclient</artifactId>
            <version>0.10</version>
        </dependency>

        <dependency>
            <!--20180717测试kafka修改-->
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka_2.9.2</artifactId>
            <version>0.8.1.1</version>
            <!--<groupId>org.apache.kafka</groupId>-->
            <!--<artifactId>kafka_2.12</artifactId>-->
            <!--<version>1.0.0</version>-->
            <exclusions>
                <exclusion>
                    <groupId>org.apache.zookeeper</groupId>
                    <artifactId>zookeeper</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>log4j</groupId>
                    <artifactId>log4j</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>org.apache.oozie</groupId>
            <artifactId>oozie-client</artifactId>
            <version>${oozie.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.oozie</groupId>
            <artifactId>oozie-core</artifactId>
            <version>${oozie.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.oozie</groupId>
            <artifactId>oozie-core</artifactId>
            <classifier>tests</classifier>
            <version>${oozie.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.oozie</groupId>
            <artifactId>oozie-examples</artifactId>
            <version>${oozie.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.oozie</groupId>
            <artifactId>oozie-sharelib-streaming</artifactId>
            <version>${oozie.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.oozie</groupId>
            <artifactId>oozie-sharelib-pig</artifactId>
            <version>${oozie.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.oozie</groupId>
            <artifactId>oozie-sharelib-hive</artifactId>
            <version>${oozie.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.oozie</groupId>
            <artifactId>oozie-sharelib-sqoop</artifactId>
            <version>${oozie.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.oozie</groupId>
            <artifactId>oozie-sharelib-oozie</artifactId>
            <version>${oozie.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.oozie</groupId>
            <artifactId>oozie-sharelib-distcp</artifactId>
            <version>${oozie.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.oozie</groupId>
            <artifactId>oozie-sharelib-hcatalog</artifactId>
            <version>${oozie.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.oozie</groupId>
            <artifactId>oozie-sharelib-hive2</artifactId>
            <version>${oozie.version}</version>
        </dependency>


        <dependency>
            <groupId>org.jdom</groupId>
            <artifactId>jdom</artifactId>
            <version>1.1</version>
        </dependency>

        <!-- <dependency>
             <groupId>HadoopAuth</groupId>
             <artifactId>HadoopAuth</artifactId>
             <version>1.0-cdh5.0.0</version>
         </dependency>    -->


        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>${junit.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.mockito</groupId>
            <artifactId>mockito-all</artifactId>
            <version>${mockito.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.mockito</groupId>
            <artifactId>mockito-core</artifactId>
            <version>${mockito.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.pig</groupId>
            <artifactId>pig</artifactId>
            <version>${pig.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hive</groupId>
            <artifactId>hive-common</artifactId>
            <version>${hive.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hive</groupId>
            <artifactId>hive-serde</artifactId>
            <version>${hive.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.avro</groupId>
            <artifactId>avro</artifactId>
            <version>${avro.version}</version>
        </dependency>
        <dependency>
            <groupId>com.force.api</groupId>
            <artifactId>force-wsc</artifactId>
            <version>${salesforce.version}</version>
        </dependency>
        <dependency>
            <groupId>com.force.api</groupId>
            <artifactId>force-partner-api</artifactId>
            <version>${salesforce.partner.version}</version>
        </dependency>
        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <version>${mysql.connector.version}</version>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-jar-plugin</artifactId>
                <configuration>
                    <archive>
                        <manifest>
                            <mainClass>ExampleDriver</mainClass>
                        </manifest>
                    </archive>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>1.7</source>
                    <target>1.7</target>
                </configuration>
            </plugin>

            <!--
            <sourceDirectory>src/main/java</sourceDirectory>
            <plugin>
                <groupId>org.scala-tools</groupId>
                <artifactId>maven-scala-plugin</artifactId>
                <executions>
                    <execution>
                        <goals>
                            <goal>compile</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <scalaVersion>${scala.version}</scalaVersion>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-eclipse-plugin</artifactId>
                <configuration>
                    <downloadSources>true</downloadSources>
                    <buildcommands>
                        <buildcommand>ch.epfl.lamp.sdt.core.scalabuilder</buildcommand>
                    </buildcommands>
                    <additionalProjectnatures>
                        <projectnature>ch.epfl.lamp.sdt.core.scalanature</projectnature>
                    </additionalProjectnatures>
                    <classpathContainers>
                        <classpathContainer>org.eclipse.jdt.launching.JRE_CONTAINER</classpathContainer>
                        <classpathContainer>ch.epfl.lamp.sdt.launching.SCALA_CONTAINER</classpathContainer>
                    </classpathContainers>
                </configuration>
            </plugin>
            -->
        </plugins>
    </build>

    <!--
    <reporting>
        <plugins>
            <plugin>
                <groupId>org.scala-tools</groupId>
                <artifactId>maven-scala-plugin</artifactId>
                <configuration>
                    <scalaVersion>${scala.version}</scalaVersion>
                </configuration>
            </plugin>
        </plugins>
    </reporting>
        -->
</project>

具体代码：

package sitech;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.client.Scan;
import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
import org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil;
import org.apache.hadoop.hbase.mapreduce.TableOutputFormat;
import org.apache.hadoop.hbase.mapreduce.TableReducer;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.GenericOptionsParser;

import java.io.IOException;

public class HdfsToHbase {
    private static class HdfsToHbaseMapper extends Mapper<LongWritable, Text, Text, Put>{
        @SuppressWarnings("deprecation")
        //
        //ImmutableBytesWritable rowkey = new ImmutableBytesWritable();
        @Override
        protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
           String [] values=value.toString().split(",");
            String rowkey=values[0];
            String name = values[1];
            String age = values[2];
            Put put = new Put(Bytes.toBytes(rowkey));
            put.add(Bytes.toBytes("info"), Bytes.toBytes("name"), Bytes.toBytes(name));
            put.add(Bytes.toBytes("info"), Bytes.toBytes("age"), Bytes.toBytes(age));
            context.write(new Text(rowkey), put);
        }
    }

    public static  class HdfsToHbaseReducer extends TableReducer<Text, Put, NullWritable> {

        @Override
        protected void reduce(Text key, Iterable<Put> values, Context context) throws IOException, InterruptedException {
            //读出来的每一行数据写入到 fruit_hdfs 表中
            for (Put put : values) {
                context.write(NullWritable.get(), put);
            }
        }
    }

    public static Job createSubmittableJob(Configuration conf, String[] args) throws IOException {
        String hbase_table_name = args[0];
        String inPath = args[1];
        conf.set("hbase_table_name", hbase_table_name);
        conf.set("inPath", inPath);
        Job job = Job.getInstance(conf, HdfsToHbase.class.getSimpleName());

        TableMapReduceUtil.initTableReducerJob(
                hbase_table_name,        // output table
                HdfsToHbaseReducer.class,    // reducer class
                job);
        job.setNumReduceTasks(0);
        job.setReducerClass(HdfsToHbaseReducer.class);
        job.setJarByClass(HdfsToHbase.class);
        job.setMapperClass(HdfsToHbaseMapper.class);
        job.setMapOutputKeyClass(Text.class);
        job.setMapOutputValueClass(Put.class);

        FileInputFormat.addInputPath(job, new Path(args[1]));
        job.setOutputFormatClass(TableOutputFormat.class);


        return job;
    }

    public static void main(String[] args) throws InterruptedException, IOException, ClassNotFoundException {
        Configuration conf = HBaseConfiguration.create();
        //设置压缩格式
        conf.set("mapreduce.output.fileoutputformat.compress", "false");
        //设置内存
        conf.set("mapreduce.map.memory.mb", "2048");
        //
        //conf.set("hbase.zookeeper.property.clientPort", "11001");//TODO zk client端口   9501
        //conf.set("hbase.zookeeper.quorum", "");//TODO zk 服务器  
        //conf.set("hbase.zookeeper.quorum", "");
        //生产环境
        conf.set("hbase.zookeeper.property.clientPort", "9501");
        conf.set("hbase.zookeeper.quorum", "jfhadoop005,jfhadoop006,jfhadoop007");

        String[] otherArgs = new GenericOptionsParser(conf, args)
                .getRemainingArgs();
        if (otherArgs.length != 2) {
            System.exit(-1);
        }
        Job job = createSubmittableJob(conf, otherArgs);
        System.exit(job.waitForCompletion(true) ? 0 : 1);
    }
}

创建hbase 表

create 'myinfo','info'

查看数据

成功！

 

 

小小聪
关注
专栏目录
mapreduce HDFS 写入hbase 表中
servletwjx的博客
03-29 231
import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.hbase.HBaseConfiguration; import org.apache.hadoop.hbase.client.Put; import org.apache.hadoop....
java hdfs导入hbase_MapReduce将HDFS文本数据导入HBase中
weixin_39569051的博客
02-26 565
HBase本身提供了很多种数据导入的方式，通常有两种常用方式：使用HBase提供的TableOutputFormat，原理是通过一个Mapreduce作业将数据导入HBase另一种方式就是使用HBase原生Client API本文就是示范如何通过MapReduce作业从一个文件读取数据并写入到HBase中。首先启动Hadoop与HBase，然后创建一个空表，用于后面导入数据：hbase(main)...
MapReduce-读取文件写入HBase
weixin_30846599的博客
09-17 207
MapReduce直接写入HBase 代码如下 package com.hbase.mapreduce; import java.io.IOException; import org.apache.commons.cli.CommandLine; import org.apache.commons.cli.CommandLineParser; import org.apac...
HDFS 通过mapreduce 进行 HBase 导入导出
08-07
NULL 博文链接：https://jsh0401.iteye.com/blog/2096103
Hadoop MapReduce统计结果直接输出hbase
maoxiao_jsd的专栏
01-02 1533
MapReduce统计结果直接输出hbase，我使用的是hadoop1.0.4版本和hbase 0.94版本，hadoop和hbase安装伪分布式。1.hadoop安装这里就不讲了。 2.hbase安装我这里将一下。 首页解压habase安装包到/home/hadoop目录。 配置hosts文件如下： 192.168.0.101   hadoop.master 复制代
利用MapReduce将文件内容写入Hbase表
xiushuiguande的博客
03-09 4422
HBase – Hadoop Database，是一个高可靠性、高性能、面向列、可伸缩的分布式存储系统，利用HBase技术可在廉价PC Server上搭建起大规模结构化存储集群。 HBase是Google Bigtable的开源实现，类似Google Bigtable利用GFS作为其文件存储系统，HBase利用Hadoop HDFS作为其文件存储系统；Google运行MapReduce来处理Bi...
MapReduce hdfs文件写入hbase表
servletwjx的博客
03-29 289
@[TOMapReduce hdfs文件写入hbase表 import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.hbase.HBaseConfiguration; import org.apache.hadoop.hbase.client.Put...
MapReduce实现将HDFS中的数据解析并写入到HBase表中
weixin_43230682的博客
07-23 1569
目录 一、准备工作 二、功能实现 1、创建Maven工程并导入依赖 2、编写自定义Mapper类 3、编写自定义Reducer类 4、编写自定义Runner类 5、运行代码并查看结果 在上一篇博客中实现了HBase表之间的数据迁移功能： HBase表之间的数据迁移（使用MapReduce实现） 那么，本文继续结合小案例介绍如何使用MapReduce将HDFS中的数据写入到HBase表中。 功能实现： 一、准备工作 1）启动好Zookeeper、Hadoop、HBase服务； [
【HBase基础教程】6、HBase之读取MapReduce数据写入HBase
andieguo的专栏
03-05 8715
本blog将介绍利用MapReduce操作HBase，借助最熟悉的单词计数案例WordCount，将WordCount的统计结果存储到HBase，而不是HDFS。开发环境 硬件环境：Centos 6.5 服务器4台（一台为Master节点，三台为Slave节点） 软件环境：Java 1.7.0_45、Eclipse Juno Service Release 2、hadoop-1.2.1、hbase-
MapReduce中，从HDFS读取数据计算后写入HBase
sean-xiao的专栏
02-26 3738
基于上个例子。做一下简单的改造。http://blog.csdn.net/demonxyy/article/details/79320628在原本的例子中，从HDFS中读取数据计算之后再写会HDFS里，现在讲Reducer类改造一下，把计算后的数据。写入到HBase当中，写完之后我们会使用HBase的命令查询一下写入数据。打开原有的Reducer类，代码如下：import org.apache.h...
mapReduce 将csv文件从本地或者hdfs 导入 hbase表
pot成长之路
01-23 2607
import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.hbase.HBaseConfiguration; import org.apache.hadoop.hbase.client.Put; import org.apache.hadoop.hbase.mapreduce.TableOutputFormat...
parquet 形式MapReduce hbase 数据写入hdfs
servletwjx的博客
05-28 512
package com.sitech; import com.google.common.collect.Lists; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; import org.apache.h...
HBase 与 MapReduce（HDFS ＜==＞ HBase）
weixin_44387652的博客
03-16 771
HBase 与 MapReduce（数据：HDFS <==> HBase）1、业务流程2、HBaseToHDFS2.1、导入依赖2.2、代码实现2.3、jar 包运行3、HDFSToHBase3.1、准备数据3.2、代码实现3.3、jar 包运行 1、业务流程 为什么需要用 MapReduce 去访问 HBase 的数据？ 答：加快分析速度和扩展分析能力。 MapReduce 访问 HBase 数据作分析一定是在离线分析的场景下应用： 2、HBaseToHDFS 2.1、导入依赖 <?x
向HBase中导入数据2：使用MapReduce从HDFS或本地文件中读取数据并写入HBase（只使用Map逐条查询）
陈杰
03-16 3957
一、在HBase中创建空表二、准备好要写入HBase的文件（可能存在HDFS或者本地，需要修改输入文件路径，HDFS使用hdfs://开头，本地文件使用file://开头）例如我有这样一份文件：其保存在HDFS上三、检查能否调用hadoop读取该文件package cn.edu.shu.ces.chenjie.tianyi.hadoop; import java.io.IOException; ...
MapReduce输出结果导入HBase
lemon
01-17 1137
MapReduce输出结果导入HBase 这里以wordCount为例简单介绍下,代码实现MapReduce计算完数据导入到HBase中 Job.java package com.shsxt.mapre.tohbase; import java.io.IOException; import org.apache.hadoop.conf.Configuration; import org.apa...
Apache HBase MapReduce实现导出数据到HBase
猫君之上的博客
04-03 314
源HBASE表数据： 2.4.1 :030 > scan 'mktest:mk1' ROW COLUMN+CELL ...
mapreduce读取文件内容输出到hbase
小魏的博客
06-30 908
package hadoop; import java.io.IOException; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.hbase.HBaseConfiguration; import org.apache.hadoop
Mapreduce操作HBase
weixin_30608131的博客
01-21 115
这个操作和普通的Mapreduce还不太一样，比如普通的Mapreduce输入可以是txt文件等，Mapreduce可以直接读取Hive中的表的数据(能够看见是以类似txt文件形式)，但Mapreduce操作Hbase却和前二者不一样 它有专门的Mapper 这个叫TableMapper，这个Mapper继承自Mapper，普通的Mapper有4个输入泛型，但这里的TableMapper却只有两...
使用MapReduce程序实现从hbase读写数据输出到hdfs分布式文件系统中
阿中丶20
08-12 813
将hbase中的数据迁移到hdfs分布式文件系统中 package com.briup.hbase; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.conf.Configured; import org.apache.hadoop.fs.Path; import org.apache.hadoop.hbase.client.Result; import org.apache.hadoop.hbase.clie.
Hadoop入门：HDFS、MapReduce与HBase解析
"对Hadoop中的HDFS、MapReduce、HBase系列知识的介绍，适合初学者了解和学习大数据处理框架的入门资料。" Hadoop是一个开源的分布式计算框架，最初由Doug Cutting创建，目的是为了支持大规模数据处理。它被设计成...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值