spark使用java读取hbase数据做分布式计算

最新推荐文章于 2022-05-30 10:15:44 发布

克终

最新推荐文章于 2022-05-30 10:15:44 发布

阅读量3.8k

点赞数

分类专栏：分布式大数据处理引擎

分布式大数据处理引擎专栏收录该内容

25 篇文章 1 订阅

订阅专栏

问题导读：
1.如何初始化sparkContext？
2.如何设置查询条件？
3.如何获得hbase查询结果Result？

由于spark提供的hbaseTest是scala版本，并没有提供java版。我将scala版本改为java版本，并根据数据做了些计算操作。

程序目的：查询出hbase满足条件的用户，统计各个等级个数。

代码如下，西面使用的hbase是0.94注释已经写详细：

package com.sdyc.ndspark.sys;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.client.Scan;
import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
import org.apache.hadoop.hbase.mapreduce.TableInputFormat;
import org.apache.hadoop.hbase.util.Base64;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.spark.api.java.JavaPairRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.api.java.function.Function2;
import org.apache.spark.api.java.function.PairFunction;
import scala.Tuple2;

import java.io.ByteArrayOutputStream;
import java.io.DataOutputStream;
import java.io.IOException;
import java.io.Serializable;
import java.util.List;

public class HbaseTest implements Serializable {

public Log log = LogFactory.getLog(HbaseTest.class);

    static String convertScanToString(Scan scan) throws IOException {
        ByteArrayOutputStream out = new ByteArrayOutputStream();
        DataOutputStream dos = new DataOutputStream(out);
        scan.write(dos);
        return Base64.encodeBytes(out.toByteArray());
    }

    public void start() {
        //初始化sparkContext，这里必须在jars参数里面放上Hbase的jar，
        // 否则会报unread block data异常
        JavaSparkContext sc = new JavaSparkContext("spark://nowledgedata-n3:7077", "hbaseTest",
                "/home/hadoop/software/spark-0.8.1",
                new String[]{"target/ndspark.jar", "target\\dependency\\hbase-0.94.6.jar"});

        //使用HBaseConfiguration.create()生成Configuration
        // 必须在项目classpath下放上hadoop以及hbase的配置文件。
        Configuration conf = HBaseConfiguration.create();
        //设置查询条件，这里值返回用户的等级
        Scan scan = new Scan();
        scan.setStartRow(Bytes.toBytes("195861-1035177490"));
        scan.setStopRow(Bytes.toBytes("195861-1072173147"));
        scan.addFamily(Bytes.toBytes("info"));
        scan.addColumn(Bytes.toBytes("info"), Bytes.toBytes("levelCode"));

        try {
            //需要读取的hbase表名
            String tableName = "usertable";
            conf.set(TableInputFormat.INPUT_TABLE, tableName);
            conf.set(TableInputFormat.SCAN, convertScanToString(scan));

            //获得hbase查询结果Result
            JavaPairRDD hBaseRDD = sc.newAPIHadoopRDD(conf,
                    TableInputFormat.class, ImmutableBytesWritable.class,
                    Result.class);

            //从result中取出用户的等级，并且每一个算一次
            JavaPairRDD levels = hBaseRDD.map(
                    new PairFunction, Integer, Integer>() {
                        @Override
                        public Tuple2 call(
                                Tuple2 immutableBytesWritableResultTuple2)
                                throws Exception {
                            byte[] o = immutableBytesWritableResultTuple2._2().getValue(
                                    Bytes.toBytes("info"), Bytes.toBytes("levelCode"));
                            if (o != null) {
                                return new Tuple2(Bytes.toInt(o), 1);
                            }
                            return null;
                        }
                    });

            //数据累加
            JavaPairRDD counts = levels.reduceByKey(new Function2() {
                public Integer call(Integer i1, Integer i2) {
                    return i1 + i2;
                }
            });

            //打印出最终结果
            List> output = counts.collect();
            for (Tuple2 tuple : output) {
                System.out.println(tuple._1 + ": " + tuple._2);
            }

        } catch (Exception e) {
            log.warn(e);
        }

}

public static void main(String[] args) throws InterruptedException {

new HbaseTest().start();

System.exit(0);
}
}
复制代码
注意：如果使用的是hbase0.96.1.1-hadoop2

convertScanToString函数需要改为：

http://www.v-cnc.com/news/show/18566/

http://www.v-cnc.com/news/show/18565/

http://www.v-cnc.com/news/show/18564/

http://www.v-cnc.com/news/show/18563/

http://www.v-cnc.com/news/show/18562/

http://www.v-cnc.com/news/show/18558/

http://www.v-cnc.com/news/show/18555/

http://www.v-cnc.com/news/show/18551/

http://www.v-cnc.com/news/show/18548/

http://www.v-cnc.com/news/show/18546/

http://www.v-cnc.com/news/show/18543/

http://www.v-cnc.com/news/show/18539/

http://www.v-cnc.com/news/show/18536/

http://www.v-cnc.com/news/show/18533/

http://www.v-cnc.com/news/show/18530/

http://www.v-cnc.com/news/show/18527/

http://www.v-cnc.com/news/show/18523/

http://www.v-cnc.com/news/show/18520/

http://www.v-cnc.com/news/show/18517/

http://www.v-cnc.com/news/show/18515/

http://www.v-cnc.com/news/show/18512/

http://www.v-cnc.com/news/show/18511/

http://www.v-cnc.com/news/show/18510/

http://www.v-cnc.com/news/show/18508/

http://www.v-cnc.com/news/show/18506/

http://www.v-cnc.com/news/show/18505/

http://www.v-cnc.com/news/show/18503/

http://www.v-cnc.com/news/show/18502/

http://www.v-cnc.com/news/show/18501/

http://www.v-cnc.com/news/show/18500/

http://www.v-cnc.com/news/show/18499/

http://www.v-cnc.com/news/show/18497/

http://www.v-cnc.com/news/show/18496/

http://www.v-cnc.com/news/show/18495/

http://www.v-cnc.com/news/show/18494/

http://www.v-cnc.com/news/show/18493/

http://www.v-cnc.com/news/show/18492/

http://www.v-cnc.com/news/show/18491/

http://www.v-cnc.com/news/show/18490/

http://www.v-cnc.com/news/show/18489/

http://www.v-cnc.com/news/show/18488/

http://www.v-cnc.com/news/show/18486/

http://www.v-cnc.com/news/show/18485/

http://www.v-cnc.com/news/show/18484/

http://www.v-cnc.com/news/show/18483/

http://www.v-cnc.com/news/show/18482/

http://www.v-cnc.com/news/show/18481/

http://www.v-cnc.com/news/show/18480/

http://www.v-cnc.com/news/show/18479/

http://www.v-cnc.com/news/show/18478/

http://www.v-cnc.com/news/show/18477/

http://www.v-cnc.com/news/show/18476/

http://www.v-cnc.com/news/show/18475/

http://www.v-cnc.com/news/show/18474/

http://www.v-cnc.com/news/show/18473/

http://www.v-cnc.com/news/show/18472/

http://www.v-cnc.com/news/show/18471/

http://www.v-cnc.com/news/show/18470/

http://www.v-cnc.com/news/show/18469/

http://www.v-cnc.com/news/show/18468/

http://www.v-cnc.com/news/show/18467/

http://www.v-cnc.com/news/show/18466/

http://www.v-cnc.com/news/show/18465/

http://www.v-cnc.com/news/show/18464/

http://www.v-cnc.com/news/show/18463/

http://www.v-cnc.com/news/show/18462/

http://www.v-cnc.com/news/show/18461/

http://www.v-cnc.com/news/show/18460/

http://www.v-cnc.com/news/show/18459/

http://www.v-cnc.com/news/show/18458/

http://www.v-cnc.com/news/show/18457/

http://www.v-cnc.com/news/show/18456/

http://www.v-cnc.com/news/show/18455/

http://www.v-cnc.com/news/show/18454/

http://www.v-cnc.com/news/show/18453/

http://www.v-cnc.com/news/show/18452/

http://www.v-cnc.com/news/show/18451/

http://www.v-cnc.com/news/show/18450/

http://www.v-cnc.com/news/show/18449/

http://www.v-cnc.com/news/show/18448/

http://www.v-cnc.com/news/show/18447/

http://www.v-cnc.com/news/show/18446/

http://www.v-cnc.com/news/show/18445/

http://www.v-cnc.com/news/show/18444/

http://www.v-cnc.com/news/show/18443/

http://www.v-cnc.com/news/show/18442/

http://www.v-cnc.com/news/show/18441/

http://www.v-cnc.com/news/show/18440/

http://www.v-cnc.com/news/show/18439/

http://www.v-cnc.com/news/show/18438/

http://www.v-cnc.com/news/show/18437/

http://www.v-cnc.com/news/show/18436/

http://www.v-cnc.com/news/show/18435/

http://www.v-cnc.com/news/show/18434/

http://www.v-cnc.com/news/show/18433/

http://www.v-cnc.com/news/show/18432/

http://www.v-cnc.com/news/show/18431/

http://www.v-cnc.com/news/show/18430/

http://www.v-cnc.com/news/show/18429/

http://www.v-cnc.com/news/show/18428/

http://www.v-cnc.com/news/show/18427/

http://www.v-cnc.com/news/show/18426/

http://www.v-cnc.com/news/show/18425/

http://www.v-cnc.com/news/show/18424/

http://www.v-cnc.com/news/show/18423/

http://www.v-cnc.com/news/show/18422/

http://www.v-cnc.com/news/show/18421/

http://www.v-cnc.com/news/show/18420/

http://www.v-cnc.com/news/show/18419/

http://www.v-cnc.com/news/show/18418/

http://www.v-cnc.com/news/show/18417/

http://www.v-cnc.com/news/show/18416/

http://www.v-cnc.com/news/show/18415/

http://www.v-cnc.com/news/show/18414/

http://www.v-cnc.com/news/show/18413/

http://www.v-cnc.com/news/show/18412/

http://www.v-cnc.com/news/show/18411/

http://www.v-cnc.com/news/show/18410/

http://www.v-cnc.com/news/show/18409/

http://www.v-cnc.com/news/show/18408/

http://www.v-cnc.com/news/show/18407/

http://www.v-cnc.com/news/show/18406/

http://www.v-cnc.com/news/show/18405/

http://www.v-cnc.com/news/show/18404/

http://www.v-cnc.com/news/show/18403/

http://www.v-cnc.com/news/show/18402/

http://www.v-cnc.com/news/show/18401/

http://www.v-cnc.com/news/show/18400/

http://www.v-cnc.com/news/show/18399/

http://www.v-cnc.com/news/show/18398/

http://www.v-cnc.com/news/show/18397/

http://www.v-cnc.com/news/show/18396/

http://www.v-cnc.com/news/show/18395/

http://www.v-cnc.com/news/show/18394/

http://www.v-cnc.com/news/show/18393/

http://www.v-cnc.com/news/show/18392/

http://www.v-cnc.com/news/show/18390/

http://www.v-cnc.com/news/show/18389/

http://www.v-cnc.com/news/show/18388/

http://www.v-cnc.com/news/show/18387/

http://www.v-cnc.com/news/show/18386/

http://www.v-cnc.com/news/show/18385/

http://www.v-cnc.com/news/show/18384/

http://www.v-cnc.com/news/show/18383/

http://www.v-cnc.com/news/show/18382/

http://www.v-cnc.com/news/show/18381/

http://www.v-cnc.com/news/show/18380/

http://www.v-cnc.com/news/show/18379/

http://www.v-cnc.com/news/show/18378/

http://www.v-cnc.com/news/show/18377/

http://www.v-cnc.com/news/show/18376/

http://www.v-cnc.com/news/show/18375/

http://www.v-cnc.com/news/show/18374/

http://www.v-cnc.com/news/show/18373/

http://www.v-cnc.com/news/show/18372/

http://www.v-cnc.com/news/show/18371/

http://www.v-cnc.com/news/show/18370/

http://www.v-cnc.com/news/show/18369/

http://www.v-cnc.com/news/show/18368/

http://www.v-cnc.com/news/show/18367/

http://www.v-cnc.com/news/show/18366/

http://www.v-cnc.com/news/show/18365/

http://www.v-cnc.com/news/show/18364/

http://www.v-cnc.com/news/show/18363/

http://www.v-cnc.com/news/show/18362/

http://www.v-cnc.com/news/show/18361/

http://www.v-cnc.com/news/show/18360/

http://www.v-cnc.com/news/show/18359/

http://www.v-cnc.com/news/show/18358/

http://www.v-cnc.com/news/show/18357/

http://www.v-cnc.com/news/show/18356/

http://www.v-cnc.com/news/show/18355/

http://www.v-cnc.com/news/show/18354/

http://www.v-cnc.com/news/show/18353/

http://www.v-cnc.com/news/show/18352/

http://www.v-cnc.com/news/show/18351/

http://www.v-cnc.com/news/show/18350/

http://www.v-cnc.com/news/show/18349/

http://www.v-cnc.com/news/show/18348/

http://www.v-cnc.com/news/show/18347/

http://www.v-cnc.com/news/show/18346/

http://www.v-cnc.com/news/show/18345/

http://www.v-cnc.com/news/show/18344/

http://www.v-cnc.com/news/show/18343/

http://www.v-cnc.com/news/show/18342/

http://www.v-cnc.com/news/show/18341/

http://www.v-cnc.com/news/show/18340/

http://www.v-cnc.com/news/show/18339/

http://www.v-cnc.com/news/show/18338/

http://www.v-cnc.com/news/show/18337/

http://www.v-cnc.com/news/show/18336/

http://www.v-cnc.com/news/show/18335/

http://www.v-cnc.com/news/show/18334/

http://www.v-cnc.com/news/show/18333/

http://www.v-cnc.com/news/show/18332/

http://www.v-cnc.com/news/show/18331/

http://www.v-cnc.com/news/show/18330/

http://www.v-cnc.com/news/show/18329/

http://www.v-cnc.com/news/show/18328/

http://www.v-cnc.com/news/show/18327/

克终

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
spark使用java读取hbase数据做分布式计算

spark使用java读取hbase数据做分布式计算
复制链接

扫一扫

专栏目录

spark使用java读取hbase数据做分布式计算

“相关推荐”对你有帮助么？