Scala PrefixFilter

最新推荐文章于 2021-12-03 00:10:07 发布

power0405hf

最新推荐文章于 2021-12-03 00:10:07 发布

阅读量868

点赞数

本文链接：https://blog.csdn.net/power0405hf/article/details/50301867

版权

scala 同时被 3 个专栏收录

48 篇文章 0 订阅

订阅专栏

spark

34 篇文章 0 订阅

订阅专栏

HBase

14 篇文章 0 订阅

订阅专栏

How to read from hbase using spark

TableInputFormat has the following attributes:

SCAN_ROW_START
SCAN_ROW_STOP
conf.set(TableInputFormat.SCAN_ROW_START, "startrowkey")
conf.set(TableInputFormat.SCAN_ROW_STOP, "stoprowkey")

val myConf = HBaseConfiguration.create()
    myConf.set("hbase.zookeeper.quorum", "master4,slave41,slave42")
    myConf.set("hbase.master", "master4:60000")
    myConf.set("hbase.zookeeper.property.clientPort", "2181")
    myConf.set("hbase.defaults.for.version.skip", "true")
    myConf.set(TableInputFormat.INPUT_TABLE, "test")
    myConf.set(TableInputFormat.SCAN_ROW_START,"1")
    myConf.set(TableInputFormat.SCAN_ROW_STOP,"2")//不会取到stopRowkey
    myConf.set(TableInputFormat.SCAN_COLUMNS, "cf:1Column") 
val hbaseRDD = sc.newAPIHadoopRDD(myConf, classOf[TableInputFormat],
      classOf[org.apache.hadoop.hbase.io.ImmutableBytesWritable],
      classOf[org.apache.hadoop.hbase.client.Result])

    val count = hbaseRDD.count()
    val newHbaseRDD = hbaseRDD.map{ case(_,result) =>
      val key = Bytes.toString(result.getRow)
      key
    }
    newHbaseRDD.collect().foreach(
    println(_)
    )
//output
1_0
1_1
1_2