使用NewHadoopRDD實現HBase分布式Scan操作

最新推荐文章于 2021-09-14 10:18:07 发布

Xiao 伙伴

最新推荐文章于 2021-09-14 10:18:07 发布

阅读量803

点赞数

分类专栏： hbase入门文章标签： hbase RDD 分布式 spark

本文链接：https://blog.csdn.net/qq_28890765/article/details/70880613

版权

hbase入门专栏收录该内容

2 篇文章 0 订阅

订阅专栏

實現十分簡單，作爲上文的補充，直接上測試代碼

def main(args: Array[String]): Unit{
val sparkConf = new SparkConf().setAppName("HBaseDistributedScanExample " + tableName).setMaster("local[*]")
sparkConf.registerKryoClasses(Array(classOf[ImmutableBytesWritable]))
val sc = new SparkContext(sparkConf)
val job: Job = Job.getInstance(conf)

val scan = new Scan()
scan.setCaching(100)
TableMapReduceUtil.initTableMapperJob(TableName.valueOf(tableName), scan,
classOf[IdentityTableMapper], null, null, job)
val jConf = new JobConf(job.getConfiguration)
SparkHadoopUtil.get.addCredentials(jConf)
val rdd = new NewHadoopRDD[ImmutableBytesWritable, Result](sc,
classOf[TableInputFormat],
classOf[ImmutableBytesWritable],
classOf[Result], job.getConfiguration).map((r: (ImmutableBytesWritable, Result)) => r)
rdd.foreach(v => println(Bytes.toString(v._1.get())))
println("Length: " + rdd.map(r => r._1.copyBytes()).collect().length)
}