demo-Spark读取Json数据 ,通过BulkLoadd导入Hbase

最新推荐文章于 2024-05-13 02:23:45 发布

xiaolin_xinji

最新推荐文章于 2024-05-13 02:23:45 发布

阅读量554

点赞数

分类专栏： Spark Hbase 文章标签： spark

本文链接：https://blog.csdn.net/weixin_44131414/article/details/104540924

版权

Spark 同时被 2 个专栏收录

25 篇文章 0 订阅

订阅专栏

Hbase

3 篇文章 0 订阅

订阅专栏

通过调用Hbase API,put的方法将大批量的数据一条一条的导入Hbase中,不仅速度慢,还存在的其他问题,如对Hbase集群造成压力,如CPU和网络资源的使用率。它更合适的应用场景是一般是线上业务运行时，记录单条插入，如报文记录，处理记录，写入后htable对象即释放。

采用Spark + Bulk Load 写入 HBase批量写入数据: 优势

BulkLoad 不会写 WAL，也不会产生 flush 以及 split。
如果我们大量调用 PUT 接口插入数据，可能会导致大量的 GC 操作。除了影响性能之外，严重时甚至可能会对 HBase 节点的稳定性造成影响，采用 BulkLoad 无此顾虑。
过程中没有大量的接口调用消耗性能。
可以利用 Spark 强大的计算能力。

当前开发环境依赖:

  <properties>
    <hadoop.version>2.6.0-cdh5.15.1</hadoop.version>
    <scala.version>2.11.8</scala.version>
    <spark.version>2.4.4</spark.version>
    <hbase.version>1.2.0-cdh5.15.1</hbase.version>
  </properties>


  	 <dependency>
          <groupId>org.apache.spark</groupId>
          <artifactId>spark-sql_2.11</artifactId>
          <version>${spark.version}</version>
     </dependency>
	<dependency>
      <groupId>org.apache.hbase</groupId>
      <artifactId>hbase-client</artifactId>
      <version>${hbase.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.hbase</groupId>
      <artifactId>hbase-server</artifactId>
      <version>${hbase.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.hbase</groupId>
      <artifactId>hbase-common</artifactId>
      <version>${hbase.version}</version>
    </dependency>

测试用例代码实现:

package main.scala.com.xiaolin.hbase


import com.xiaolin.utils.HdfsUtil
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.{FileSystem, Path}
import org.apache.hadoop.hbase.{HBaseConfiguration, HColumnDescriptor, HConstants, HTableDescriptor, KeyValue, TableName}
import org.apache.hadoop.hbase.client.ConnectionFactory
import org.apache.hadoop.hbase.io.ImmutableBytesWritable
import org.apache.hadoop.hbase.mapreduce.{HFileOutputFormat2, LoadIncrementalHFiles, TableOutputFormat}
import org.apache.hadoop.hbase.util.Bytes
import org.apache.hadoop.mapreduce.Job
import org.apache.spark.sql.SparkSession
import com.xiaolin.flink.utils.HbaseUtil

import scala.collection.mutable.ListBuffer

/**
 * 测试用例:
 *    数据从json中,以 Hfile方式导入Hbase
 *    BulkLoad
 *
 * @author linzhy
 */
object DataHiveToHbaseBulkLoad {

  val zookeeperQuorum = "hadoop001:2181"
  val dataSourcePath = "data/data-test.json"
  val hdfsRootPath = "hdfs://hadoop001:9000"
  val hFilePath = "hdfs://hadoop001:9000/bulkload/hfile/"
  val tableName = "person"
  val familyName = "cf1"


  //设置用户
  System.setProperty("user.name", "hadoop")
  System.setProperty("HADOOP_USER_NAME", "hadoop")

  def main(args: Array[String]): Unit = {

    val spark = SparkSession.builder()
      .appName(this.getClass.getSimpleName)
      .master("local[2]")
      .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer") //必须序列化
      .getOrCreate()

    val hadoopConf = new Configuration()
    hadoopConf.set("fs.defaultFS", hdfsRootPath)
    hadoopConf.set("dfs.client.use.datanode.hostname","true");
    val fileSystem = FileSystem.get(hadoopConf)

    val hbaseConf = HBaseConfiguration.create(hadoopConf)
    hbaseConf.set(HConstants.ZOOKEEPER_QUORUM, zookeeperQuorum)
    hbaseConf.set(TableOutputFormat.OUTPUT_TABLE, tableName)
    //如果导入数据量过大,可以适当修改默认值32  
    hbaseConf.set("hbase.mapreduce.bulkload.max.hfiles.perRegion.perFamily","3200")

    val hbaseConn = ConnectionFactory.createConnection(hbaseConf)
    val admin = hbaseConn.getAdmin
    val regionLocator = hbaseConn.getRegionLocator(TableName.valueOf(tableName))

    // 0. 准备程序运行的环境
    // 如果 HBase 表不存在，就创建一个新表
    HbaseUtil.createTable(tableName,familyName,admin)

    // 如果存放 HFile文件的路径已经存在，就删除掉
    HdfsUtil.deleteFile(hFilePath,fileSystem)

    // 1. 清洗需要存放到 HFile 中的数据，rowKey 一定要排序，否则会报错：
    // java.io.IOException: Added a key not lexically larger than previous.

    val dataFrame = spark.read.json(dataSourcePath).select("sessionid","sdkversion","requestdate","email","title")

    val columns = dataFrame.columns.dropWhile(_=="sessionid").sortBy(x=>(x,true)) //排序

    //数据处理
    val data = dataFrame.rdd.map(jsonstr => {
      val rowkey = jsonstr.getAs[String]("sessionid")
      val ik = new ImmutableBytesWritable(Bytes.toBytes(rowkey))
      var linkedList = new ListBuffer[KeyValue]()
      columns.map(column => {
        val columnValue = jsonstr.getAs[String](column).toString
        val kv = new KeyValue(Bytes.toBytes(rowkey), Bytes.toBytes(familyName), Bytes.toBytes(column), Bytes.toBytes(columnValue))
        linkedList.append(kv)
      })
      (ik, linkedList)
    }).flatMapValues(s => { //打散 排序
        val value = s.iterator
        value
      }
    ).sortByKey()

     // 2. Save Hfiles on HDFS
    val table = hbaseConn.getTable(TableName.valueOf(tableName))
    val job = Job.getInstance(hbaseConf)
    job.setMapOutputKeyClass(classOf[ImmutableBytesWritable])
    job.setMapOutputValueClass(classOf[KeyValue])
    HFileOutputFormat2.configureIncrementalLoadMap(job, table)

    data.saveAsNewAPIHadoopFile(
      hFilePath,
      classOf[ImmutableBytesWritable],
      classOf[KeyValue],
      classOf[HFileOutputFormat2],
      hbaseConf
    )

    //  3. Bulk load Hfiles to Hbase
    val bulkLoader = new LoadIncrementalHFiles(hbaseConf)
    bulkLoader.doBulkLoad(new Path(hFilePath), admin, table, regionLocator)

    hbaseConn.close()
    fileSystem.close()
    spark.stop()
  }

}

遇到的bug:

1.产生原因:
在制作HFile文件的时候，一定要主键排序。Put进去会自动排序。但自己做成HFile文件不会自动排序。所有一定要排序好，从主键列族列按照字典排序

java.io.IOException: Added a key not lexically larger than previous. Current cell = 03IQiB2M1F75PyLlKSyA7YmtT18wRcLB/cf1:requestdate/1582794808636/Put/vlen=10/seqid=0, lastCell = 03IQiB2M1F75PyLlKSyA7YmtT18wRcLB/cf1:sdkversion/1582794808636/Put/vlen=11/seqid=0
	at org.apache.hadoop.hbase.io.hfile.AbstractHFileWriter.checkKey(AbstractHFileWriter.java:204)
	at org.apache.hadoop.hbase.io.hfile.HFileWriterV2.append(HFileWriterV2.java:265)
	at org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:996)
	at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:199)
	at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:152)
	at org.apache.spark.internal.io.HadoopMapReduceWriteConfigUtil.write(SparkHadoopWriter.scala:358)
	at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$4.apply(SparkHadoopWriter.scala:132)
	at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$4.apply(SparkHadoopWriter.scala:129)
	at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1394)
	at org.apache.spark.internal.io.SparkHadoopWriter$.org$apache$spark$internal$io$SparkHadoopWriter$$executeTask(SparkHadoopWriter.scala:141)
	at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:83)
	at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:78)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
	at org.apache.spark.scheduler.Task.run(Task.scala:123)
	at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)

2.产生原因:
对象需要序列化

Caused by: org.apache.spark.SparkException: 
	Job aborted due to stage failure: Task 0.0 in stage 1.0 (TID 1) 
	had a not serializable result: org.apache.hadoop.hbase.io.ImmutableBytesWritable

xiaolin_xinji

关注

0
点赞
踩
3

收藏

觉得还不错? 一键收藏
0
评论
demo-Spark读取Json数据 ,通过BulkLoadd导入Hbase

通过调用Hbase API,put的方法将大批量的数据一条一条的导入Hbase中,不仅速度慢,还存在的其他问题,如对Hbase集群造成压力,如CPU和网络资源的使用率。它更合适的应用场景是一般是线上业务运行时，记录单条插入，如报文记录，处理记录，写入后htable对象即释放。采用Spark + Bulk Load 写入 HBase批量写入数据: 优势BulkLoad 不会写 WAL，也不会...
复制链接

扫一扫

专栏目录