flink连接kafka整合hbase，scala

最新推荐文章于 2024-07-27 20:16:59 发布

给我一个苹果

最新推荐文章于 2024-07-27 20:16:59 发布

阅读量1.6k

点赞数 1

文章标签：大数据

本文链接：https://blog.csdn.net/weixin_43705952/article/details/106436156

版权

本文详细介绍了如何使用Scala编程语言，通过Apache Flink连接Kafka消费JSON格式的数据，并将解析后的数据存入HBase数据库。整个过程涵盖了数据的获取、解析以及入库操作。

摘要由CSDN通过智能技术生成

解析kafka当中的json格式的数据，入hbase

import java.util.Properties

import com.alibaba.fastjson.{JSON, JSONObject}
import org.apache.flink.api.common.serialization.SimpleStringSchema
import org.apache.flink.contrib.streaming.state.RocksDBStateBackend
import org.apache.flink.streaming.api.CheckpointingMode
import org.apache.flink.streaming.api.environment.CheckpointConfig
import org.apache.flink.streaming.api.scala.{DataStream, StreamExecutionEnvironment}
import org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer011



object IncrementOrder {
  def main(args: Array[String]): Unit = {
    val environment: StreamExecutionEnvironment = StreamExecutionEnvironment.getExecutionEnvironment

    //隐式转换
    import org.apache.flink.api.scala._
    //checkpoint配置
    environment.enableCheckpointing(100);
    environment.getCheckpointConfig.setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE);
    environment.getCheckpointConfig.setMinPauseBetweenCheckpoints(500);
    environment.getCheckpointConfig.setCheckpointTimeout(60000);
    environment.getCheckpointConfig.setMaxConcurrentCheckpoints(1);
    environment.getCheckpointConfig.enableExternalizedCheckpoints(CheckpointConfig.ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);
  environment.setStateBackend(new RocksDBStateBackend("hdfs://node01:8020/flink_kafka/checkpoints&#