delete hive_FlinkSql读取Kakfa数据写入Hive

准备

01 - 准备json数据

{
    "id":"1","name":"张飞","age":"100","dt":"2020-12-15 10:10:10","ts":"1607998210000"}{
    "id":"2","name":"吕布","age":"8100","dt":"2020-12-15 11:10:11","ts":"1608001811000"}{
    "id":"4","name":"帝天","age":"444","dt":"2020-12-16 13:10:13","ts":"1608095413000"}{
    "id":"4","name":"嬴政","age":"2200","dt":"2020-12-18 14:10:14","ts":"1608271814000"}{
    "id":"5","name":"赵姬","age":"2206","dt":"2020-12-19 15:10:15","ts":"1608361815000"}{
    "id":"6","name":"吕不韦","age":"2206","dt":"2020-12-20 16:10:16","ts":"1608451816000"}{
    "id":"7","name":"嬴驷","age":"2207","dt":"2020-12-27 17:10:17","ts":"1609060217000"}{
    "id":"8","name":"嬴渠梁","age":"2208","dt":"2020-12-28 18:10:18","ts":"1609150218000"}{
    "id":"9","name":"嬴异人","age":"2209","dt":"2020-12-23 19:10:19","ts":"1608721819000"}{
    "id":"10","name":"嫪毐","age":"2210","dt":"2020-12-22 20:10:20","ts":"1608639020000"}{
    "id":"11","name":"李斯","age":"2210","dt":"2020-12-22 20:10:20","ts":"1608639020000"}

02 - 创建Kafka Topic

bin/kafka-topics.sh --create --topic mytest --zookeeper hadoop01:2181 --replication-factor 2 --partitions 1bin/kafka-topics.sh --delete --topic mytest --zookeeper hadoop01:2181bin/kafka-console-producer.sh --broker-list hadoop01:6667 --topic mytest

03 - Flink 1.12准备及配置

Flink lib目录

Flink配置

jobmanager.rpc.address: hadoop01jobmanager.rpc.port: 6123state.backend: rocksdbstate.checkpoints.dir: hdfs:///flink/flink-checkpointsstate.savepoints.dir: hdfs:///flink/flink-savepointsjobmanager.archive.fs.dir: hdfs:///flink/completed-jobshistoryserver.web.port: 8082historyserver.archive.fs.dir: hdfs:///flink/completed-jobs/

Hadoop classpath配置(必须)

export HADOOP_HOME=/usr/hdp/2.6.1.0-129/hadoopexport HADOOP_CLASSPATH=`hadoop classpath`

POM文件

<?xml version="1.0" encoding="UTF-8"?>         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">            iot-online        com.yunmi        1.0-SNAPSHOT        4.0.0    flinksql12                        org.apache.flink            flink-table-api-scala-bridge_2.11            1.12.0            provided                                    org.apache.flink            flink-table-planner_2.11            1.12.0            provided                                    org.apache.flink