【FLINK】flink 读取Kafka数据写入Hive表中

最新推荐文章于 2024-07-24 10:08:06 发布
最新推荐文章于 2024-07-24 10:08:06 发布
阅读量5.6k 收藏 38
点赞数 6
分类专栏： Flink 文章标签： Flink Kafka Hive
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/Zsigner/article/details/116564806
版权

背景：对实时的历史数据持久化保存到Hdfs中，可以使用Hive Sql 追溯历史数据给业务参考多维度指标

版本信息：Flink 1.11.2，Scala 2.11，Kafka 2.12， Hive 2.3.7

 

1、引入pom.xml内容

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.zen.test</groupId>
    <artifactId>flink_test</artifactId>
    <version>1.0-SNAPSHOT</version>

    <packaging>jar</packaging>
    <properties>
        <flink.version>1.11.2</flink.version>
        <hadoop.version>2.6.0</hadoop.version>
        <hbase.version>1.4.9</hbase.version>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <maven.compiler.encoding>UTF-8</maven.compiler.encoding>
    </properties>

    <dependencies>


        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-table-common</artifactId>
            <version>${flink.version}</version>
            <exclusions>
                <exclusion>
                    <artifactId>commons-compress</artifactId>
                    <groupId>org.apache.commons</groupId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-shaded-hadoop-2-uber</artifactId>
            <version>2.6.5-10.0</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-table-planner-blink -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-table-planner-blink_2.11</artifactId>
            <version>${flink.version}</version>
            <exclusions>
                <exclusion>
                    <artifactId>scala-reflect</artifactId>
                    <groupId>org.scala-lang</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>scala-xml_2.11</artifactId>
                    <groupId>org.scala-lang.modules</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>scala-parser-combinators_2.11</artifactId>
                    <groupId>org.scala-lang.modules</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>flink-table-api-scala_2.11</artifactId>
                    <groupId>org.apache.flink</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>flink-scala_2.11</artifactId>
                    <groupId>org.apache.flink</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>flink-streaming-scala_2.11</artifactId>
                    <groupId>org.apache.flink</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>avatica-core</artifactId>
                    <groupId>org.apache.calcite.avatica</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>javassist</artifactId>
                    <groupId>org.javassist</groupId>
                </exclusion>
            </exclusions>
        </dependency>


        <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-connector-hive -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-connector-hive_2.11</artifactId>
            <version>${flink.version}</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-sql-connector-hive-2.3.6 -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-sql-connector-hive-2.3.6_2.11</artifactId>
            <version>1.11.2</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.apache.hive/hive-exec -->
        <dependency>
            <groupId>org.apache.hive</groupId>
            <artifactId>hive-exec</artifactId>
            <version>2.3.4</version>
            <exclusions>

                <exclusion>
                    <artifactId>dropwizard-metrics-hadoop-metrics2-reporter</artifactId>
                    <groupId>com.github.joshelser</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>calcite-core</artifactId>
                    <groupId>org.apache.calcite</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>libthrift</artifactId>
                    <groupId>org.apache.thrift</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hive-vector-code-gen</artifactId>
                    <groupId>org.apache.hive</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>metrics-jvm</artifactId>
                    <groupId>io.dropwizard.metrics</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>curator-client</artifactId>
                    <groupId>org.apache.curator</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>orc-core</artifactId>
                    <groupId>org.apache.orc</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>zookeeper</artifactId>
                    <groupId>org.apache.zookeeper</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>metrics-json</artifactId>
                    <groupId>io.dropwizard.metrics</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hive-storage-api</artifactId>
                    <groupId>org.apache.hive</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>calcite-druid</artifactId>
                    <groupId>org.apache.calcite</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hive-common</artifactId>
                    <groupId>org.apache.hive</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>avatica-metrics</artifactId>
                    <groupId>org.apache.calcite.avatica</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>avatica</artifactId>
                    <groupId>org.apache.calcite.avatica</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hive-service-rpc</artifactId>
                    <groupId>org.apache.hive</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hive-serde</artifactId>
                    <groupId>org.apache.hive</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hive-llap-common</artifactId>
                    <groupId>org.apache.hive</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hive-llap-client</artifactId>
                    <groupId>org.apache.hive</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hive-llap-tez</artifactId>
                    <groupId>org.apache.hive</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hive-shims-common</artifactId>
                    <groupId>org.apache.hive.shims</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hadoop-yarn-server-resourcemanager</artifactId>
                    <groupId>org.apache.hadoop</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hive-shims-0.23</artifactId>
                    <groupId>org.apache.hive.shims</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hive-shims-scheduler</artifactId>
                    <groupId>org.apache.hive.shims</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hive-shims</artifactId>
                    <groupId>org.apache.hive</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>guava</artifactId>
                    <groupId>com.google.guava</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>commons-codec</artifactId>
                    <groupId>commons-codec</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>commons-logging</artifactId>
                    <groupId>commons-logging</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>junit</artifactId>
                    <groupId>junit</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>antlr-runtime</artifactId>
                    <groupId>org.antlr</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>slf4j-api</artifactId>
                    <groupId>org.slf4j</groupId>
                </exclusion>
            </exclusions>
        </dependency>


        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-table-api-java-bridge_2.11</artifactId>
            <version>${flink.version}</version>
            <exclusions>
                <exclusion>
                    <artifactId>flink-runtime_2.11</artifactId>
                    <groupId>org.apache.flink</groupId>
                </exclusion>
            </exclusions>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.apache.hbase/hbase-client -->
        <dependency>
            <groupId>org.apache.hbase</groupId>
            <artifactId>hbase-client</artifactId>
            <version>${hbase.version}</version>
            <exclusions>
                <exclusion>
                    <artifactId>metrics-core</artifactId>
                    <groupId>com.yammer.metrics</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>zookeeper</artifactId>
                    <groupId>org.apache.zookeeper</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>apacheds-i18n</artifactId>
                    <groupId>org.apache.directory.server</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>api-asn1-api</artifactId>
                    <groupId>org.apache.directory.api</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>api-util</artifactId>
                    <groupId>org.apache.directory.api</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>apacheds-kerberos-codec</artifactId>
                    <groupId>org.apache.directory.server</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hadoop-auth</artifactId>
                    <groupId>org.apache.hadoop</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hadoop-mapreduce-client-core</artifactId>
                    <groupId>org.apache.hadoop</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>commons-codec</artifactId>
                    <groupId>commons-codec</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hadoop-common</artifactId>
                    <groupId>org.apache.hadoop</groupId>
                </exclusion>
            </exclusions>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.apache.hbase/hbase-common -->
        <dependency>
            <groupId>org.apache.hbase</groupId>
            <artifactId>hbase-common</artifactId>
            <version>${hbase.version}</version>
            <exclusions>
                <exclusion>
                    <artifactId>avro</artifactId>
                    <groupId>org.apache.avro</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hadoop-mapreduce-client-core</artifactId>
                    <groupId>org.apache.hadoop</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>commons-codec</artifactId>
                    <groupId>commons-codec</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hadoop-common</artifactId>
                    <groupId>org.apache.hadoop</groupId>
                </exclusion>
            </exclusions>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.apache.hbase/hbase-server -->
        <dependency>
            <groupId>org.apache.hbase</groupId>
            <artifactId>hbase-server</artifactId>
            <version>${hbase.version}</version>
            <exclusions>
                <exclusion>
                    <artifactId>metrics-core</artifactId>
                    <groupId>com.yammer.metrics</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>zookeeper</artifactId>
                    <groupId>org.apache.zookeeper</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>metrics-core</artifactId>
                    <groupId>io.dropwizard.metrics</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hadoop-auth</artifactId>
                    <groupId>org.apache.hadoop</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hadoop-mapreduce-client-core</artifactId>
                    <groupId>org.apache.hadoop</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>avro</artifactId>
                    <groupId>org.apache.avro</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hbase-common</artifactId>
                    <groupId>org.apache.hbase</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>commons-codec</artifactId>
                    <groupId>commons-codec</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>commons-httpclient</artifactId>
                    <groupId>commons-httpclient</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>commons-logging</artifactId>
                    <groupId>commons-logging</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hadoop-client</artifactId>
                    <groupId>org.apache.hadoop</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hadoop-common</artifactId>
                    <groupId>org.apache.hadoop</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hadoop-hdfs</artifactId>
                    <groupId>org.apache.hadoop</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>httpclient</artifactId>
                    <groupId>org.apache.httpcomponents</groupId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>${hadoop.version}</version>
            <exclusions>
                <exclusion>
                    <artifactId>hadoop-yarn-client</artifactId>
                    <groupId>org.apache.hadoop</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hadoop-yarn-common</artifactId>
                    <groupId>org.apache.hadoop</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hadoop-yarn-server-common</artifactId>
                    <groupId>org.apache.hadoop</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hadoop-yarn-api</artifactId>
                    <groupId>org.apache.hadoop</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>zookeeper</artifactId>
                    <groupId>org.apache.zookeeper</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hadoop-mapreduce-client-common</artifactId>
                    <groupId>org.apache.hadoop</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hadoop-mapreduce-client-app</artifactId>
                    <groupId>org.apache.hadoop</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hadoop-mapreduce-client-jobclient</artifactId>
                    <groupId>org.apache.hadoop</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hadoop-mapreduce-client-core</artifactId>
                    <groupId>org.apache.hadoop</groupId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>com.typesafe</groupId>
            <artifactId>config</artifactId>
            <version>1.3.3</version>
        </dependency>
        <dependency>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpclient</artifactId>
            <version>4.5.1</version>
            <exclusions>
                <exclusion>
                    <artifactId>commons-codec</artifactId>
                    <groupId>commons-codec</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>httpcore</artifactId>
                    <groupId>org.apache.httpcomponents</groupId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>com.alibaba</groupId>
            <artifactId>fastjson</artifactId>
            <version>1.2.74</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-clients -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-clients_2.12</artifactId>
            <version>${flink.version}</version>
            <exclusions>
                <exclusion>
                    <artifactId>flink-core</artifactId>
                    <groupId>org.apache.flink</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>commons-cli</artifactId>
                    <groupId>commons-cli</groupId>
                </exclusion>
            </exclusions>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-streaming-java -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-streaming-java_2.12</artifactId>
            <version>${flink.version}</version>
            <exclusions>
                <exclusion>
                    <artifactId>flink-core</artifactId>
                    <groupId>org.apache.flink</groupId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>log4j</groupId>
            <artifactId>log4j</artifactId>
            <version>1.2.17</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-runtime -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-runtime_2.12</artifactId>
            <version>${flink.version}</version>
            <exclusions>

                <exclusion>
                    <artifactId>grizzled-slf4j_2.12</artifactId>
                    <groupId>org.clapper</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>flink-core</artifactId>
                    <groupId>org.apache.flink</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>commons-cli</artifactId>
                    <groupId>commons-cli</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>slf4j-api</artifactId>
                    <groupId>org.slf4j</groupId>
                </exclusion>
            </exclusions>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-connector-filesystem -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-connector-filesystem_2.12</artifactId>
            <version>${flink.version}</version>
        </dependency>


        <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-connector-kafka -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-connector-kafka_2.12</artifactId>
            <version>1.11.2</version>
        </dependency>


        <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-avro -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-avro</artifactId>
            <version>${flink.version}</version>
            <exclusions>
                <exclusion>
                    <artifactId>avro</artifactId>
                    <groupId>org.apache.avro</groupId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-avro-confluent-registry</artifactId>
            <version>${flink.version}</version>
        </dependency>


        <dependency>
            <groupId>org.apache.kudu</groupId>
            <artifactId>kudu-client</artifactId>
            <version>1.7.0</version>
            <exclusions>
                <exclusion>
                    <artifactId>slf4j-api</artifactId>
                    <groupId>org.slf4j</groupId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.12</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-hadoop-fs</artifactId>
            <version>${flink.version}</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common -->
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>${hadoop.version}</version>
            <exclusions>
                <exclusion>
                    <artifactId>curator-framework</artifactId>
                    <groupId>org.apache.curator</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>curator-client</artifactId>
                    <groupId>org.apache.curator</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>curator-recipes</artifactId>
                    <groupId>org.apache.curator</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>htrace-core</artifactId>
                    <groupId>org.htrace</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>zookeeper</artifactId>
                    <groupId>org.apache.zookeeper</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>commons-httpclient</artifactId>
                    <groupId>commons-httpclient</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>jets3t</artifactId>
                    <groupId>net.java.dev.jets3t</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hadoop-auth</artifactId>
                    <groupId>org.apache.hadoop</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>commons-configuration</artifactId>
                    <groupId>commons-configuration</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>commons-el</artifactId>
                    <groupId>commons-el</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>commons-beanutils</artifactId>
                    <groupId>commons-beanutils</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>commons-digester</artifactId>
                    <groupId>commons-digester</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>commons-beanutils-core</artifactId>
                    <groupId>commons-beanutils</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>jasper-runtime</artifactId>
                    <groupId>tomcat</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>jersey-json</artifactId>
                    <groupId>com.sun.jersey</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>jackson-jaxrs</artifactId>
                    <groupId>org.codehaus.jackson</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>jackson-xc</artifactId>
                    <groupId>org.codehaus.jackson</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>avro</artifactId>
                    <groupId>org.apache.avro</groupId>
                </exclusion>

                <exclusion>
                    <artifactId>apacheds-i18n</artifactId>
                    <groupId>org.apache.directory.server</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>apacheds-kerberos-codec</artifactId>
                    <groupId>org.apache.directory.server</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>api-util</artifactId>
                    <groupId>org.apache.directory.api</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>api-asn1-api</artifactId>
                    <groupId>org.apache.directory.api</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>guava</artifactId>
                    <groupId>com.google.guava</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>commons-collections</artifactId>
                    <groupId>commons-collections</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>commons-logging</artifactId>
                    <groupId>commons-logging</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>commons-compress</artifactId>
                    <groupId>org.apache.commons</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>commons-math3</artifactId>
                    <groupId>org.apache.commons</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>slf4j-api</artifactId>
                    <groupId>org.slf4j</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>slf4j-log4j12</artifactId>
                    <groupId>org.slf4j</groupId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-hdfs</artifactId>
            <version>${hadoop.version}</version>
            <exclusions>
                <exclusion>
                    <artifactId>htrace-core</artifactId>
                    <groupId>org.htrace</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>jasper-runtime</artifactId>
                    <groupId>tomcat</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>commons-el</artifactId>
                    <groupId>commons-el</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>guava</artifactId>
                    <groupId>com.google.guava</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>commons-logging</artifactId>
                    <groupId>commons-logging</groupId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>joda-time</groupId>
            <artifactId>joda-time</artifactId>
            <version>2.9.1</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/com.stumbleupon/async -->
        <dependency>
            <groupId>com.stumbleupon</groupId>
            <artifactId>async</artifactId>
            <version>1.4.1</version>
            <exclusions>
                <exclusion>
                    <artifactId>slf4j-api</artifactId>
                    <groupId>org.slf4j</groupId>
                </exclusion>
            </exclusions>
        </dependency>


        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <version>5.1.35</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/redis.clients/jedis -->
        <dependency>
            <groupId>redis.clients</groupId>
            <artifactId>jedis</artifactId>
            <version>2.9.0</version>
        </dependency>

    </dependencies>

    <build>
        <resources>
            <resource>
                <directory>src/main/resources</directory>
                <includes>
                    <!--包含文件夹以及子文件夹下所有资源-->
                    <include>**/*.*</include>
                </includes>
            </resource>
        </resources>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>2.5.1</version>
                <configuration>
                    <source>${maven.compiler.source}</source>
                    <target>${maven.compiler.target}</target>
                    <encoding>UTF-8</encoding>
                    <!--<encoding>${project.build.sourceEncoding}</encoding>-->
                </configuration>
            </plugin>

            <plugin>
                <groupId>net.alchim31.maven</groupId>
                <artifactId>scala-maven-plugin</artifactId>
                <version>3.2.0</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>compile</goal>
                            <goal>testCompile</goal>
                        </goals>
                        <configuration>
                            <args>
                                <!--<arg>-make:transitive</arg>-->
                                <arg>-dependencyfile</arg>
                                <arg>${project.build.directory}/.scala_dependencies</arg>
                            </args>

                        </configuration>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.18.1</version>
                <configuration>
                    <useFile>false</useFile>
                    <disableXmlReport>true</disableXmlReport>
                    <includes>
                        <include>**/*com.xiaoe.Test.*</include>
                        <include>**/*Suite.*</include>
                    </includes>
                </configuration>
            </plugin>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>3.0.0</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <filters>
                                <filter>
                                    <artifact>*:*</artifact>
                                    <excludes>
                                        <!--
                                        zip -d learn_spark.jar META-INF/*.RSA META-INF/*.DSA META-INF/*.SF
                                        -->
                                        <exclude>META-INF/*.SF</exclude>
                                        <exclude>META-INF/*.DSA</exclude>
                                        <exclude>META-INF/*.RSA</exclude>
                                    </excludes>
                                </filter>
                            </filters>
                           
                        </configuration>
                    </execution>

                </executions>
            </plugin>
        </plugins>
    </build>

</project>

2、使用Table环境

StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();



        EnvironmentSettings tableEnvSettings = EnvironmentSettings.newInstance()
                .useBlinkPlanner()
                .inStreamingMode()
                .build();
        StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env, tableEnvSettings);
        tableEnv.getConfig().getConfiguration().set(ExecutionCheckpointingOptions.CHECKPOINTING_MODE, CheckpointingMode.EXACTLY_ONCE);
        tableEnv.getConfig().getConfiguration().set(ExecutionCheckpointingOptions.CHECKPOINTING_INTERVAL, Duration.ofSeconds(20));

3、使用HiveCatalog

HiveCatalog catalog = new HiveCatalog(
                "myhive",
                "db_test_tmp",
                "/usr/local/service/hive/conf"
        );
        tableEnv.registerCatalog("myhive", catalog);
        tableEnv.useCatalog("myhive");

4、完整编码

package com.source.kafka;

import org.apache.flink.streaming.api.CheckpointingMode;
import org.apache.flink.streaming.api.environment.ExecutionCheckpointingOptions;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.table.api.EnvironmentSettings;
import org.apache.flink.table.api.SqlDialect;
import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
import org.apache.flink.table.catalog.hive.HiveCatalog;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.time.Duration;

public class HiveTest {

    private final static Logger logger = LoggerFactory.getLogger(hiveTest.class);

    public static void main(String[] args) throws Exception {

        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        EnvironmentSettings tableEnvSettings = EnvironmentSettings.newInstance()
                .useBlinkPlanner()
                .inStreamingMode()
                .build();
        StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env, tableEnvSettings);
        tableEnv.getConfig().getConfiguration().set(ExecutionCheckpointingOptions.CHECKPOINTING_MODE, CheckpointingMode.EXACTLY_ONCE);
        tableEnv.getConfig().getConfiguration().set(ExecutionCheckpointingOptions.CHECKPOINTING_INTERVAL, Duration.ofSeconds(20));

        HiveCatalog catalog = new HiveCatalog(
                "myhive",
                "db_test_tmp",
                "/usr/local/service/hive/conf"
        );
        tableEnv.registerCatalog("myhive", catalog);
        tableEnv.useCatalog("myhive");


        tableEnv.executeSql("DROP TABLE IF EXISTS db_test_tmp.test_kafka_source");
        tableEnv.executeSql("CREATE TABLE db_test_tmp.test_kafka_source (\n" +
                "  id INT,\n" +
                "  name STRING,\n" +
                "  age INT\n" +
                ") WITH (\n" +
                " 'connector' = 'kafka'," +
                " 'topic' = 'hive_test',\n" +
                " 'properties.bootstrap.servers' = '127.0.0.1:9092',\n" +
                " 'properties.group.id' = 'group_test_01',\n" +
                " 'format' = 'json',\n"+
                " 'is_generic' = 'false',\n"+
                " 'scan.startup.mode' = 'latest-offset')");

        tableEnv.getConfig().setSqlDialect(SqlDialect.HIVE);
        tableEnv.executeSql("DROP TABLE IF EXISTS db_test_tmp.test_hive_sink");
        tableEnv.executeSql("CREATE TABLE IF NOT EXISTS  db_test_tmp.test_hive_sink (\n" +
                "  id INT,\n" +
                "  name STRING,\n" +
                "  age INT\n" +
                ") STORED AS PARQUET\n" +
                "TBLPROPERTIES (\n" +
                "  'sink.partition-commit.policy.kind' = 'metastore,success-file'\n" +
                ")");

        tableEnv.getConfig().setSqlDialect(SqlDialect.DEFAULT);
        tableEnv.executeSql("INSERT INTO db_test_tmp.test_hive_sink SELECT id,name,age FROM test_kafka_source");
    }
}

5、打包、运行、验证

 5.1 打包

mvn clean install

 5.2 运行 

flink run -m yarn-cluster -ynm test -ys 1   -ytm 1024m   -yD state.checkpoints.dir=hdfs://127.0.0.1:4007/flink/checkpoints   -c com.source.kafka.HiveTest  /tmp/original-flink_test-1.0-SNAPSHOT.jar

 5.3 验证

  5.3.1 kafka 验证

kafka 新建topic，写入数据
#新建topic 
./kafka-topics.sh --zookeeper localhost:2181 --topic hive_test --partitions 1 --replication-factor 1 --create


#写入数据
./kafka-console-producer.sh --broker-list localhost:9092 --topic hive_test
> {"id":1,"name":"zhangsan","age":100}
> {"id":10,"name":"lisi","age":10}
> {"id":9,"name":"wangwu","age":9}
> {"id":8,"name":"maliu","age":8}
> {"id":18,"name":"shi","age":18}


#验证 是否写入，消费
./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic hive_test

 {"id":1,"name":"zhangsan","age":100}

 {"id":10,"name":"lisi","age":10}

 {"id":9,"name":"wangwu","age":9}

 {"id":8,"name":"maliu","age":8}

 {"id":18,"name":"shi","age":18}

 5.3.2 Hive验证

hive>select * from db_test_tmp.hive_test_sink;

 

注意：test_kafka_source 无法在hive中查询 FAILED: SemanticException Line 0:-1 Invalid column reference 'TOK_ALLCOLREF'

如有需要可以参考https://blog.csdn.net/Zsigner/article/details/111285065

6、拓展-生成Hive Partition表

 6.1 修改source 表及sink表

  6.1.1 source 编码

tableEnv.executeSql("CREATE TABLE db_test_tmp.test_kafka_source (\n" +
                "  id INT,\n" +
                "  name STRING,\n" +
                "  age INT,\n" +
                "  statdate STRING\n" +
                ") WITH (\n" +
                " 'connector' = 'kafka'," +
                " 'topic' = 'kafka_test',\n" +
                " 'properties.bootstrap.servers' = '127.0.0.1:9092',\n" +
                " 'properties.group.id' = 'hive_test_09',\n" +
                " 'format' = 'json',\n"+
                " 'scan.startup.mode' = 'latest-offset')");

  6.1.2 sink table 编码

tableEnv.executeSql("CREATE TABLE IF NOT EXISTS  db_test_tmp.test_hive_sink (\n" +
                "  id INT,\n" +
                "  name STRING,\n" +
                "  age INT\n" +
                ") PARTITIONED BY (statdate String) STORED AS PARQUET\n" +
                "TBLPROPERTIES (\n" +
                "  'partition.time-extractor.timestamp-pattern'='$statdate 00:00:00',\n" +
                "  'stream-source.consume-order' = 'partition-time',\n" +
                "  'stream-source.enable' = 'true',\n" +
                "  'sink.partition-commit.policy.kind' = 'metastore,success-file'\n" +
                ")");

6.2 完整代码

package com.source.kafka;

import org.apache.flink.streaming.api.CheckpointingMode;
import org.apache.flink.streaming.api.environment.ExecutionCheckpointingOptions;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.table.api.EnvironmentSettings;
import org.apache.flink.table.api.SqlDialect;
import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
import org.apache.flink.table.catalog.hive.HiveCatalog;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.time.Duration;

public class HivePartitionTest {

    private final static Logger logger = LoggerFactory.getLogger(HivePartitionTest.class);

    public static void main(String[] args) throws Exception {

        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();


        EnvironmentSettings tableEnvSettings = EnvironmentSettings.newInstance()
                .useBlinkPlanner()
                .inStreamingMode()
                .build();
        StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env, tableEnvSettings);
        tableEnv.getConfig().getConfiguration().set(ExecutionCheckpointingOptions.CHECKPOINTING_MODE, CheckpointingMode.EXACTLY_ONCE);
        tableEnv.getConfig().getConfiguration().set(ExecutionCheckpointingOptions.CHECKPOINTING_INTERVAL, Duration.ofSeconds(20));

        HiveCatalog catalog = new HiveCatalog(
                "myhive",
                "db_test_tmp",
                "/usr/local/service/hive/conf"
        );
        tableEnv.registerCatalog("myhive", catalog);
        tableEnv.useCatalog("myhive");


        tableEnv.executeSql("DROP TABLE IF EXISTS db_test_tmp.test_kafka_source");
        tableEnv.executeSql("CREATE TABLE db_test_tmp.test_kafka_source (\n" +
                "  id INT,\n" +
                "  name STRING,\n" +
                "  age INT,\n" +
                "  statdate STRING\n" +
                ") WITH (\n" +
                " 'connector' = 'kafka'," +
                " 'topic' = 'kafka_test',\n" +
                " 'properties.bootstrap.servers' = '127.0.0.1:9092',\n" +
                " 'properties.group.id' = 'hive_test_09',\n" +
                " 'format' = 'json',\n"+
                " 'scan.startup.mode' = 'latest-offset')");

        tableEnv.getConfig().setSqlDialect(SqlDialect.HIVE);
        tableEnv.executeSql("DROP TABLE  db_test_tmp.test_hive_sink");
        tableEnv.executeSql("CREATE TABLE IF NOT EXISTS  db_test_tmp.test_hive_sink (\n" +
                "  id INT,\n" +
                "  name STRING,\n" +
                "  age INT\n" +
                ") PARTITIONED BY (statdate String) STORED AS PARQUET\n" +
                "TBLPROPERTIES (\n" +
                "  'partition.time-extractor.timestamp-pattern'='$statdate 00:00:00',\n" +
                "  'stream-source.consume-order' = 'partition-time',\n" +
                "  'stream-source.enable' = 'true',\n" +
                "  'sink.partition-commit.policy.kind' = 'metastore,success-file'\n" +
                ")");

        tableEnv.getConfig().setSqlDialect(SqlDialect.DEFAULT);
        tableEnv.executeSql("INSERT INTO db_test_tmp.test_hive_sink SELECT  id,name,age,DATE_FORMAT(statdate,'yyyy-MM-dd') FROM test_kafka_source");
    }
}

 6.3 验证

  6.3.1 Kafka 验证

kafka 新建topic，写入数据
#新建topic 
./kafka-topics.sh --zookeeper localhost:2181 --topic kafka_test --partitions 1 --replication-factor 1 --create


#写入数据
./kafka-console-producer.sh --broker-list localhost:9092 --topic kafka_test
>{"id":1,"name":"zhangsan","age":100,"statdate":"2021-05-09 15:42:34"}
>{"id":10,"name":"lisi","age":10,"statdate":"2021-05-10 15:46:20"}


#验证 是否写入，消费
./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic hive_test

{"id":1,"name":"zhangsan","age":100,"statdate":"2021-05-09 15:42:34"}
{"id":10,"name":"lisi","age":10,"statdate":"2021-05-10 15:46:20"}

  6.3.2 Hive验证

hive> show partitions db_test_tmp.test_hive_sink;

hive> select * from db_test_tmp.test_hive_sink;

加深学习理解：

官网链接​​：https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/connectors/table/hive/hive_read_write/

Table 参数详解：https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/connectors/table/filesystem/#streaming-sink

Zsigner
关注
  • 6
    点赞
  • 踩
  • 38
    收藏
    觉得还不错? 一键收藏
  • 打赏
    打赏
  • 26
    评论
专栏目录
消息推送平台的实时数仓？！flink消费kafka消息入到hive
BASK2311的博客
05-11 598
这数据仓库整个安装环境和调试过程确实折腾人，多次推倒重来（甚至不惜重装系统重来）。还好最后输入Kafka一条消息，在hive表里能看到一条记录，能看到结果之后，折腾或许是值得的。如果想学Java项目的，强烈推荐我的开源项目消息推送平台Austin（8K stars），可以用作毕业设计，可以用作校招，可以看看生产环境是怎么推送消息的。开源项目消息推送平台austin仓库地址：作者：Java3y链接：https://juejin.cn/post/723151921389849810。
使用FlinkCatalog将kafka的数据写入hive
qq_40382400的博客
10-16 997
【代码】使用FlinkCatalog将kafka的数据写入hive。
通过 Kafka Connect 将 Kafka 主题中的数据导出到 Hive 是一种常见的数据集成模式
最新发布
靡不有初，鲜克有终
07-24 213
通过 Kafka Connect 将 Kafka 主题中的数据导出到 Hive 是一种常见的数据集成模式，它允许用户将实时数据流无缝地存储到 Hive 中，以便进行批处理和数据仓库操作。- 对于将数据从 Kafka 导出到 Hive，你需要配置 Connector 以从 Kafka 主题读取数据，并将其写入到 Hive 表中。通过这种方式，Kafka Connect 提供了一个强大而灵活的框架，允许你将实时数据流与数据仓库技术相结合，实现数据的实时分析和历史分析。
【Flink-Kafka-To-Hive】使用 Flink 实现 Kafka 数据写入 Hive
weixin_53543905的博客
12-20 2170
需求描述：1、数据从 Kafka 写入 Hive。2、相关配置存放于 Mysql 中，通过 Mysql 进行动态读取。3、此案例中的 Kafka 是进行了 Kerberos 安全认证的，如果不需要自行修改。4、Flink 集成 Kafka 写入 Hive 需要进行 checkpoint 才能落盘至 HDFS。5、先在 Hive 中创建表然后动态获取 Hive 的表结构。6、Kafka 数据为 Json 格式，通过 FlatMap 扁平化处理后，根据表结构封装到 Row 中后完成写入。7、写入时转换成临时视图
flink sql kafka写数据到hive
weixin_41197407的博客
05-27 2521
step1:flink连接hive配置 首先确保启动hive metastore远程服务 flink安装目录下新建文件夹hive-conf,然后vi hive-site.xml，配置hive仓库地址和metastore远程服务连接地址。（此配置为了在flink集群中启动hive客户端来远程访问hive metastore服务）具体如下： 新建hive-conf文件夹： hive-site.xml配置 <configuration> <property> <nam
Flink-读Kafka写Hive表
ZacksTang的博客
08-29 926
1. 目标 使用Flink读取Kafka数据并实时写入Hive表。 2. 环境配置 EMR环境：Hadoop 3.3.3, Hive 3.1.3, Flink 1.16.0 根据官网描述： https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/hive/overview/ 当前Flink ...
Flink 读取Kafka写入Hive
weixin_34080571的博客
06-03 7959
2019独角兽企业重金招聘Python工程师标准>>> ...
1.30.Flink SQL案例将Kafka数据写入hive
涂作权的博客
07-05 3008
1.30.Flink SQL案例将Kafka数据写入hive 1.30.1.1.场景，环境，配置准备 1.30.1.2.案例代码 1.30.1.2.1.编写pom.xml文件 1.30.1.2.2.Maven工程resources下编写配置文件log4j2.properties 1.30.1.2.3.Maven工程resources下编写配置文件logback.xml 1.30.1.2.4.Maven工程resources下编写配置文件project-config-test.properties 1.30.
Flink读取Kafka数据写入Hive
weixin_40278610的博客
03-14 9045
本文针对数据库CDC(change data capture)场景设计，探讨基于Flink1.12最新版本提供的实时写入Hive的技术可行性，下面为本地IDEA程序案例可供参考。
Flink1.11 读取kafka数据写入hive，未完待续
qq_31866793的博客
07-08 2705
昨天晚上Flink1.11出了，这次改动很多，我只关心hive这一部分。 目前尝试了几个小时用代码读取hive，安装官网的文档，没成功，先蹭个热点，记录下。 先贴一下依赖吧： 注意：反正各种报错，看社区有说需要flink-clients.jar 手动去下载导入依赖 <dependencies> <dependency> <groupId>org.apache.flink</groupId> <...
FLINK 1.12.2 读取KAFKA写入HIVE的完整示例
arwenlin的博客
05-21 3744
本文只提供FLINK 1.12.2 读取KAFKA写入HIVE的完整示例，具体说明可以参考《FLINK 1.12.2 流式写入HDFS（hive）的几种方式》和《FLINK 1.12.2 读取KAFKA的2种方式》。 目录 1依赖 2代码 3发布集群 3.1打包后上传到集群执行 3.2IDE远程发布集群执行 1依赖 <dependencies> <dependency> <groupId>com.fasterxml.jackson.core</
hive+kafka安装包
06-29
kafka_2.12-1.1.0.tgz、apache-hive-0.13.1-bin.tar安装包
flink1.13 读取kafka写入hive
qq_36770189的博客
12-09 2424
代码如下所示： public class KafkaToHive { public static void main(String[] args) { StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); //1.12以后默认都是EventTime,这是过期方法,并且不在提IngestionTime //env.setStreamTi
flink 读取hive表 写入到kafka方案
qq_31866793的博客
05-19 2340
场景： 使用flinksql读取hive表数据写入到kafka hive表大约80个字段 500W数据 遇到问题： 内存爆炸，4G内存都不够，这肯定不行。 GC严重： 解决方案： 无 总结: 目前不可用。
Flink 读取 Kafka 消息写入 Hudi 表无报错但没有写入任何记录的解决方法
Laurence的技术博客
01-31 991
本问题发生的场景是：使用 Kafka Connect 的 Debezium MySQL Source Connector 将 MySQL 的 CDC 数据 （Confluent Avro 格式）接入到 Kafka 之后，通过 Flink 读取并解析这些 CDC 数据，然后写入到 Hudi 表中。在测试过程中发现：启动写入作业后，Hudi 表中迟迟没有数据写入，而 Flink 作业也没有报错。实际上，这应该是一比较常见的问题，我们的测试环境并没有特别的配置，大多数初次进行集成的开发者，大概率都会遇到这一问题，
FlinkSQL消费Kafka写入Hive表
qq_32068809的博客
10-13 7536
flinkSQL消费kafka实时写入hive表
FlinkSQL Kafka to Hive
L, there!
10-22 2223
import java.time.Duration import com.sm.common.conf.PropManager import com.sm.constants.Constants import com.sm.utils.FlinkUtils import org.apache.flink.streaming.api.{CheckpointingMode, TimeCharacteristic} import org.apache.flink.streaming.api.environme.
Flink实战之Kafka To Hive
weixin_41608066的博客
11-24 1920
背景 传统的入库任务一般借助于MapReduce或者Spark来写hive表，一般都是天级别最多小时级别的任务。随着实时性要求越来越高，传统的入库不太能满足需求。Flink完全基于流式处理，同时也支持了写Hive表。本文介绍一下如果通过FlinkSQL实现kafka数据入库hive，并能够实时可查。 Hive Catalog 由于写hive表必须基于hive catalog，所以需要注册hive catalog。同时可以在一个job内切换catalog，如果我们不想把kafka的source table注册
streamX 平台 采用flink sql 读取 kafka 的数据写入 hive
zuixue_lei的博客
05-13 1987
组件：streamX、flink 1.12、hive 2.x 一、查询 flink 官网，找寻写hive 的信息。 Apache Flink 1.12 Documentation: Hive 信息找寻步骤： 1、先找到 flink 连接外部信息的目录。 2、因为我们是要采用 flink sql 的方式，所以要看 table&sql。 3、找到 hive。 4、先看概览。 在 “概览” 中，有连接 hive 的信息：...
flink消费kafka数据后 再写入hive
05-15
可以通过Flink的Hive Sink将消费的Kafka数据写入Hive中。具体步骤如下： 1. 在Flink中创建Kafka数据源，并使用Flink的DataStream API将数据转换为需要写入Hive的格式。 2. 使用Flink的Hive Sink将数据写入Hive中。Hive Sink需要配置Hive JDBC连接信息和表结构信息。 下面是一个示例代码，可以根据实际情况进行修改： ```java // 创建Flink的执行环境 StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); // 创建Kafka数据源 Properties props = new Properties(); props.setProperty("bootstrap.servers", "localhost:9092"); props.setProperty("group.id", "test-group"); FlinkKafkaConsumer<String> consumer = new FlinkKafkaConsumer<>("test-topic", new SimpleStringSchema(), props); // 读取Kafka数据 DataStream<String> stream = env.addSource(consumer); // 转换数据格式 DataStream<Tuple2<String, Integer>> data = stream.flatMap(new FlatMapFunction<String, Tuple2<String, Integer>>() { @Override public void flatMap(String s, Collector<Tuple2<String, Integer>> collector) throws Exception { // 根据实际情况进行数据转换 String[] parts = s.split(","); collector.collect(new Tuple2<>(parts[0], Integer.parseInt(parts[1]))); } }); // 配置Hive Sink HiveConf hiveConf = new HiveConf(); hiveConf.addResource(new Path("/path/to/hive-site.xml")); HiveCatalog hiveCatalog = new HiveCatalog("myHiveCatalog", "default", hiveConf); env.registerCatalog("myHiveCatalog", hiveCatalog); env.useCatalog("myHiveCatalog"); HiveTableSink hiveTableSink = new HiveTableSink.Builder() .setPartitions(Collections.singletonList("dt='20220101'")) .setTable(new ObjectPath("default", "test")) .setHiveConf(hiveConf) .setOverwrite(true) .setStaticPartitionInsert(false) .build(); // 写入Hive data.addSink(hiveTableSink); // 执行任务 env.execute("Kafka to Hive"); ``` 需要注意的是，在执行程序前需要先创建好Hive表，并且表的结构要与代码中配置的一致。另外，Hive Sink在写入数据时会自动创建分区，因此在写入数据前需要确保分区已经存在。
评论 26
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

Zsigner

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付：¥1
获取中
扫码支付

您的余额不足，请更换扫码支付或充值

打赏作者

实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值