【FLINK】flink 读取Kafka数据写入Hive表中

最新推荐文章于 2024-09-20 09:50:22 发布
最新推荐文章于 2024-09-20 09:50:22 发布
阅读量5.7k 收藏 39
点赞数 7
分类专栏： Flink 文章标签： Flink Kafka Hive
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/Zsigner/article/details/116564806
版权

背景：对实时的历史数据持久化保存到Hdfs中，可以使用Hive Sql 追溯历史数据给业务参考多维度指标

版本信息：Flink 1.11.2，Scala 2.11，Kafka 2.12， Hive 2.3.7

 

1、引入pom.xml内容

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.zen.test</groupId>
    <artifactId>flink_test</artifactId>
    <version>1.0-SNAPSHOT</version>

    <packaging>jar</packaging>
    <properties>
        <flink.version>1.11.2</flink.version>
        <hadoop.version>2.6.0</hadoop.version>
        <hbase.version>1.4.9</hbase.version>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <maven.compiler.encoding>UTF-8</maven.compiler.encoding>
    </properties>

    <dependencies>


        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-table-common</artifactId>
            <version>${flink.version}</version>
            <exclusions>
                <exclusion>
                    <artifactId>commons-compress</artifactId>
                    <groupId>org.apache.commons</groupId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-shaded-hadoop-2-uber</artifactId>
            <version>2.6.5-10.0</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-table-planner-blink -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-table-planner-blink_2.11</artifactId>
            <version>${flink.version}</version>
            <exclusions>
                <exclusion>
                    <artifactId>scala-reflect</artifactId>
                    <groupId>org.scala-lang</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>scala-xml_2.11</artifactId>
                    <groupId>org.scala-lang.modules</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>scala-parser-combinators_2.11</artifactId>
                    <groupId>org.scala-lang.modules</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>flink-table-api-scala_2.11</artifactId>
                    <groupId>org.apache.flink</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>flink-scala_2.11</artifactId>
                    <groupId>org.apache.flink</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>flink-streaming-scala_2.11</artifactId>
                    <groupId>org.apache.flink</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>avatica-core</artifactId>
                    <groupId>org.apache.calcite.avatica</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>javassist</artifactId>
                    <groupId>org.javassist</groupId>
                </exclusion>
            </exclusions>
        </dependency>


        <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-connector-hive -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-connector-hive_2.11</artifactId>
            <version>${flink.version}</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-sql-connector-hive-2.3.6 -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-sql-connector-hive-2.3.6_2.11</artifactId>
            <version>1.11.2</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.apache.hive/hive-exec -->
        <dependency>
            <groupId>org.apache.hive</groupId>
            <artifactId>hive-exec</artifactId>
            <version>2.3.4</version>
            <exclusions>

                <exclusion>
                    <artifactId>dropwizard-metrics-hadoop-metrics2-reporter</artifactId>
                    <groupId>com.github.joshelser</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>calcite-core</artifactId>
                    <groupId>org.apache.calcite</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>libthrift</artifactId>
                    <groupId>org.apache.thrift</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hive-vector-code-gen</artifactId>
                    <groupId>org.apache.hive</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>metrics-jvm</artifactId>
                    <groupId>io.dropwizard.metrics</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>curator-client</artifactId>
                    <groupId>org.apache.curator</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>orc-core</artifactId>
                    <groupId>org.apache.orc</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>zookeeper</artifactId>
                    <groupId>org.apache.zookeeper</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>metrics-json</artifactId>
                    <groupId>io.dropwizard.metrics</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hive-storage-api</artifactId>
                    <groupId>org.apache.hive</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>calcite-druid</artifactId>
                    <groupId>org.apache.calcite</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hive-common</artifactId>
                    <groupId>org.apache.hive</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>avatica-metrics</artifactId>
                    <groupId>org.apache.calcite.avatica</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>avatica</artifactId>
                    <groupId>org.apache.calcite.avatica</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hive-service-rpc</artifactId>
                    <groupId>org.apache.hive</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hive-serde</artifactId>
                    <groupId>org.apache.hive</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hive-llap-common</artifactId>
                    <groupId>org.apache.hive</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hive-llap-client</artifactId>
                    <groupId>org.apache.hive</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hive-llap-tez</artifactId>
                    <groupId>org.apache.hive</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hive-shims-common</artifactId>
                    <groupId>org.apache.hive.shims</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hadoop-yarn-server-resourcemanager</artifactId>
                    <groupId>org.apache.hadoop</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hive-shims-0.23</artifactId>
                    <groupId>org.apache.hive.shims</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hive-shims-scheduler</artifactId>
                    <groupId>org.apache.hive.shims</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hive-shims</artifactId>
                    <groupId>org.apache.hive</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>guava</artifactId>
                    <groupId>com.google.guava</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>commons-codec</artifactId>
                    <groupId>commons-codec</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>commons-logging</artifactId>
                    <groupId>commons-logging</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>junit</artifactId>
                    <groupId>junit</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>antlr-runtime</artifactId>
                    <groupId>org.antlr</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>slf4j-api</artifactId>
                    <groupId>org.slf4j</groupId>
                </exclusion>
            </exclusions>
        </dependency>


        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-table-api-java-bridge_2.11</artifactId>
            <version>${flink.version}</version>
            <exclusions>
                <exclusion>
                    <artifactId>flink-runtime_2.11</artifactId>
                    <groupId>org.apache.flink</groupId>
                </exclusion>
            </exclusions>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.apache.hbase/hbase-client -->
        <dependency>
            <groupId>org.apache.hbase</groupId>
            <artifactId>hbase-client</artifactId>
            <version>${hbase.version}</version>
            <exclusions>
                <exclusion>
                    <artifactId>metrics-core</artifactId>
                    <groupId>com.yammer.metrics</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>zookeeper</artifactId>
                    <groupId>org.apache.zookeeper</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>apacheds-i18n</artifactId>
                    <groupId>org.apache.directory.server</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>api-asn1-api</artifactId>
                    <groupId>org.apache.directory.api</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>api-util</artifactId>
                    <groupId>org.apache.directory.api</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>apacheds-kerberos-codec</artifactId>
                    <groupId>org.apache.directory.server</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hadoop-auth</artifactId>
                    <groupId>org.apache.hadoop</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hadoop-mapreduce-client-core</artifactId>
                    <groupId>org.apache.hadoop</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>commons-codec</artifactId>
                    <groupId>commons-codec</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hadoop-common</artifactId>
                    <groupId>org.apache.hadoop</groupId>
                </exclusion>
            </exclusions>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.apache.hbase/hbase-common -->
        <dependency>
            <groupId>org.apache.hbase</groupId>
            <artifactId>hbase-common</artifactId>
            <version>${hbase.version}</version>
            <exclusions>
                <exclusion>
                    <artifactId>avro</artifactId>
                    <groupId>org.apache.avro</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hadoop-mapreduce-client-core</artifactId>
                    <groupId>org.apache.hadoop</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>commons-codec</artifactId>
                    <groupId>commons-codec</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hadoop-common</artifactId>
                    <groupId>org.apache.hadoop</groupId>
                </exclusion>
            </exclusions>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.apache.hbase/hbase-server -->
        <dependency>
            <groupId>org.apache.hbase</groupId>
            <artifactId>hbase-server</artifactId>
            <version>${hbase.version}</version>
            <exclusions>
                <exclusion>
                    <artifactId>metrics-core</artifactId>
                    <groupId>com.yammer.metrics</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>zookeeper</artifactId>
                    <groupId>org.apache.zookeeper</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>metrics-core</artifactId>
                    <groupId>io.dropwizard.metrics</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hadoop-auth</artifactId>
                    <groupId>org.apache.hadoop</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hadoop-mapreduce-client-core</artifactId>
                    <groupId>org.apache.hadoop</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>avro</artifactId>
                    <groupId>org.apache.avro</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hbase-common</artifactId>
                    <groupId>org.apache.hbase</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>commons-codec</artifactId>
                    <groupId>commons-codec</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>commons-httpclient</artifactId>
                    <groupId>commons-httpclient</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>commons-logging</artifactId>
                    <groupId>commons-logging</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hadoop-client</artifactId>
                    <groupId>org.apache.hadoop</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hadoop-common</artifactId>
                    <groupId>org.apache.hadoop</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hadoop-hdfs</artifactId>
                    <groupId>org.apache.hadoop</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>httpclient</artifactId>
                    <groupId>org.apache.httpcomponents</groupId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>${hadoop.version}</version>
            <exclusions>
                <exclusion>
                    <artifactId>hadoop-yarn-client</artifactId>
                    <groupId>org.apache.hadoop</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hadoop-yarn-common</artifactId>
                    <groupId>org.apache.hadoop</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hadoop-yarn-server-common</artifactId>
                    <groupId>org.apache.hadoop</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hadoop-yarn-api</artifactId>
                    <groupId>org.apache.hadoop</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>zookeeper</artifactId>
                    <groupId>org.apache.zookeeper</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hadoop-mapreduce-client-common</artifactId>
                    <groupId>org.apache.hadoop</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hadoop-mapreduce-client-app</artifactId>
                    <groupId>org.apache.hadoop</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hadoop-mapreduce-client-jobclient</artifactId>
                    <groupId>org.apache.hadoop</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hadoop-mapreduce-client-core</artifactId>
                    <groupId>org.apache.hadoop</groupId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>com.typesafe</groupId>
            <artifactId>config</artifactId>
            <version>1.3.3</version>
        </dependency>
        <dependency>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpclient</artifactId>
            <version>4.5.1</version>
            <exclusions>
                <exclusion>
                    <artifactId>commons-codec</artifactId>
                    <groupId>commons-codec</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>httpcore</artifactId>
                    <groupId>org.apache.httpcomponents</groupId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>com.alibaba</groupId>
            <artifactId>fastjson</artifactId>
            <version>1.2.74</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-clients -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-clients_2.12</artifactId>
            <version>${flink.version}</version>
            <exclusions>
                <exclusion>
                    <artifactId>flink-core</artifactId>
                    <groupId>org.apache.flink</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>commons-cli</artifactId>
                    <groupId>commons-cli</groupId>
                </exclusion>
            </exclusions>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-streaming-java -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-streaming-java_2.12</artifactId>
            <version>${flink.version}</version>
            <exclusions>
                <exclusion>
                    <artifactId>flink-core</artifactId>
                    <groupId>org.apache.flink</groupId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>log4j</groupId>
            <artifactId>log4j</artifactId>
            <version>1.2.17</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-runtime -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-runtime_2.12</artifactId>
            <version>${flink.version}</version>
            <exclusions>

                <exclusion>
                    <artifactId>grizzled-slf4j_2.12</artifactId>
                    <groupId>org.clapper</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>flink-core</artifactId>
                    <groupId>org.apache.flink</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>commons-cli</artifactId>
                    <groupId>commons-cli</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>slf4j-api</artifactId>
                    <groupId>org.slf4j</groupId>
                </exclusion>
            </exclusions>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-connector-filesystem -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-connector-filesystem_2.12</artifactId>
            <version>${flink.version}</version>
        </dependency>


        <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-connector-kafka -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-connector-kafka_2.12</artifactId>
            <version>1.11.2</version>
        </dependency>


        <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-avro -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-avro</artifactId>
            <version>${flink.version}</version>
            <exclusions>
                <exclusion>
                    <artifactId>avro</artifactId>
                    <groupId>org.apache.avro</groupId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-avro-confluent-registry</artifactId>
            <version>${flink.version}</version>
        </dependency>


        <dependency>
            <groupId>org.apache.kudu</groupId>
            <artifactId>kudu-client</artifactId>
            <version>1.7.0</version>
            <exclusions>
                <exclusion>
                    <artifactId>slf4j-api</artifactId>
                    <groupId>org.slf4j</groupId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.12</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-hadoop-fs</artifactId>
            <version>${flink.version}</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common -->
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>${hadoop.version}</version>
            <exclusions>
                <exclusion>
                    <artifactId>curator-framework</artifactId>
                    <groupId>org.apache.curator</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>curator-client</artifactId>
                    <groupId>org.apache.curator</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>curator-recipes</artifactId>
                    <groupId>org.apache.curator</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>htrace-core</artifactId>
                    <groupId>org.htrace</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>zookeeper</artifactId>
                    <groupId>org.apache.zookeeper</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>commons-httpclient</artifactId>
                    <groupId>commons-httpclient</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>jets3t</artifactId>
                    <groupId>net.java.dev.jets3t</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hadoop-auth</artifactId>
                    <groupId>org.apache.hadoop</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>commons-configuration</artifactId>
                    <groupId>commons-configuration</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>commons-el</artifactId>
                    <groupId>commons-el</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>commons-beanutils</artifactId>
                    <groupId>commons-beanutils</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>commons-digester</artifactId>
                    <groupId>commons-digester</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>commons-beanutils-core</artifactId>
                    <groupId>commons-beanutils</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>jasper-runtime</artifactId>
                    <groupId>tomcat</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>jersey-json</artifactId>
                    <groupId>com.sun.jersey</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>jackson-jaxrs</artifactId>
                    <groupId>org.codehaus.jackson</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>jackson-xc</artifactId>
                    <groupId>org.codehaus.jackson</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>avro</artifactId>
                    <groupId>org.apache.avro</groupId>
                </exclusion>

                <exclusion>
                    <artifactId>apacheds-i18n</artifactId>
                    <groupId>org.apache.directory.server</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>apacheds-kerberos-codec</artifactId>
                    <groupId>org.apache.directory.server</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>api-util</artifactId>
                    <groupId>org.apache.directory.api</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>api-asn1-api</artifactId>
                    <groupId>org.apache.directory.api</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>guava</artifactId>
                    <groupId>com.google.guava</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>commons-collections</artifactId>
                    <groupId>commons-collections</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>commons-logging</artifactId>
                    <groupId>commons-logging</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>commons-compress</artifactId>
                    <groupId>org.apache.commons</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>commons-math3</artifactId>
                    <groupId>org.apache.commons</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>slf4j-api</artifactId>
                    <groupId>org.slf4j</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>slf4j-log4j12</artifactId>
                    <groupId>org.slf4j</groupId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-hdfs</artifactId>
            <version>${hadoop.version}</version>
            <exclusions>
                <exclusion>
                    <artifactId>htrace-core</artifactId>
                    <groupId>org.htrace</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>jasper-runtime</artifactId>
                    <groupId>tomcat</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>commons-el</artifactId>
                    <groupId>commons-el</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>guava</artifactId>
                    <groupId>com.google.guava</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>commons-logging</artifactId>
                    <groupId>commons-logging</groupId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>joda-time</groupId>
            <artifactId>joda-time</artifactId>
            <version>2.9.1</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/com.stumbleupon/async -->
        <dependency>
            <groupId>com.stumbleupon</groupId>
            <artifactId>async</artifactId>
            <version>1.4.1</version>
            <exclusions>
                <exclusion>
                    <artifactId>slf4j-api</artifactId>
                    <groupId>org.slf4j</groupId>
                </exclusion>
            </exclusions>
        </dependency>


        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <version>5.1.35</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/redis.clients/jedis -->
        <dependency>
            <groupId>redis.clients</groupId>
            <artifactId>jedis</artifactId>
            <version>2.9.0</version>
        </dependency>

    </dependencies>

    <build>
        <resources>
            <resource>
                <directory>src/main/resources</directory>
                <includes>
                    <!--包含文件夹以及子文件夹下所有资源-->
                    <include>**/*.*</include>
                </includes>
            </resource>
        </resources>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>2.5.1</version>
                <configuration>
                    <source>${maven.compiler.source}</source>
                    <target>${maven.compiler.target}</target>
                    <encoding>UTF-8</encoding>
                    <!--<encoding>${project.build.sourceEncoding}</encoding>-->
                </configuration>
            </plugin>

            <plugin>
                <groupId>net.alchim31.maven</groupId>
                <artifactId>scala-maven-plugin</artifactId>
                <version>3.2.0</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>compile</goal>
                            <goal>testCompile</goal>
                        </goals>
                        <configuration>
                            <args>
                                <!--<arg>-make:transitive</arg>-->
                                <arg>-dependencyfile</arg>
                                <arg>${project.build.directory}/.scala_dependencies</arg>
                            </args>

                        </configuration>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.18.1</version>
                <configuration>
                    <useFile>false</useFile>
                    <disableXmlReport>true</disableXmlReport>
                    <includes>
                        <include>**/*com.xiaoe.Test.*</include>
                        <include>**/*Suite.*</include>
                    </includes>
                </configuration>
            </plugin>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>3.0.0</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <filters>
                                <filter>
                                    <artifact>*:*</artifact>
                                    <excludes>
                                        <!--
                                        zip -d learn_spark.jar META-INF/*.RSA META-INF/*.DSA META-INF/*.SF
                                        -->
                                        <exclude>META-INF/*.SF</exclude>
                                        <exclude>META-INF/*.DSA</exclude>
                                        <exclude>META-INF/*.RSA</exclude>
                                    </excludes>
                                </filter>
                            </filters>
                           
                        </configuration>
                    </execution>

                </executions>
            </plugin>
        </plugins>
    </build>

</project>

2、使用Table环境

StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();



        EnvironmentSettings tableEnvSettings = EnvironmentSettings.newInstance()
                .useBlinkPlanner()
                .inStreamingMode()
                .build();
        StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env, tableEnvSettings);
        tableEnv.getConfig().getConfiguration().set(ExecutionCheckpointingOptions.CHECKPOINTING_MODE, CheckpointingMode.EXACTLY_ONCE);
        tableEnv.getConfig().getConfiguration().set(ExecutionCheckpointingOptions.CHECKPOINTING_INTERVAL, Duration.ofSeconds(20));

3、使用HiveCatalog

HiveCatalog catalog = new HiveCatalog(
                "myhive",
                "db_test_tmp",
                "/usr/local/service/hive/conf"
        );
        tableEnv.registerCatalog("myhive", catalog);
        tableEnv.useCatalog("myhive");

4、完整编码

package com.source.kafka;

import org.apache.flink.streaming.api.CheckpointingMode;
import org.apache.flink.streaming.api.environment.ExecutionCheckpointingOptions;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.table.api.EnvironmentSettings;
import org.apache.flink.table.api.SqlDialect;
import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
import org.apache.flink.table.catalog.hive.HiveCatalog;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.time.Duration;

public class HiveTest {

    private final static Logger logger = LoggerFactory.getLogger(hiveTest.class);

    public static void main(String[] args) throws Exception {

        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        EnvironmentSettings tableEnvSettings = EnvironmentSettings.newInstance()
                .useBlinkPlanner()
                .inStreamingMode()
                .build();
        StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env, tableEnvSettings);
        tableEnv.getConfig().getConfiguration().set(ExecutionCheckpointingOptions.CHECKPOINTING_MODE, CheckpointingMode.EXACTLY_ONCE);
        tableEnv.getConfig().getConfiguration().set(ExecutionCheckpointingOptions.CHECKPOINTING_INTERVAL, Duration.ofSeconds(20));

        HiveCatalog catalog = new HiveCatalog(
                "myhive",
                "db_test_tmp",
                "/usr/local/service/hive/conf"
        );
        tableEnv.registerCatalog("myhive", catalog);
        tableEnv.useCatalog("myhive");


        tableEnv.executeSql("DROP TABLE IF EXISTS db_test_tmp.test_kafka_source");
        tableEnv.executeSql("CREATE TABLE db_test_tmp.test_kafka_source (\n" +
                "  id INT,\n" +
                "  name STRING,\n" +
                "  age INT\n" +
                ") WITH (\n" +
                " 'connector' = 'kafka'," +
                " 'topic' = 'hive_test',\n" +
                " 'properties.bootstrap.servers' = '127.0.0.1:9092',\n" +
                " 'properties.group.id' = 'group_test_01',\n" +
                " 'format' = 'json',\n"+
                " 'is_generic' = 'false',\n"+
                " 'scan.startup.mode' = 'latest-offset')");

        tableEnv.getConfig().setSqlDialect(SqlDialect.HIVE);
        tableEnv.executeSql("DROP TABLE IF EXISTS db_test_tmp.test_hive_sink");
        tableEnv.executeSql("CREATE TABLE IF NOT EXISTS  db_test_tmp.test_hive_sink (\n" +
                "  id INT,\n" +
                "  name STRING,\n" +
                "  age INT\n" +
                ") STORED AS PARQUET\n" +
                "TBLPROPERTIES (\n" +
                "  'sink.partition-commit.policy.kind' = 'metastore,success-file'\n" +
                ")");

        tableEnv.getConfig().setSqlDialect(SqlDialect.DEFAULT);
        tableEnv.executeSql("INSERT INTO db_test_tmp.test_hive_sink SELECT id,name,age FROM test_kafka_source");
    }
}

5、打包、运行、验证

 5.1 打包

mvn clean install

 5.2 运行 

flink run -m yarn-cluster -ynm test -ys 1   -ytm 1024m   -yD state.checkpoints.dir=hdfs://127.0.0.1:4007/flink/checkpoints   -c com.source.kafka.HiveTest  /tmp/original-flink_test-1.0-SNAPSHOT.jar

 5.3 验证

  5.3.1 kafka 验证

kafka 新建topic，写入数据
#新建topic 
./kafka-topics.sh --zookeeper localhost:2181 --topic hive_test --partitions 1 --replication-factor 1 --create


#写入数据
./kafka-console-producer.sh --broker-list localhost:9092 --topic hive_test
> {"id":1,"name":"zhangsan","age":100}
> {"id":10,"name":"lisi","age":10}
> {"id":9,"name":"wangwu","age":9}
> {"id":8,"name":"maliu","age":8}
> {"id":18,"name":"shi","age":18}


#验证 是否写入，消费
./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic hive_test

 {"id":1,"name":"zhangsan","age":100}

 {"id":10,"name":"lisi","age":10}

 {"id":9,"name":"wangwu","age":9}

 {"id":8,"name":"maliu","age":8}

 {"id":18,"name":"shi","age":18}

 5.3.2 Hive验证

hive>select * from db_test_tmp.hive_test_sink;

 

注意：test_kafka_source 无法在hive中查询 FAILED: SemanticException Line 0:-1 Invalid column reference 'TOK_ALLCOLREF'

如有需要可以参考https://blog.csdn.net/Zsigner/article/details/111285065

6、拓展-生成Hive Partition表

 6.1 修改source 表及sink表

  6.1.1 source 编码

tableEnv.executeSql("CREATE TABLE db_test_tmp.test_kafka_source (\n" +
                "  id INT,\n" +
                "  name STRING,\n" +
                "  age INT,\n" +
                "  statdate STRING\n" +
                ") WITH (\n" +
                " 'connector' = 'kafka'," +
                " 'topic' = 'kafka_test',\n" +
                " 'properties.bootstrap.servers' = '127.0.0.1:9092',\n" +
                " 'properties.group.id' = 'hive_test_09',\n" +
                " 'format' = 'json',\n"+
                " 'scan.startup.mode' = 'latest-offset')");

  6.1.2 sink table 编码

tableEnv.executeSql("CREATE TABLE IF NOT EXISTS  db_test_tmp.test_hive_sink (\n" +
                "  id INT,\n" +
                "  name STRING,\n" +
                "  age INT\n" +
                ") PARTITIONED BY (statdate String) STORED AS PARQUET\n" +
                "TBLPROPERTIES (\n" +
                "  'partition.time-extractor.timestamp-pattern'='$statdate 00:00:00',\n" +
                "  'stream-source.consume-order' = 'partition-time',\n" +
                "  'stream-source.enable' = 'true',\n" +
                "  'sink.partition-commit.policy.kind' = 'metastore,success-file'\n" +
                ")");

6.2 完整代码

package com.source.kafka;

import org.apache.flink.streaming.api.CheckpointingMode;
import org.apache.flink.streaming.api.environment.ExecutionCheckpointingOptions;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.table.api.EnvironmentSettings;
import org.apache.flink.table.api.SqlDialect;
import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
import org.apache.flink.table.catalog.hive.HiveCatalog;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.time.Duration;

public class HivePartitionTest {

    private final static Logger logger = LoggerFactory.getLogger(HivePartitionTest.class);

    public static void main(String[] args) throws Exception {

        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();


        EnvironmentSettings tableEnvSettings = EnvironmentSettings.newInstance()
                .useBlinkPlanner()
                .inStreamingMode()
                .build();
        StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env, tableEnvSettings);
        tableEnv.getConfig().getConfiguration().set(ExecutionCheckpointingOptions.CHECKPOINTING_MODE, CheckpointingMode.EXACTLY_ONCE);
        tableEnv.getConfig().getConfiguration().set(ExecutionCheckpointingOptions.CHECKPOINTING_INTERVAL, Duration.ofSeconds(20));

        HiveCatalog catalog = new HiveCatalog(
                "myhive",
                "db_test_tmp",
                "/usr/local/service/hive/conf"
        );
        tableEnv.registerCatalog("myhive", catalog);
        tableEnv.useCatalog("myhive");


        tableEnv.executeSql("DROP TABLE IF EXISTS db_test_tmp.test_kafka_source");
        tableEnv.executeSql("CREATE TABLE db_test_tmp.test_kafka_source (\n" +
                "  id INT,\n" +
                "  name STRING,\n" +
                "  age INT,\n" +
                "  statdate STRING\n" +
                ") WITH (\n" +
                " 'connector' = 'kafka'," +
                " 'topic' = 'kafka_test',\n" +
                " 'properties.bootstrap.servers' = '127.0.0.1:9092',\n" +
                " 'properties.group.id' = 'hive_test_09',\n" +
                " 'format' = 'json',\n"+
                " 'scan.startup.mode' = 'latest-offset')");

        tableEnv.getConfig().setSqlDialect(SqlDialect.HIVE);
        tableEnv.executeSql("DROP TABLE  db_test_tmp.test_hive_sink");
        tableEnv.executeSql("CREATE TABLE IF NOT EXISTS  db_test_tmp.test_hive_sink (\n" +
                "  id INT,\n" +
                "  name STRING,\n" +
                "  age INT\n" +
                ") PARTITIONED BY (statdate String) STORED AS PARQUET\n" +
                "TBLPROPERTIES (\n" +
                "  'partition.time-extractor.timestamp-pattern'='$statdate 00:00:00',\n" +
                "  'stream-source.consume-order' = 'partition-time',\n" +
                "  'stream-source.enable' = 'true',\n" +
                "  'sink.partition-commit.policy.kind' = 'metastore,success-file'\n" +
                ")");

        tableEnv.getConfig().setSqlDialect(SqlDialect.DEFAULT);
        tableEnv.executeSql("INSERT INTO db_test_tmp.test_hive_sink SELECT  id,name,age,DATE_FORMAT(statdate,'yyyy-MM-dd') FROM test_kafka_source");
    }
}

 6.3 验证

  6.3.1 Kafka 验证

kafka 新建topic，写入数据
#新建topic 
./kafka-topics.sh --zookeeper localhost:2181 --topic kafka_test --partitions 1 --replication-factor 1 --create


#写入数据
./kafka-console-producer.sh --broker-list localhost:9092 --topic kafka_test
>{"id":1,"name":"zhangsan","age":100,"statdate":"2021-05-09 15:42:34"}
>{"id":10,"name":"lisi","age":10,"statdate":"2021-05-10 15:46:20"}


#验证 是否写入，消费
./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic hive_test

{"id":1,"name":"zhangsan","age":100,"statdate":"2021-05-09 15:42:34"}
{"id":10,"name":"lisi","age":10,"statdate":"2021-05-10 15:46:20"}

  6.3.2 Hive验证

hive> show partitions db_test_tmp.test_hive_sink;

hive> select * from db_test_tmp.test_hive_sink;

加深学习理解：

官网链接​​：https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/connectors/table/hive/hive_read_write/

Table 参数详解：https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/connectors/table/filesystem/#streaming-sink

flink sql kafka写数据到hive
weixin_41197407的博客
05-27 2591
step1:flink连接hive配置 首先确保启动hive metastore远程服务 flink安装目录下新建文件夹hive-conf,然后vi hive-site.xml，配置hive仓库地址和metastore远程服务连接地址。（此配置为了在flink集群中启动hive客户端来远程访问hive metastore服务）具体如下： 新建hive-conf文件夹： hive-site.xml配置 <configuration> <property> <nam
flink1.13解决读取kafka数据写入hive中hive客户端查询不到数据问题
m0_37592814的博客
06-14 2637
flink1.13版本前可参考如下方式:https://blog.csdn.net/m0_37592814/article/details/108044830 flink1.1
Flink-读Kafka写Hive表
ZacksTang的博客
08-29 1139
1. 目标 使用Flink读取Kafka数据并实时写入Hive表。 2. 环境配置 EMR环境：Hadoop 3.3.3, Hive 3.1.3, Flink 1.16.0 根据官网描述： https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/hive/overview/ 当前Flink ...
【flink-stream开发中，将 kafka 的数据写入 hbase 中】
最新发布
2203_75584759的博客
09-20 646
创建hive外部表关联hbase表（先创建hbase表，再创建hive外部表关联）
Flink 读取Kafka写入Hive
weixin_34080571的博客
06-03 7985
2019独角兽企业重金招聘Python工程师标准>>> ...
1.30.Flink SQL案例将Kafka数据写入hive
涂作权的博客
07-05 3088
1.30.Flink SQL案例将Kafka数据写入hive 1.30.1.1.场景，环境，配置准备 1.30.1.2.案例代码 1.30.1.2.1.编写pom.xml文件 1.30.1.2.2.Maven工程resources下编写配置文件log4j2.properties 1.30.1.2.3.Maven工程resources下编写配置文件logback.xml 1.30.1.2.4.Maven工程resources下编写配置文件project-config-test.properties 1.30.
Flink读取Kafka数据写入Hive
weixin_40278610的博客
03-14 9084
本文针对数据库CDC(change data capture)场景设计，探讨基于Flink1.12最新版本提供的实时写入Hive的技术可行性，下面为本地IDEA程序案例可供参考。
flink1.11.0读取kafka数据写入hive中hive无分区信息及读取不到数据解决
m0_37592814的博客
08-29 4631
一、前言 在上一博客中写了flink1.11.0读取kafka数据写入到hive中，发现hive中无法查询flink通过scala写入的数据，搜了些资料查找原因，参考了下文章:https://zhuanlan.zhihu.com/p/157899980里无法读取hive数据的原因，但里面比较明确给出的解决方案是修改源码，我觉得太麻烦了。查了下官方和阅读些flink源码，终于找到一种我认为比较便捷的解决方案，具体分析方法如下： 完整的flink读取kafka数据动态写出hive，实现实时数仓的代码dem.
Flink1.11.0读取kafka数据动态写入hive中(更新-解决hive查询不到数据问题)
m0_37592814的博客
08-16 9744
一、主要流程 flink 1.11.0 hive 2.3.4 kafka 2.11 hadoop 2.7.2 scala 2.1.11 流批混合，读取kafka 数据量写入到hive中 二、主要步骤 1.在flink sql 客户端中创建hive 分区表 flink sql 客户端配置在上一篇文章https://blog.csdn.net/m0_37592814/article/details/108038823 建表语句 use wm; CREATE TABLE ods_...
Flink1.11 读取kafka数据写入hive，未完待续
qq_31866793的博客
07-08 2716
昨天晚上Flink1.11出了，这次改动很多，我只关心hive这一部分。 目前尝试了几个小时用代码读取hive，安装官网的文档，没成功，先蹭个热点，记录下。 先贴一下依赖吧： 注意：反正各种报错，看社区有说需要flink-clients.jar 手动去下载导入依赖 <dependencies> <dependency> <groupId>org.apache.flink</groupId> <...
FLINK 1.12.2 读取KAFKA写入HIVE的完整示例
arwenlin的博客
05-21 3801
本文只提供FLINK 1.12.2 读取KAFKA写入HIVE的完整示例，具体说明可以参考《FLINK 1.12.2 流式写入HDFS（hive）的几种方式》和《FLINK 1.12.2 读取KAFKA的2种方式》。 目录 1依赖 2代码 3发布集群 3.1打包后上传到集群执行 3.2IDE远程发布集群执行 1依赖 <dependencies> <dependency> <groupId>com.fasterxml.jackson.core</
hive+kafka安装包
06-29
kafka_2.12-1.1.0.tgz、apache-hive-0.13.1-bin.tar安装包
FlinkSQL消费Kafka写入Hive表
qq_32068809的博客
10-13 7690
flinkSQL消费kafka实时写入hive表
FlinkSQL Kafka to Hive
L, there!
10-22 2254
import java.time.Duration import com.sm.common.conf.PropManager import com.sm.constants.Constants import com.sm.utils.FlinkUtils import org.apache.flink.streaming.api.{CheckpointingMode, TimeCharacteristic} import org.apache.flink.streaming.api.environme.
【Flink-Kafka-To-Hive】使用 Flink 实现 Kafka 数据写入 Hive
weixin_53543905的博客
12-20 2461
需求描述：1、数据从 Kafka 写入 Hive。2、相关配置存放于 Mysql 中，通过 Mysql 进行动态读取。3、此案例中的 Kafka 是进行了 Kerberos 安全认证的，如果不需要自行修改。4、Flink 集成 Kafka 写入 Hive 需要进行 checkpoint 才能落盘至 HDFS。5、先在 Hive 中创建表然后动态获取 Hive 的表结构。6、Kafka 数据为 Json 格式，通过 FlatMap 扁平化处理后，根据表结构封装到 Row 中后完成写入。7、写入时转换成临时视图
Flink实战之Kafka To Hive
weixin_41608066的博客
11-24 1948
背景 传统的入库任务一般借助于MapReduce或者Spark来写hive表，一般都是天级别最多小时级别的任务。随着实时性要求越来越高，传统的入库不太能满足需求。Flink完全基于流式处理，同时也支持了写Hive表。本文介绍一下如果通过FlinkSQL实现kafka数据入库hive，并能够实时可查。 Hive Catalog 由于写hive表必须基于hive catalog，所以需要注册hive catalog。同时可以在一个job内切换catalog，如果我们不想把kafka的source table注册
streamX 平台 采用flink sql 读取 kafka 的数据写入 hive
zuixue_lei的博客
05-13 2060
组件：streamX、flink 1.12、hive 2.x 一、查询 flink 官网，找寻写hive 的信息。 Apache Flink 1.12 Documentation: Hive 信息找寻步骤： 1、先找到 flink 连接外部信息的目录。 2、因为我们是要采用 flink sql 的方式，所以要看 table&sql。 3、找到 hive。 4、先看概览。 在 “概览” 中，有连接 hive 的信息：...
消息推送平台的实时数仓？！flink消费kafka消息入到hive
BASK2311的博客
05-11 637
这数据仓库整个安装环境和调试过程确实折腾人，多次推倒重来（甚至不惜重装系统重来）。还好最后输入Kafka一条消息，在hive表里能看到一条记录，能看到结果之后，折腾或许是值得的。如果想学Java项目的，强烈推荐我的开源项目消息推送平台Austin（8K stars），可以用作毕业设计，可以用作校招，可以看看生产环境是怎么推送消息的。开源项目消息推送平台austin仓库地址：作者：Java3y链接：https://juejin.cn/post/723151921389849810。
flink消费kafka数据后 再写入hive
05-15
可以通过Flink的Hive Sink将消费的Kafka数据写入Hive中。具体步骤如下： 1. 在Flink中创建Kafka数据源，并使用Flink的DataStream API将数据转换为需要写入Hive的格式。 2. 使用Flink的Hive Sink将数据写入Hive中。...
评论 26
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

Zsigner

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付：¥1
获取中
扫码支付

您的余额不足，请更换扫码支付或充值

打赏作者

实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值