Flink Kafka读取数据并写入Redis实现Exactly-Once

最新推荐文章于 2024-06-24 08:59:02 发布
最新推荐文章于 2024-06-24 08:59:02 发布
阅读量2.1k 收藏 8
点赞数 1
分类专栏： Flink
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/zuochang_liu/article/details/106507609
版权

博客《2020大数据面试真题(持续更新)》已迁移至微信公众号！！！     喜欢的朋友可以关注

最新面试题将会在微信公众号更新！

关注公众号即可获得免费学习资源，获得免费指导！！！

 

pom.xml

<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.  See the NOTICE file
distributed with this work for additional information
regarding copyright ownership.  The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License.  You may obtain a copy of the License at

  http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>

	<groupId>org.myorg.quickstart</groupId>
	<artifactId>quickstart</artifactId>
	<version>0.1</version>
	<packaging>jar</packaging>

	<name>Flink Quickstart Job</name>
	<url>http://www.myorganization.org</url>

	<properties>
		<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
		<flink.version>1.9.1</flink.version>
		<java.version>1.8</java.version>
		<scala.binary.version>2.11</scala.binary.version>
		<maven.compiler.source>${java.version}</maven.compiler.source>
		<maven.compiler.target>${java.version}</maven.compiler.target>
	</properties>

	<repositories>
		<repository>
			<id>apache.snapshots</id>
			<name>Apache Development Snapshot Repository</name>
			<url>https://repository.apache.org/content/repositories/snapshots/</url>
			<releases>
				<enabled>false</enabled>
			</releases>
			<snapshots>
				<enabled>true</enabled>
			</snapshots>
		</repository>
	</repositories>

	<dependencies>
		<!-- Apache Flink dependencies -->
		<!-- These dependencies are provided, because they should not be packaged into the JAR file. -->
		<dependency>
			<groupId>org.apache.flink</groupId>
			<artifactId>flink-java</artifactId>
			<version>${flink.version}</version>
			<scope>provided</scope>
		</dependency>
		<dependency>
			<groupId>org.apache.flink</groupId>
			<artifactId>flink-streaming-java_${scala.binary.version}</artifactId>
			<version>${flink.version}</version>
			<scope>provided</scope>
		</dependency>

		<!-- Add connector dependencies here. They must be in the default scope (compile). -->

		<!-- Example:

		<dependency>
			<groupId>org.apache.flink</groupId>
			<artifactId>flink-connector-kafka-0.10_${scala.binary.version}</artifactId>
			<version>${flink.version}</version>
		</dependency>
		-->

		<!-- Add logging framework, to produce console output when running in the IDE. -->
		<!-- These dependencies are excluded from the application JAR by default. -->
		<dependency>
			<groupId>org.slf4j</groupId>
			<artifactId>slf4j-log4j12</artifactId>
			<version>1.7.7</version>
			<scope>runtime</scope>
		</dependency>
		<dependency>
			<groupId>log4j</groupId>
			<artifactId>log4j</artifactId>
			<version>1.2.17</version>
			<scope>runtime</scope>
		</dependency>

		<!-- 高效的异步HttpClient -->
		<dependency>
			<groupId>org.apache.httpcomponents</groupId>
			<artifactId>httpasyncclient</artifactId>
			<version>4.1.4</version>
		</dependency>

		<!-- -->
		<dependency>
			<groupId>com.alibaba</groupId>
			<artifactId>fastjson</artifactId>
			<version>1.2.51</version>
		</dependency>


		<dependency>
			<groupId>org.apache.flink</groupId>
			<artifactId>flink-connector-kafka_${scala.binary.version}</artifactId>
			<version>${flink.version}</version>
		</dependency>

		<dependency>
			<groupId>org.apache.flink</groupId>
			<artifactId>flink-connector-redis_${scala.binary.version}</artifactId>
			<version>1.1.5</version>
		</dependency>

	</dependencies>

	<build>
		<plugins>

			<!-- Java Compiler -->
			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-compiler-plugin</artifactId>
				<version>3.1</version>
				<configuration>
					<source>${java.version}</source>
					<target>${java.version}</target>
				</configuration>
			</plugin>

			<!-- We use the maven-shade plugin to create a fat jar that contains all necessary dependencies. -->
			<!-- Change the value of <mainClass>...</mainClass> if your program entry point changes. -->
			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-shade-plugin</artifactId>
				<version>3.0.0</version>
				<executions>
					<!-- Run shade goal on package phase -->
					<execution>
						<phase>package</phase>
						<goals>
							<goal>shade</goal>
						</goals>
						<configuration>
							<artifactSet>
								<excludes>
									<exclude>org.apache.flink:force-shading</exclude>
									<exclude>com.google.code.findbugs:jsr305</exclude>
									<exclude>org.slf4j:*</exclude>
									<exclude>log4j:*</exclude>
								</excludes>
							</artifactSet>
							<filters>
								<filter>
									<!-- Do not copy the signatures in the META-INF folder.
									Otherwise, this might cause SecurityExceptions when using the JAR. -->
									<artifact>*:*</artifact>
									<excludes>
										<exclude>META-INF/*.SF</exclude>
										<exclude>META-INF/*.DSA</exclude>
										<exclude>META-INF/*.RSA</exclude>
									</excludes>
								</filter>
							</filters>
							<transformers>
								<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
									<mainClass>org.myorg.quickstart.StreamingJob</mainClass>
								</transformer>
							</transformers>
						</configuration>
					</execution>
				</executions>
			</plugin>
		</plugins>

		<pluginManagement>
			<plugins>

				<!-- This improves the out-of-the-box experience in Eclipse by resolving some warnings. -->
				<plugin>
					<groupId>org.eclipse.m2e</groupId>
					<artifactId>lifecycle-mapping</artifactId>
					<version>1.0.0</version>
					<configuration>
						<lifecycleMappingMetadata>
							<pluginExecutions>
								<pluginExecution>
									<pluginExecutionFilter>
										<groupId>org.apache.maven.plugins</groupId>
										<artifactId>maven-shade-plugin</artifactId>
										<versionRange>[3.0.0,)</versionRange>
										<goals>
											<goal>shade</goal>
										</goals>
									</pluginExecutionFilter>
									<action>
										<ignore/>
									</action>
								</pluginExecution>
								<pluginExecution>
									<pluginExecutionFilter>
										<groupId>org.apache.maven.plugins</groupId>
										<artifactId>maven-compiler-plugin</artifactId>
										<versionRange>[3.1,)</versionRange>
										<goals>
											<goal>testCompile</goal>
											<goal>compile</goal>
										</goals>
									</pluginExecutionFilter>
									<action>
										<ignore/>
									</action>
								</pluginExecution>
							</pluginExecutions>
						</lifecycleMappingMetadata>
					</configuration>
				</plugin>
			</plugins>
		</pluginManagement>
	</build>

	<!-- This profile helps to make things run out of the box in IntelliJ -->
	<!-- Its adds Flink's core classes to the runtime class path. -->
	<!-- Otherwise they are missing in IntelliJ, because the dependency is 'provided' -->
	<profiles>
		<profile>
			<id>add-dependencies-for-IDEA</id>

			<activation>
				<property>
					<name>idea.version</name>
				</property>
			</activation>

			<dependencies>
				<dependency>
					<groupId>org.apache.flink</groupId>
					<artifactId>flink-java</artifactId>
					<version>${flink.version}</version>
					<scope>compile</scope>
				</dependency>
				<dependency>
					<groupId>org.apache.flink</groupId>
					<artifactId>flink-streaming-java_${scala.binary.version}</artifactId>
					<version>${flink.version}</version>
					<scope>compile</scope>
				</dependency>
			</dependencies>
		</profile>
	</profiles>

</project>

flink-config.propertites (此配置文件最好不要放在工程里面，可以单独放在服务器目录，方便修改)

topics=test
group.id=lzc
bootstrap.servers=bigdata1:9092,bigdata2:9092
auto.offset.reset=earliest
enable.auto.commit=false
checkpoint.interval=10000
redis.host=localhost
redis.pwd=123456
redis.db=0
FlinkKafkaToRedis
package org.myorg.quickstart.kafka;

import org.apache.flink.api.common.functions.FlatMapFunction;
import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.common.serialization.SimpleStringSchema;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.api.java.tuple.Tuple3;
import org.apache.flink.api.java.utils.ParameterTool;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.util.Collector;

public class FlinkKafkaToRedis {

    public static void main(String[] args) throws Exception{
        //传入配置文件路径即可
        ParameterTool parameters = ParameterTool.fromPropertiesFile(args[0]);

        DataStream<String> lines = FlinkUtil.createKafkaStream(parameters, SimpleStringSchema.class);

        SingleOutputStreamOperator<String> words = lines.flatMap(new FlatMapFunction<String, String>() {
            @Override
            public void flatMap(String words, Collector<String> collector) throws Exception {
                for (String word : words.split(" ")) {
                    collector.collect(word);
                }
            }
        });
        SingleOutputStreamOperator<Tuple2<String, Integer>> word = words.map(new MapFunction<String, Tuple2<String, Integer>>() {
            @Override
            public Tuple2<String, Integer> map(String word) throws Exception {
                return new Tuple2<>(word, 1);
            }
        });

        //为了保证程序出现问题可以继续累加
        SingleOutputStreamOperator<Tuple2<String, Integer>> sum = word.keyBy(0).sum(1);


        sum.map(new MapFunction<Tuple2<String, Integer>, Tuple3<String,String,String>>() {
            @Override
            public Tuple3<String, String, String> map(Tuple2<String, Integer> tp) throws Exception {

                return Tuple3.of("word_count",tp.f0,tp.f1.toString());
            }
        }).addSink(new MyRedisSink());

        FlinkUtil.getEnv().execute("kafkaSource");

    }

}

自定义工具类FlinkUtil
package org.myorg.quickstart.kafka;

import org.apache.flink.api.common.restartstrategy.RestartStrategies;
import org.apache.flink.api.common.serialization.DeserializationSchema;
import org.apache.flink.api.java.utils.ParameterTool;
import org.apache.flink.streaming.api.CheckpointingMode;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.CheckpointConfig;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer;
import java.util.Arrays;
import java.util.List;
import java.util.Properties;

public class FlinkUtil {

    private static StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

    /**
     * createKafkaStream
     * @param parameters
     * @param clazz
     * @param <T>
     * @return
     * @throws Exception
     */
    public static <T> DataStream<T>  createKafkaStream(ParameterTool parameters, Class<? extends DeserializationSchema<T>> clazz) throws Exception{
         env.getConfig().setGlobalJobParameters(parameters);
        //本地环境读取hdfs需要设置，集群上不需要
        //System.setProperty("HADOOP_USER_NAME","root");

        //默认情况下，检查点被禁用。要启用检查点
        env.enableCheckpointing(parameters.getLong("checkpoint.interval",5000L),CheckpointingMode.EXACTLY_ONCE);
        //设置重启策略 默认不停重启
        env.getConfig().setRestartStrategy(RestartStrategies.fixedDelayRestart(3,2000));

        //设置state存储的后端(建议在flink配置文件里面配置)
        //env.setStateBackend(new FsStateBackend("hdfs://namenode:40010/flink/checkpoints"));

        //程序异常退出或人为cancel掉，不删除checkpoint数据(默认是会删除)
        env.getCheckpointConfig().enableExternalizedCheckpoints(CheckpointConfig.ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);


        Properties properties = new Properties();
        properties.setProperty("bootstrap.servers", parameters.getRequired("bootstrap.servers"));
        properties.setProperty("group.id", parameters.getRequired("group.id"));
        //如果没有记录偏移量  第一次从最开始消费
        properties.setProperty("auto.offset.reset",parameters.get("auto.offset.reset","earliest"));
        //kafka的消费者不自动提交偏移量 而是交给flink通过checkpoint管理
        properties.setProperty("enable.auto.commit",parameters.get("enable.auto.commit","false"));

        String topics = parameters.getRequired("topics");
        List<String> topicList = Arrays.asList(topics.split(","));

        //Source : Kafka, 从Kafka中读取数据
        FlinkKafkaConsumer<T> kafkaConsumer = new FlinkKafkaConsumer<T>(
                topicList,
                clazz.newInstance(),
                properties);

        //flink checkpoint成功后还要向kafka特殊的topic中写入偏移量  默认是true
        kafkaConsumer.setCommitOffsetsOnCheckpoints(true);

        return env.addSource(kafkaConsumer);

    }


    /**
     * 获取执行环境
     * @return
     */
    public static StreamExecutionEnvironment getEnv(){
        return env;
    }
}

自定义redisSink(MyRedisSink)
package org.myorg.quickstart.kafka;

import org.apache.flink.api.java.tuple.Tuple3;
import org.apache.flink.api.java.utils.ParameterTool;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.streaming.api.functions.sink.RichSinkFunction;
import redis.clients.jedis.Jedis;

public class MyRedisSink extends RichSinkFunction<Tuple3<String,String,String>> {

    private transient Jedis jedis;
    @Override
    public void open(Configuration parameters) throws Exception {
        super.open(parameters);
        //获取全局参数
        ParameterTool params = (ParameterTool) getRuntimeContext().getExecutionConfig().getGlobalJobParameters();
        String host = params.getRequired("redis.host");
        String pwd = params.getRequired("redis.pwd");
        int db = params.getInt("redis.db", 0);
        jedis = new Jedis(host, 6379, 5000);
        jedis.auth(pwd);
        jedis.select(db);
    }

    @Override
    public void invoke(Tuple3<String, String, String> value, Context context) throws Exception {

        if (!jedis.isConnected()){
            jedis.connect();
        }

        jedis.hset(value.f0,value.f1,value.f2);

    }

    @Override
    public void close() throws Exception {
        super.close();
        jedis.close();

    }


}

大数据私房菜
关注
  • 1
    点赞
  • 踩
  • 8
    收藏
    觉得还不错? 一键收藏
  • 打赏
    打赏
  • 0
    评论
专栏目录
Flink 数据保存 Redis(自定义Redis Sink ) 实现exactly-once
weixin_43947279的博客
11-23 1715
通过如下链接：Flink官方文档，我们知道数据保存到 Redis 的容错机制是 at least once。所以我们通过幂等操作，使用新数据覆盖旧数据的方式，以此来实现 exactly-once 。 1.代码部分 1.1 config.properties配置文件 redis.host=192.168.204.210 redis.port=6379 redis.password=123456 redis.timeout=5000 redis.db=0 1.2 FlinkUtils工具类 package c
flink读取kafka数据保存到redis
s243471087的博客
08-08 4680
测试阶段采用了三台虚拟机，一台为master，两台为slave，已经在集群安装好了flink，kafka，zookeeper，redis。flink没有采用高可用模式，所以不需要hadoop，需要注意redis因为没有安装集群模式，是在每一台机器上安装的，这样的话导致master中的redis不会被用到，而任务开始后，slave1或者slave2中的redis是由flink的任务来选择的。 服务开...
Flink读取kafka中数据写入到Redis案例
wanjialin的博客
02-22 1915
案例准备： 1、启动Redis 2、启动zookeeper 3、启动Kafka 4、打开IDEA 代码实现： def main(args: Array[String]): Unit = { val env: StreamExecutionEnvironment = StreamExecutionEnvironment.getExecutionEnvironment val properties = new Properties() properties.setProperty("b
Flink-05 Flink Java 3分钟上手 Redis FlinkJedisPoolConfig 从Kafka写入Redis FlinkKafkaConsumer消费 结果写入Redis
最新发布
永远好奇，无限进步！
06-24 690
上节我们已经实现了，对Kafka数据的消费和计算，最终把结果输出到了控制台上。Flink-05 Flink Java 3分钟上手 Redis FlinkJedisPoolConfig 从Kafka写入Redis FlinkKafkaConsumer消费 结果写入Redis
Flink接收Kafka流数据实时向Redis中插入
weixin_42393724的博客
11-06 882
Flink接收Kafka流数据实时向Redis中插入 此文档主要写了：Flink接入kafka数据，并做“flatMap（）”算子操作，处理后将数据插入到Redis中 。 相关代码如下： 一、相关依赖 <!-- Flink相关--> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-jav
Flink sink算子，kafka数据写入redis
热门推荐
不是一枚开发
05-20 2万+
1. 概述   Redis 提供了string、hash、list、set、sorted set、bitmap等一系列常用的数据结构。 Flink 没有直接提供官方的 Redis 连接器，但 Bahir 项目提供了 Flink-Redis 的连接工具 2. 使用 2.1 引入依赖    服务器上的flink版本与引入的依赖之间有冲突，使用exclusions排除 <dependency> <groupId>org.apache.bahir</groupId
Flink笔记(二十三)：Flink 整合 Kafka (实现 Exactly-Once)
01-07
本文仅用来介绍 Flink 整合 Kafka 实现 Exactly-Once。 1.什么是Exactly-Once  恰好处理一次的意思。不管在处理的时候是否有异常发生，计算的结果都一样。即使在发现机器或者软件故障时，都不会出现数据丢失以及...
flink-connector-redis-2.10-1.1.5-API文档-中文版.zip
07-14
赠送jar包：flink-connector-redis_2.10-1.1.5.jar； 赠送原API文档：flink-connector-redis_2.10-1.1.5-javadoc.jar； 赠送源代码：flink-connector-redis_2.10-1.1.5-sources.jar； 赠送Maven依赖信息文件：flink-...
Flink读取Kafka数据保存到Redis的解决方案教程
06-15
大数据发展史： ...针对以上知识我们通过flink读取kafka保存到redis方式快速让大家学习flink如何使用，以及我们如果搭建高性能的flink应用，这个课程属于快速实战篇。 Flink + kafka + redis 实时计算
Flink实时读取Kafka数据批量聚合（定时按数量）写入Mysql.rar
12-27
标题中的"Flink实时读取Kafka数据批量聚合（定时/按数量）写入Mysql"是一个典型的实时数据处理场景，涉及到大数据技术栈中的三个关键组件：Apache Flink、Apache Kafka和MySQL。以下是对这些技术及其在该场景下应用...
Flink入门：读取Kafka实时数据流，实现WordCount
01-07
本文主要介绍Flink接收一个Kafka文本数据流，进行WordCount词频统计，然后输出到标准输出上。通过本文你可以了解如何编写和运行Flink程序。 代码拆解 首先要设置Flink的执行环境： // 创建Flink执行环境 ...
redis && kafka搭建
02-03
redis && kafka搭建
Flink笔记(二十四)：Flink 数据 保存 Redis(自定义Redis Sink)
01-07
本文主要来介绍 Flink 读取 Kafka数据，并实时下沉(Sink)数据到 Redis 的过程。        通过如下链接：Flink官方文档，我们知道数据保存到 Redis 的容错机制是 at least once。所以我们通过幂等操作，使用新数据覆盖旧数据的方式，以此来实现 exactly-once 。 1.代码部分 1.1 config.properties配置文件 bootstrap.server=192.168.204.210:9092,192.168.204.211:9092,192.168.204.212:9092 group.id=testGroup auto.offse
flinksql读写redis
weixin_50210283的博客
10-12 1657
最近有个需求，需要使用flinksql读写redis，由于官网上并没有redis的connector，在网上找了很久，开源的几个connector又没法满足要求，所有这里就自己动手实现了一个。已经适配了各个版本的flink，从flink1.12到flink1.15。
flink把kafka数据写进redis的简单实现
qq_40822132的博客
08-05 1040
flink把kafka数据写进redis的简单实现
用Flink将kafka的数据存到redis中
fasdfafsd的博客
08-27 1188
依赖 <dependency> <groupId>org.apache.bahir</groupId> <artifactId>flink-connector-redis_2.11</artifactId> <version>1.0</version> </dependency> 代码 /...
从kafka中获取数据写入到redis中
zhaoxiangchong的专栏
10-28 9592
大家：    好！从kafka中获取数据写入到redis中，需要用到spark中的redis客户端配置，请参考前面的博客(https://blog.csdn.net/zhaoxiangchong/article/details/78379883)。 第一步 要先将数据打入到kafka中，请参照我以前的博客 https://blog.csdn.net/zhaoxiangchong/article...
Flink教程(8) Checkpoint 状态管理 读取Kafka并写入Redis（实现exactly once）
Java Spring Elasticsearch
01-14 2031
一、前言 结语
Flink中StateBackend的方式
JinVijay的博客
02-23 1758
FsStateBackend /** * 从Kafka读取数据处理完毕写入Redis */ public class KafkaToRedis { public static void main(String[] args) throws Exception { StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); //开启checkp
Flink实现Kafka到Mysql的Exactly-Once
05-25
在Flink中实现Kafka到MySQL的Exactly-Once，可以使用Flink的两个特性：可重复性检查点（exactly-once checkpointing）和事务。具体的步骤如下： 1. 将Kafka作为Flink的数据源，使用Flink的Kafka Consumer将数据流...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

大数据私房菜

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付：¥1
获取中
扫码支付

您的余额不足，请更换扫码支付或充值

打赏作者

实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值