记录一下:
从Kafka读取数据再写到Kafka中并且要保证exactly-once案例
public class KafkaToKafkaExactlyOnceDemo {
public static void main(String[] args) throws Exception {
//创建一个执行环境
StreamExecutionEnvironment env = StreamExecutionEnvironment
.createLocalEnvironmentWithWebUI(new Configuration());
//不设置任何重启策略 但是开启checkpoint 默认是无限重启(最多可以重启Integer.Maxvalue次) 可以将程序运行的状态保存起来
env.enableCheckpointing(10000);
Properties properties = new Properties();
properties
.setProperty("bootstrap.servers","linux01:9092,linux02:9092,linux03:9092");
properties.setProperty("group.id","doit9988");
properties.setProperty("auto.offset.reset", "earliest");
//不自动提交偏移量
properties.setProperty("enable.auto.commit", "false");
//设置消费者的事务隔离级别:只读取已经提交事务的数据,脏数据不读
properties.setProperty("isolation.level","read_committed");
FlinkKafkaConsumer<String> kafkaConsumer = new FlinkKafkaConsumer<>("wc",
new SimpleStringSchema(), properties);
//checkpoint成功后,将偏移量是否写入到kafka特殊的topic中
kafkaConsumer.setCommitOffsetsOnCheckpoints(false);
DataStreamSource<String> kafkaLines = env.addSource(kafkaConsumer);
SingleOutputStreamOperator<String> filtered = kafkaLines
.filter(l -> !l.startsWith("error"));
//使用env创建dataStream
//文件内容是spark hadoop java 开启端口号 nc -lk 8888
DataStreamSource<String> socketLines = env.socketTextStream("linux01", 8888);
SingleOutputStreamOperator<String> errorStream = socketLines
.map(new MapFunction<String, String>() {
@Override
public String map(String value) throws Exception {
if (value.equals("error")) {
throw new RuntimeException("数据出现异常!");
}
return value.toUpperCase();
}
});
//将错误数据和kafka读取出来的数据union到一起
DataStream<String> union = filtered.union(errorStream);
//这个是保证transaction.timeout.ms不超过事务超时的时候broker允许的最大值
properties.setProperty("transaction.timeout.ms",1000 * 60 * 5+"");
//将数据再写回到kafka中
String topic ="topic22";
FlinkKafkaProducer<String> MyProducer = new FlinkKafkaProducer<String>(
topic,
new MyKafkaSerializationSchema(topic),
properties,
FlinkKafkaProducer.Semantic.EXACTLY_ONCE);
union.addSink(MyProducer);
//启动
env.execute("KafkaToKafkaExactlyOnceDemo");
}
}
代码准备好之后开始启动各个集群
1)首先需要保证虚拟机的时间一致
date -s "2021-05-20 10:15:40"
2)所有节点启动zookeeper
/opt/apps/zookeeper-3.4.6/bin/zkServer.sh start
3)所有节点启动kafka
/opt/apps/kafka_2.12-2.6.2/bin/kafka-server-start.sh -daemon /opt/apps/kafka_2.12-2.6.2/config/server.properties
4)启动flink
/opt/apps/flink-1.12.1/bin/start-cluster.sh
进程中如果有 StandaloneSessionClusterEntrypoint 有这个进行说明flink启动成功
5)在安装redis的虚拟机上启动redis ,需要查看数据存储情况
redis-server /usr/local/redis/redis.conf
集群准备好之后开启8888端口号,需要作为监听端口
nc -lk 8888
1、创建topic22
/opt/apps/kafka_2.12-2.6.2/bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic topic22 --replication-factor 2 --partitions 3
2、消费者命令行
/opt/apps/kafka_2.12-2.6.2/bin/kafka-console-consumer.sh --bootstrap-server linux01:9092,linux02:9092,linux03:9092 --topic topic22 --from-beginning
3、生产者命令行
/opt/apps/kafka_2.12-2.6.2/bin/kafka-console-producer.sh --broker-list linux01:9092,linux02:9092,linux03:9092 --topic wc
之后输入数据验证,这种情况是根据checkpointing开始重复写入了数据,如果要保证ExactlyOnce,则需要设计事务隔离等级,即只读取确认提交的数据
在命令行消费者末尾添加:
–consumer-property isolation.level=read_committed
这时读取的数据只会是已经确认提交过是数据,不会重复读取