【极数系列】Flink集成KafkaSink &amp；实时输出数据

最新推荐文章于 2024-06-13 10:51:35 发布

2401_84181286

最新推荐文章于 2024-06-13 10:51:35 发布

阅读量592

点赞数 30

分类专栏： 2024年程序员学习文章标签： flink 大数据

本文链接：https://blog.csdn.net/2401_84181286/article/details/137501072

版权

2024年程序员学习专栏收录该内容

62 篇文章 0 订阅

订阅专栏

fastjson
${fastjson.version}

org.apache.flink flink-java ${flink.version} org.apache.flink flink-streaming-scala_2.12 ${flink.version} org.apache.flink flink-clients ${flink.version} org.apache.logging.log4j log4j-slf4j-impl ${log4j.version} org.apache.logging.log4j log4j-api ${log4j.version} org.apache.logging.log4j log4j-core ${log4j.version} org.apache.flink flink-connector-kafka 3.0.2-1.18 org.apache.flink flink-connector-base 1.18.0 ${project.name} src/main/resources src/main/java **/*.xml org.apache.maven.plugins maven-shade-plugin 3.1.1 package shade org.apache.flink:force-shading org.google.code.flindbugs:jar305 org.slf4j:* org.apache.logging.log4j:* *:* META-INF/*.SF META-INF/*.DSA META-INF/*.RSA org.aurora.KafkaStreamingJob org.springframework.boot spring-boot-maven-plugin ${spring.boot.version} true ${project.build.finalName} repackage maven-compiler-plugin ${maven.plugin.version} ${java.version} ${java.version} UTF-8 -parameters aliyun-repos https://maven.aliyun.com/nexus/content/groups/public/ false aliyun-plugin https://maven.aliyun.com/nexus/content/groups/public/ false

6.3 配置文件

（1）application.properties

#kafka集群地址
kafka.bootstrapServers=localhost:9092
#kafka主题
kafka.topic=topic_a
#kafka消费者组
kafka.group=aurora_group

（2）log4j2.properties

rootLogger.level=INFO
rootLogger.appenderRef.console.ref=ConsoleAppender
appender.console.name=ConsoleAppender
appender.console.type=CONSOLE
appender.console.layout.type=PatternLayout
appender.console.layout.pattern=%d{HH:mm:ss,SSS} %-5p %-60c %x - %m%n
log.file=D:\tmprootLogger.level=INFO
rootLogger.appenderRef.console.ref=ConsoleAppender
appender.console.name=ConsoleAppender
appender.console.type=CONSOLE
appender.console.layout.type=PatternLayout
appender.console.layout.pattern=%d{HH:mm:ss,SSS} %-5p %-60c %x - %m%n
log.file=D:\tmp

6.4 创建sink作业

package com.aurora;

import org.apache.flink.api.common.eventtime.WatermarkStrategy;
import org.apache.flink.api.common.functions.FlatMapFunction;
import org.apache.flink.api.common.serialization.SimpleStringSchema;
import org.apache.flink.api.java.utils.ParameterTool;
import org.apache.flink.connector.base.DeliveryGuarantee;
import org.apache.flink.connector.kafka.sink.KafkaRecordSerializationSchema;
import org.apache.flink.connector.kafka.sink.KafkaSink;
import org.apache.flink.connector.kafka.source.KafkaSource;
import org.apache.flink.connector.kafka.source.KafkaSourceBuilder;
import org.apache.flink.connector.kafka.source.enumerator.initializer.OffsetsInitializer;
import org.apache.flink.runtime.state.StateBackend;
import org.apache.flink.runtime.state.filesystem.FsStateBackend;
import org.apache.flink.streaming.api.CheckpointingMode;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.CheckpointConfig;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.util.Collector;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.util.ArrayList;

/**

@author 浅夏的猫
@description kafka 连接器使用demo作业
@datetime 22:21 2024/2/1
*/
public class KafkaSinkStreamingJob {

private static final Logger logger = LoggerFactory.getLogger(KafkaSinkStreamingJob.class);

public static void main(String[] args) throws Exception {

//=1.获取参数================
//定义文件路径
String propertiesFilePath = “E:\project\aurora_dev\aurora_flink_connector_kafka\src\main\resources\application.properties”;
//方式一:直接使用内置工具类
ParameterTool paramsMap = ParameterTool.fromPropertiesFile(propertiesFilePath);

自我介绍一下，小编13年上海交大毕业，曾经在小公司待过，也去过华为、OPPO等大厂，18年进入阿里一直到现在。

深知大多数大数据工程师，想要提升技能，往往是自己摸索成长或者是报班学习，但对于培训机构动则几千的学费，着实压力不小。自己不成体系的自学效果低效又漫长，而且极易碰到天花板技术停滞不前！

因此收集整理了一份《2024年大数据全套学习资料》，初衷也很简单，就是希望能够帮助到想自学提升又不知道该从何学起的朋友。