flink+kafka+doris+springboot集成例子

VIP文章青春不流名

已于 2023-08-04 22:51:16 修改

阅读量2.3k

点赞数

文章标签： kubernetes

于 2023-08-04 10:48:38 首次发布

本文链接：https://blog.csdn.net/tt1024167802/article/details/132099520

版权

2.1.1.2 修改zookeeper.properties

2.1.1.3 修改server.properties

2.1.2.3 flink单节点启动与停止命令

2.1.3、部署doris

2.1.3.1 下载安装包并上传服务器

2.1.3.2 配置doris fe（前端）

2.1.3.3 启动doris fe（前端）

2.1.3.4 配置doris be（后端）

2.1.3.5 doris启动 be（后端）

2.1.3.5 doris启动成功验证

2.1.3.6 doris的be在fe上注册

2.1.3.6 通过doris的fe的Web UI页面创建数据库表

2.1.4、部署spring-boot的syslog-kafka-es-avro

2.1.4.1、syslog-kafka-es-avro基于netty已UDP方式监听syslog

2.1.4.2、syslog-kafka-es-avro已avro格式保存数据到kafka

2.1.5、部署spring-boot的flink-do-doris

2.1.5.1、flink-do-doris主类

三、效果验证

3.1、发送syslog日志，syslog-kafka-es-avro监听处理，存储到kafka

3.2 、查看flink消费kafka

3.3、在doris上查看入库详情

一、例子说明

1.1、概述

这是个例子，将输入写入kafka，flink消费kafka，并实时写入doris。

1.1、所需环境

软件	版本	备注
kafka_2.12-3.5.0	kafka_2.12-3.5.0	使用自带的zookeeper
flink-1.17.1	flink-1.17.1
jdk	1.8.0_202
doris	1.2.6 ( Stable )
spring-boot	2.1.17.RELEASE	syslog-kafka-es-avro
spring-boot	2.4.5	flink-do-doris
flink-doris-connector-1.17	1.4.0
elasticsearch	7.6.2
基础目录	/home
服务器	10.10.10.99	centos 7.x

1.2、执行流程

①、工具发送数据

②、spring-boot基于netty开启某端口监听，接收发送的消息内容，进行数据清洗、标准化

③、kafka product组件接收上一步产生的数据，已avro格式保存到kafka某topic上。

④、flink实时消费kafka某topic，以流的方式进行处理，输出源设置为doris

⑤、终端数据可在doris的fe页面上实时查询。

二、部署环境

2.1、中间件部署

2.1.1部署kakfa

2.1.1.1 上传解压kafka安装包

将安装包kafka_2.12-3.5.0.tar.gz上传到/home目录

tar -zxvf kafka_2.12-3.5.0.tar.gz

mv kafka_2.12-3.5.0 kafka

2.1.1.2 修改zookeeper.properties

路径：/home/kafka/config/zookeeper.properties

dataDir=/home/kafka/zookeeper
clientPort=2181
maxClientCnxns=0
admin.enableServer=false
# admin.serverPort=8080

2.1.1.3 修改server.properties

路径：/home/kafka/config/server.properties

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

#
# This configuration file is intended for use in ZK-based mode, where Apache ZooKeeper is required.
# See kafka.server.KafkaConfig for additional details and defaults
#

############################# Server Basics #############################

# The id of the broker. This must be set to a unique integer for each broker.
broker.id=0

############################# Socket Server Settings #############################

# The address the socket server listens on. If not configured, the host name will be equal to the value of
# java.net.InetAddress.getCanonicalHostName(), with