Flink读取Kafka消息同步到ElasticSearch

最新推荐文章于 2024-09-03 07:05:15 发布

赤剑吟龙

最新推荐文章于 2024-09-03 07:05:15 发布

阅读量2k

点赞数

分类专栏： Flink java 文章标签： elasticsearch flink java kafka apache

本文链接：https://blog.csdn.net/kuaidi2883/article/details/107163966

版权

本文介绍了如何在Windows环境下安装Flink，并通过编写Java程序利用Flink从Kafka消费消息，然后将这些消息同步到ElasticSearch。详细步骤包括Flink的安装、Java程序的编写、IDEA中打包jar、创建Kafka主题、上传jar到Flink以及在Kibana中查看同步结果。

摘要由CSDN通过智能技术生成

#一. 本文所需安装包版本
Jdk：1.8.0_131
Zookeeper：zookeeper-3.4.14
Kafka:kafka_2.12-2.3.0
ElasticSearch:elasticsearch-6.8.1
Kibana:kibana-6.8.1-windows-x86_64
Flink:flink-1.7.2
#二 window安装flink
去flink官网下载flink
https://flink.apache.org/downloads.html

进入flink安装bin目录
双击start-cluster.bat
浏览器输入http://localhost:8081
出现以下页面则Flink安装成功
flink安装成功.png

#三.编写Java程序同步代码

package com.deepexi.flink;

import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.common.functions.RuntimeContext;
import org.apache.flink.streaming.api.TimeCharacteristic;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSinkFunction;
import org.apache.flink.streaming.connectors.elasticsearch.RequestIndexer;
import org.apache.flink.streaming.connectors.elasticsearch.util.RetryRejectedExecutionFailureHandler;
import org.apache.flink.streaming.connectors.elasticsearch6.ElasticsearchSink;
import org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer010;
import org.apache.flink.streaming.util.serialization.SimpleStringSchema;
import org.apache.http.HttpHost;
import org.apache.log4j.Logger;
import org.elasticsearch.action.index.IndexRequest;
import org.elasticsearch.client.Requests;

import java.util.*;

/**
 * className:KafkaDemo
 * description:
 * author:
 * date:2019-07-08 14:05
 */
public class KafkaDemo {
    private static final Logger log = Logger.getLogger(KafkaDemo.class.getSimpleName());


    public static void main(String[] args) throws Exception {
        System.out.println("===============》 flink任务开始  ==============》");
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();