Flink elasticSearchSink

zhangxl-jc

已于 2022-04-28 15:11:32 修改

阅读量1.6k

点赞数

分类专栏： Flink 文章标签： flink

于 2022-04-28 15:05:10 首次发布

本文链接：https://blog.csdn.net/zhangjiaxx/article/details/124474366

版权

Flink 专栏收录该内容

2 篇文章 0 订阅

订阅专栏

该博客介绍了如何在Apache Flink中配置并使用Elasticsearch Sink将JSON数据流写入Elasticsearch。示例展示了如何定义ElasticIndexSinkFunction类，处理JSON元素并构建索引请求，以及如何在Flink流处理中设置ElasticsearchSink来发送数据到指定的Elasticsearch集群。

摘要由CSDN通过智能技术生成

<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-connector-elasticsearch6_2.12</artifactId>
    <version>1.14.4</version>
</dependency>

SinkFunction实现类：

import org.apache.flink.api.common.functions.RuntimeContext
import org.apache.flink.streaming.connectors.elasticsearch.{ElasticsearchSinkFunction, RequestIndexer}

class ElasticIndexSinkFunction(indexType:String) extends ElasticsearchSinkFunction[JSONObject]{
  override def process(element: JSONObject, runtimeContext: RuntimeContext, requestIndexer: RequestIndexer): Unit = {
    try{        
            val uniqueKey=element.getString("uniqueKey")
            val indexName=EsIndexName.getWeekMondayIndexName(indexType,element.getLongValue("startTime"))
            val request: IndexRequest = Requests.indexRequest
            .index(indexName)
            .`type`("_doc")
            .id(uniqueKey)
            .source(element)

      requestIndexer.add(request)
    }catch {
      case e:Exception=>e.printStackTrace()
    }

  }
}

数据写入es:

import org.apache.commons.configuration2.builder.fluent.Configurations
import org.apache.flink.streaming.api.scala.DataStream
import org.apache.flink.streaming.connectors.elasticsearch6.ElasticsearchSink
import org.apache.http.HttpHost

import java.util
import java.util.UUID

object EsSlinkUtil {
  def sendData(indexType: String, data: DataStream[JSONObject], jobType: String) = {
    val configs = new Configurations()
    val configuration = configs.properties("SysConfig.properties")
    val esIP = configuration.getString("eSPort_export")
    val httpHosts = new util.ArrayList[HttpHost]()
    httpHosts.add(new HttpHost(esIP, 9200, "http"))
    try {
      val elasticSink: ElasticsearchSink.Builder[JSONObject] = new ElasticsearchSink.Builder[JSONObject](httpHosts, new ElasticIndexSinkFunction(indexType))
      data.addSink(elasticSink.build())
        .name(indexType + "-" + UUID.randomUUID().toString.replaceAll("-", ""))
        .uid(indexType + "-" + jobType)
    } catch {
      case e: Exception => e.printStackTrace()
    }
  }
}