flink自定义esSink【scala版本】

最新推荐文章于 2023-05-29 14:21:21 发布

幕忆向楊

最新推荐文章于 2023-05-29 14:21:21 发布

阅读量186

点赞数

文章标签： flink scala elasticsearch

本文链接：https://blog.csdn.net/weixin_38430685/article/details/130873049

版权

1.maven依赖：

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-sql-connector-elasticsearch7_${scala.binary.version}		</artifactId>
            <version>${flink.version}</version>
            <!--<scope>provided</scope>-->
        </dependency>

2.自定义esSink类

package wangjian.sink

import com.alibaba.fastjson.JSONObject
import org.apache.flink.api.common.functions.RuntimeContext
import org.apache.flink.streaming.connectors.elasticsearch.{
   ElasticsearchSinkFunction, RequestIndexer}
import org.elasticsearch.action.index.IndexRequest
import org.elasticsearch.action.update.UpdateRequest
import org.elasticsearch.client.Requests
import org.slf4j.LoggerFactory

/**
  * @author wmy
  * @create 2023/5/25 16:38
  */
class MyEsSinkFunction extends ElasticsearchSinkFunction[JSONObject]{
   

  val LOG = LoggerFactory.getLogger(classOf[MyEsSinkFunction])

  override def process(element: JSONObject, runtimeContext: RuntimeContext, requestIndexer: RequestIndexer): Unit = {
   

    val dataType = element.getString("data_type")

    if(dataType.equals("insert")){
   
      requestIndexer.add(insertFn(element))
    }else if(dataType.equals