spark写es两种写法

最新推荐文章于 2024-08-20 00:30:32 发布

RestfulSgg

最新推荐文章于 2024-08-20 00:30:32 发布

阅读量281

点赞数 1

文章标签： spark elasticsearch

本文链接：https://blog.csdn.net/RestfulSgg/article/details/121119697

版权

本文介绍了使用Spark将数据写入Elasticsearch的两种方式：一是通过saveToES()方法，包括添加Maven依赖和具体代码实现；二是采用底层写法，直接将JSON数据写入ES。

摘要由CSDN通过智能技术生成

spark写es语法

1.通过saveToES()方法写入
步骤：
①maven依赖

<properties>
        <spark.version>2.4.3</spark.version>
        <scala.version>2.11.12</scala.version>
        <log4j.version>1.2.17</log4j.version>
        <slf4j.version>1.7.22</slf4j.version>

        <fastjson.version>1.2.47</fastjson.version>
        <httpclient.version>4.5.5</httpclient.version>
        <httpmime.version>4.3.6</httpmime.version>

        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
        <java.version>1.8</java.version>

        <mongodb-spark.version>2.0.0</mongodb-spark.version>
        <casbah.version>3.1.1</casbah.version>
    </properties>
    <dependencies>
        <!--此处放日志包，所有项目都要引用-->
        <!-- 所有子项目的日志框架 -->
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>jcl-over-slf4j</artifactId>
            <version>${slf4j.version}</version>
        </dependency>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-api</artifactId>
            <version>${slf4j.version}</version>
        </dependency>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-log4j12</artifactId>
            <version>${slf4j.version}</version>
        </dependency>
        <!-- 具体的日志实现 -->
        <dependency>
            <groupId>log4j</groupId>
            <artifactId>log4j</artifactId>
            <version>${log4j.version}</version>
        </dependency>
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>${scala.version}</version>
            <!--<scope>provided</scope>-->
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.11</artifactId>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.11</artifactId>
            <scope>provided