spark与mysql整合

maven添加msyql 依赖

      <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core -->
       <dependency>
           <groupId>org.apache.spark</groupId>
           <artifactId>spark-core_2.11</artifactId>
           <version>2.4.3</version>
       </dependency>
       <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql -->
       <dependency>
           <groupId>org.apache.spark</groupId>
           <artifactId>spark-sql_2.11</artifactId>
           <version>2.4.3</version>
       </dependency>
       <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming -->
       <dependency>
           <groupId>org.apache.spark</groupId>
           <artifactId>spark-streaming_2.11</artifactId>
           <version>2.4.3</version>
       </dependency>
       <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-mllib -->
       <dependency>
           <groupId>org.apache.spark</groupId>
           <artifactId>spark-mllib_2.11</artifactId>
           <version>2.4.3</version>
       </dependency>
       <!-- https://mvnrepository.com/artifact/com.alibaba/fastjson -->
       <dependency>
           <groupId>com.alibaba</groupId>
           <artifactId>fastjson</artifactId>
           <version>1.2.58</version>
       </dependency>
       <!-- https://mvnrepository.com/artifact/mysql/mysql-connector-java -->
       <dependency>
           <groupId>mysql</groupId>
           <artifactId>mysql-connector-java</artifactId>
           <version>5.1.47</version>
       </dependency>

spark 向mysql 写数据

spark 官方文档

1.在mysql中新建表结构

2.编写代码

    val spark = SparkSession.builder().master("local").appName("log").getOrCreate()
    val sc = spark.sparkContext
    //val fileRDD = sc.textFiles(path)
    //val lineRDD = fileRDD.tranformation
    // case class XX()
    val log = lineRDD .map(x => XX())
    val rlog = spark.createDataFrame(log)
        rlog.write
          .mode(SaveMode.Append)
          .format("jdbc")
          .option("url", "jdbc:mysql://ip:3306")
          .option("driver", "com.mysql.jdbc.Driver")
          .option("dbtable", "dbtable")
          .option("user", "us")
          .option("password", "pswd")
          .save()
    sc.stop()
    spark.close()