sparksql把JDBC 从关系型数据库中读取数据的方式创建 DataFrame报错:
20/08/26 15:29:37 ERROR Executor: Exception in task 0.0 in stage 1.0 (TID 1)
java.sql.BatchUpdateException: Duplicate entry '269' for key 'PRIMARY'
com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry '269' for key 'PRIMARY'
Job aborted due to stage failure: Task 0 in stage 1.0 failed 1 times, most recent failure: Lost task 0.0 in stage 1.0 (TID 1, localhost, executor driver): java.sq
去掉数据库里的主键递增
idea中sparksql把JDBC 从关系型数据库中读取数据的方式创建 DataFrame,代码部分
import org.apache.log4j.{Level, Logger}
import org.apache.spark.SparkConf
import org.apache.spark.sql.SparkSession
object readFile {
def main(args: Array[String]): Unit = {
Logger.getLogger("org").setLevel(Level.ERROR)
val conf: SparkConf = new SparkConf().setAppName(this.getClass.getSimpleName).setMaster("local[2]")
val spark = SparkSession.builder().config(conf).getOrCreate()
// 加载/保存方法
// val frame = spark.read.json("D:/JulyMounth/spark/sparkData/people.json")
// frame.show()
// frame.write.format("csv").mode("append").save("D:/JulyMounth/spark/sparkData/900")
//JDBC 从关系型数据库中读取数据的方式创建 DataFrame
val df = spark.read.format("jdbc")
.option("url", "jdbc:mysql://localhost:3306/mydata")
.option("driver", "com.mysql.jdbc.Driver")
.option("dbtable", "a_users")
.option("user", "root")
.option("password", "123456")
.load()
df.show()
df.write.format("jdbc")
.option("url","jdbc:mysql://localhost:3306/mydata")
.option("driver", "com.mysql.jdbc.Driver")
.option("dbtable","a_users")
.option("user","root")
.option("password","root")
.mode("append")
.save()
spark.close()
}
}