spark数据入mongo报错:
User class threw exception: java.lang.IllegalArgumentException: Missing database name. Set via the 'spark.mongodb.output.uri' or 'spark.mongodb.output.database' property
核心代码如下:
val conf = new SparkConf().setAppName("AdxTrafficDaily")
conf.set("spark.mongodb.output.uri", s"$MongoConn/$databaseName.$ResultCol?replicaSet=mgset-17888xxx&authSource=admin")
conf.set("spark.mongodb.input.uri", s"$MongoConn/$databaseName.$ResultCol?replicaSet=mgset-17888xxx&authSource=admin")
val ss = SparkSession.builder().enableHiveSupport().getOrCreate()
val mongoConn = MongoConnector(conf)
val CountResult = ss.sql(
s"""
select adx, region, hostName from
test.test
where day='%s'
""".format(Day)
)
val FilterDoc = new Document("day", Day)
mongoConn.withCollectionDo(new WriteConfig(databaseName,
ResultCol, Option(MongoConn)), {col: MongoCollection[Document] =>
col.deleteMany(FilterDoc)})
val insertRdd =CountResult.rdd.repartition(5).map(itm => {
val insertOne = new Document("day", Day).append("adx", itm(0)).append("region", itm(1)).append("hostName", itm(2))
insertOne
})
MongoSpark.save(insertRdd)
ss.stop()
明眼人可能已经看到了问题的错误在哪里了,MongoSpark插件使用的配置是从sparksession中活得的,而上面额sparksession创建语句没有吧conf加进来
原理的sparksession创建为:
val ss = SparkSession.builder().enableHiveSupport().getOrCreate()
修改后为:
val ss = SparkSession.builder().config(conf).enableHiveSupport().getOrCreate()
conf **.config(conf)**传进来了,MongoSpark就可以从sparksession中取到对应的配置,问题迎刃而解