通过flink的RichSinkFunction,实现连接MongoDB,实时写入数据(也可以自定义一个类继承RichSinkFunction)
此处需注意,由于RichSinkFunction是序列化对象,此时可以使用 @transient (private) lazy来表示不需序列化,否则可能会报异常。(
其中@trainsient
可以避免overhead,lazy
可以第一次被调用时正确地初始化以避免NPE)。
代码如下:
streamData.addSink(new RichSinkFunction[String] {
lazy val mongoClient = new MongoClient(new ServerAddress("host", port))
override def invoke(value: String): Unit = {
if (mongoClient != null) {
val data = DataUtils.MapLoader(value)
val db = mongoClient.getDatabase("db")
val collection = db.getCollection("collection")
val list = new util.ArrayList[Document]()
val doc = new Document()
val date = new DateTime().getMillis
doc.append("createtime", date)
doc.append("updatetime", date)
data.foreach(t => doc.append(t._1, t._2))
list.add(doc)
collection.insertMany(list)
}
}
})
MongoDB鉴权:
val serverAddress = new ServerAddress("host", port)
val credential: util.ArrayList[MongoCredential] = new util.ArrayList[MongoCredential]
//MongoCredential.createScramSha1Credential()三个参数分别为 用户名 数据库名称 密码
val mongoCredential1: MongoCredential = MongoCredential.createScramSha1Credential("", "", "")
credential.add(mongoCredential1)
val mongoClient = new MongoClient(ServerAddress addr, List<MongoCredential> credentialsList)
pom文件:
<dependency> <groupId>org.mongodb</groupId> <artifactId>mongo-java-driver</artifactId> <version>3.10.1</version> </dependency>