我们在flink入门之scala实现异步IO访问redis及踩坑记录(1)中采用jedis来模拟异步读取redis,那么在本篇,我们要采用redis的高级客户端lettuce,这个客户端很强大,支持异步操作,如果想具体了解的请移步:
Redis高级客户端Lettuce详解
首先我们导入依赖:
<dependency>
<groupId>io.lettuce</groupId>
<artifactId>lettuce-core</artifactId>
<version>5.0.5.RELEASE</version>
</dependency>
然后我们也是用AsyncDataStream的orderedWait方法(其中kafkaDs是从kafka读进来的数据流):
AsyncDataStream.orderedWait(kafkaDs, new AsyncRedisRead, 30L, TimeUnit.SECONDS, 1000)
自定义异步读取函数:
class AsyncRedisRead extends RichAsyncFunction[Obj1, Obj1] {
private var redisClient: RedisClient = _
private var connection: StatefulRedisConnection[String, String] = _
private var async: RedisAsyncCommands[String, String] = _
override def open(parameters: Configuration): Unit = {
super.open(parameters)
val redisUri = "redis://10.xx.xx.xxx"
redisClient = RedisClient.create(redisUri)
connection = redisClient.connect()
async = connection.async()
}
override def asyncInvoke(input: Obj1, resultFuture: ResultFuture[Obj1]): Unit = {
async.keys(s"flink*").thenAccept(new Consumer[java.util.List[String]] {
override def accept(t: List[String]): Unit = {
if (t.isEmpty) {
resultFuture.complete(Iterable[Obj1](input))
}
else {
var seq = Seq[Obj1]()
//遍历列表
for (i <- 0 until t.size()) {
val member= t.get(i)
val infoArray = member.split("_")
val ruleId = infoArray(3)
val userId = infoArray(4)
val suspectId = infoArray(5)
val alarmStatus = infoArray(6)
seq = seq :+ Obj1(input.uuid, input.wifiList, ruleId, userId, suspectId)
}
resultFuture.complete(seq)
}
}
})
}
override def close(): Unit = {
super.close()
if (redisClient != null) redisClient.shutdown()
if (connection != null) connection.close()
}
}
看上去很简单,但是这里必须要注意一点:
ResultFuture在第一次调用ResultFuture.complete时就执行完成了。随后的所有complete调用都将被忽略!!!
所以当我们想要把一个集合循环遍历返回数据时,切记不要这样写:
for (i <- 0 until t.size()) {
val member= t.get(i)
val infoArray = member.split("_")
val ruleId = infoArray(3)
val userId = infoArray(4)
val suspectId = infoArray(5)
val alarmStatus = infoArray(6)
resultFuture.complete(Iterable[Obj1](Obj1(input.uuid, input.wifiList, ruleId, userId, suspectId)))
}
这样写的话,在遍历第一条数据执行完complete后,后面的数据就不会再通过complete返回了。
至此Flink异步维表读取redis的相关操作就是这样,如有不妥的地方,欢迎留言指正