自定义flink的RedisSource,实现从redis中读取数据,借鉴了flink-connector-redis_2.11的实现思路,对redis读取操作进行封装,其中flink-connector-redis_2.11的使用和介绍可参考文末连接。项目中需要引入flink-connector-redis_2.11的maven依赖。
抽象redis数据
由于redis有不同的数据类型,所以先定义MyRedisRecord类,封装redis数据类型和数据对象
package com.jike.flink.examples.redis;
import org.apache.flink.streaming.connectors.redis.common.mapper.RedisDataType;
import java.io.Serializable;
public class MyRedisRecord implements Serializable {
private Object data;
private RedisDataType redisDataType;
public MyRedisRecord(Object data, RedisDataType redisDataType) {
this.data = data;
this.redisDataType = redisDataType;
}
public Object getData() {
return data;
}
public void setData(Object data) {
this.data = data;
}
public RedisDataType getRedisDataType() {
return redisDataType;
}
public void setRedisDataType(RedisDataType redisDataType) {
this.redisDataType = redisDataType;
}
}
定义Redis数据读取类
将redis的操作封装起来,首先定义接口类,定义支持的读取操作,例子中这只写了哈希表的get操作,可以增加更多的操作
package com.jike.flink.examples.redis;
import java.io.Serializable;
import java.util.Map;
public interface MyRedisCommandsContainer extends Serializable {
Map hget(String key);
void close();
}
定义一个MyRedisCommandsContainer接口实现类,实现对redis的读取操作,主要是调用了Jedis的API,可以支持哨兵模式和直连redis两种模式的连接redis
package com.jike.flink.examples.redis;
import org.apache.flink.util.Preconditions;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import redis.clients.jedis.Jedis;
import redis.clients.jedis.JedisPool;
import redis.clients.jedis.JedisSentinelPool;
import java.util.HashMap;
import java.util.Map;
import java.util.Set;
public class MyRedisContainer implements MyRedisCommandsContainer,Cloneable{
private static final long serialVersionUID = 1L;
private static final Logger LOG = LoggerFactory.getLogger(MyRedisContainer.class);
private final JedisPool jedisPool;
private final JedisSentinelPool jedisSentinelPool;
public MyRedisContainer(JedisPool jedisPool) {
Preconditions.checkNotNull(jedisPool, "Jedis Pool can not be null");
this.jedisPool = jedisPool;
this.jedisSentinelPool = null;
}
public MyRedisContainer(JedisSentinelPool sentinelPool) {
Preconditions.checkNotNull(sentinelPool, "Jedis Sentinel Pool can not be null");
this.jedisPool = null;