现象:
Flink TTL过期报错Falling back to default Kryo serializer because Chill serializer couldn‘t be found.
详细报错如下:
排查步骤:
1: Flink TTL代码
// 26小时过期
val ttlConfig:StateTtlConfig = StateTtlConfig
.newBuilder(Time.hours(26L))
.useProcessingTime()
.updateTtlOnCreateAndWrite()
.setStateVisibility(StateTtlConfig.StateVisibility.NeverReturnExpired)
// compact 过程中清理过期的状态数据
.cleanupInRocksdbCompactFilter(5000)
.build()
查看没有问题
2: 查看配置
flink-conf.yaml
state.backend: rocksdb
# Directory for checkpoints filesystem, when using any of the default bundled
# state backends.
#
#
state.backend.rocksdb.ttl.compaction.filter.enabled: true
state.backend.rocksdb.memory.managed: true
# block大小,默认4KB
state.backend.rocksdb.block.blocksize: 32kb
#
# # block cache大小,默认8MB,内存余量充足建议128m或256m,提升读的性能
state.backend.rocksdb.block.cache-size: 128m
#
# # 用于指定同时可以操作RocksDBStateBackend的线程数量,默认是1
state.backend.rocksdb.checkpoint.transfer.thread.num: 2
#
# # 后台负责 flush 和 compaction 的最大并发线程数,默认为1
state.backend.rocksdb.thread.num: 4
state.checkpoints.dir: hdfs://nameservice1/flink-checkpoints
state.savepoints.dir: hdfs://nameservice1/flink-savepoints
state.backend.rocksdb.localdir: /data/flink/rkdb
state.backend.incremental: true
也是没有问题
3: 查看缓存的ListState和MapState
最后排查下来是Liststate格式不对, 导致反序列化失败:
class MainFabWindowEndKeyedBroadcastProcessFunction extends KeyedBroadcastProcessFunction[String, (String, JsonNode, JsonNode), JsonNode, fdcWindowData] {
//{timestamp: [stepId, timestamp, [1: sensorAlias, 2: sensorValue, 3: unit] ]}
private var rawDataState: ListState[(Long, Long, Seq[(String, Double, String)])] = _
}
代码中使用了ListState, 里面是一个三元组, 三元组其中一个元素是列表, 在rocksdb序列化和反序列化过程中失败, 报上面的错
解决方案:
将ListState修改成MapState
case class windowRawData(stepId: Long,
timestamp: Long,
rawList: List[(String, Double, String)])
class MainFabWindowEndKeyedBroadcastProcessFunction extends KeyedBroadcastProcessFunction[String, (String, JsonNode, JsonNode), JsonNode, fdcWindowData] {
private var rawDataState: MapState[Long, windowRawData] = _
}
RocksDB 中考虑 value 值很大的极限场景;受限于 JNI bridge API 的限制,单个 value 只支持 2^31 bytes 大小,如果存在很极限的情况,可以考虑使用 MapState 来替代 ListState 或者 ValueState,因为 RocksDB 的 map state 并不是整个 map 作为 value 进行存储,而是将 map 中的一个条目作为键值对进行存储