1.flink 网络缓存
该问题出现在idea本地测试环境,原因是网络缓存不足,默认设置是64M。
java.io.IOException: Insufficient number of network buffers: required 10, but only 9 available. The total number of network buffers is currently set to 4096 of 32768 bytes each. You can increase this number by setting the configuration keys
'taskmanager.network.memory.fraction', 'taskmanager.network.memory.min', and 'taskmanager.network.memory.max'.
at org.apache.flink.runtime.io.network.buffer.NetworkBufferPool.createBufferPool(NetworkBufferPool.java:257)
at org.apache.flink.runtime.io.network.NetworkEnvironment.registerTask(NetworkEnvironment.java:199)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:618)
at java.lang.Thread.run(Thread.java:745)
本地解决方案是:
自定义createLocalEnvironmentWithWebUI 或者 createLocalEnvironment
val configuration = new Configuration()
//网络缓存占flink总内存的百分比
configuration.setFloat(TaskManagerOptions.NETWORK_MEMORY_FRACTION, 0.1f)
//网络缓存的最小内存占用
configuration.set(TaskManagerOptions.NETWORK_MEMORY_MIN, MemorySize.parse("1g"))
//网络缓存的最大内存占用
configuration.set(TaskManagerOptions.NETWORK_MEMORY_MAX,MemorySize.parse("1g"))
val env = StreamExecutionEnvironment.createLocalEnvironmentWithWebUI(configuration)