Kudu has a hard and soft memory limit. The hard memory limit is the maximum amount a Kudu process is allowed to use, and is controlled by the --memory_limit_hard_bytes flag. The soft memory limit is a percentage of the hard memory limit, controlled by the flag memory_limit_soft_percentage and with a default value of 80%, that determines the amount of memory a process may use before it will start rejecting some write operations.
There are several ways to relieve the memory pressure on Kudu:
- If the host has more memory available for Kudu, increase --memory_limit_hard_bytes.
- Increase the rate at which Kudu can flush writes from memory to disk by increasing the number of disks or increasing --maintenance_manager_num_threads. Generally, the recommended ratio of maintenance manager threads to data directories is 1:3.
- Reduce the volume of writes flowing to Kudu on the application side.
简单总结为
减少kudu内存压力的几种方式为:
- 增加内存 通过设置 --memory_limit_hard_bytes.
- 增加kudu将内存往磁盘中刷写数据的效率
1.增加磁盘数量
2.增加 maintenance_manager_num_threads - 在应用端减少写入流量