现象及分析
1、现象:
在主机XX.XX.137.250的DEBUG日志中有如下信息,而集群中的其它主机中并无相关日志:
WARN [Native-Transport-Requests-1] 2019-01-05 12:11:25,530 UDFunction.java:436 - User defined function mytestdb.averagefinal:(frozen<tuple<bigint,double>>) -> double ran longer than 500ms
WARN [Native-Transport-Requests-4] 2019-01-05 12:11:25,538 UDFunction.java:436 - User defined function mytestdb.averagefinal:(frozen<tuple<bigint,double>>,bigint,double) -> frozen<tuple<bigint,double>>ran longer than 500ms
WARN [Native-Transport-Requests-5] 2019-01-05 12:11:25,543 UDFunction.java:436 - User defined function mytestdb.averagefinal:(frozen<tuple<bigint,double>>) -> double ran longer than 500ms
DEBUG [Native-Transport-Requests-1] 2019-01-05 12:11:27,348 UDFunction.java:436 - Invocation of user defined function mytestdb.averagefinal:(frozen<tuple<bigint,double>>) -> double' failed org.apache.cassandra.exceptions.FunctionException: execution of 'mytestdb.averagefinal[frozen<tuple<bigint,double>>]' failed: java.util.concurrent.TimeoutException: User defined function mytestdb.averagefinal:(frozen<tuple<bigint,double>>) -> double ran longer than 1500ms - will stop Cassandra VM
at org.apache.cassandra.exceptions.FunctionExceptionException.create(FunctionExceptionException.java:35)~[apache-cassandra-3.11.2.jar:3.11.2]
.......
,2,、分析:
自定义函数averageFinal、averageState在XX.XX.137.250上运行超时,超过1500ms造成cassandra进程退出。
250宕机后,本来应该流入250的数据分散到集群其它集群的hint目录下(cassandra/hints),
由于hints目录的磁盘空间有限(17G), 造成各个实例的磁盘很快被写满,造成主机假死状态,不能提供服务。
解决方案
1,优化自定义函数配置(/conf/cassandra.yaml)
user_defined_function_fail_timeout:1500(可适当增大)
user_function_timeout_policy:die/die_immediate/ignore
(默认die,cassandra源码说配置ignore风险较大,不理解是什么原因)
2,调大hints目录磁盘容量,最好单独一个磁盘,空间要大点(/conf/cassandra.yaml)
Hints_directory:/cassandra/hints
3、完善集群监控:磁盘、CPU、内存等,发现问题及时处理。