搜索任务过多,并且造成了搜索风暴
告警内容
Too many search jobs found in the dispatch directory (found=3692, warning level=2000). This could negatively impact Splunk's performance, consider removing some of the old search jobs.
解决方式
下SH将删除超过 2 小时的调度工件
clean_dispatch.sh
dispatch=/opt/splunk/var/run/splunk/dispatch
splunkdir=/opt/splunk
find $dispatch -maxdepth 1 -mmin +120 2>/dev/null | while read job; do if [ ! -e "$job/save" ] ; then rm -rfv $job ; fi ; done
find $dispatch -type d -empty -name alive.token -mmin +120 2>/dev/null | xargs -i rm -Rf {}
find $splunkdir/var/run/splunk/ -type f -name "session-*" -mmin +120 2>/dev/null | xargs -i rm -Rf {}
您也可以手动清除调度文件夹中的所有作业。建议从较旧的开始
如果本来就是正常任务,导致很多搜索超时,那么就需要根据要求增加硬件或者修改limit.conf