文末有彩蛋,这么多年从来没这样处理过问题
背景
操作系统是 centos 6,RabbitMQ 3.8.1.
服务器突然无法连接,重启后查看/var/log/messages,报错日志如下:
epmd: erlang mapper daemon cant fork 2'nd time Cannot allocate memoryepmd: erlang mapper daemon cant fork 2'nd time Cannot allocate memoryepmd: erlang mapper daemon cant fork 2'nd time Cannot allocate memoryepmd: erlang mapper daemon cant fork 2'nd time Cannot allocate memorysshd[1121]: error: fork: Cannot allocate memorysshd[1121]: error: fork: Cannot allocate memorysshd[1121]: error: fork: Cannot allocate memorysshd[1121]: error: fork: Cannot allocate memorysshd[1121]: error: fork: Cannot allocate memorysshd[1121]: error: fork: Cannot allocate memorysshd[1121]: error: fork: Cannot allocate memorysshd[1121]: error: fork: Cannot allocate memory
使用locate erl_crash.dump找到崩溃日志 erl_crash.dump,使用 https://github.com/ferd/recon/edit/master/script/erl_crashdump_analyzer.sh 分析结果如下:
analyzing erl_crash.dump, generated on: Wed Apr 22 16:39:10 2020 Slogan: init terminating in do_boot ({error,{cannot_read_enabled_plugins_file,/etc/rabbitmq/enabled_plugins,eacces}})Memory:=== processes: 17 Mb processes_used: 17 Mb system: 16 Mb atom: 0 Mb atom_used: 0 Mb binary: 0 Mb code: 6 Mb ets: 1 Mb --- total: 34 MbDifferent message queue lengths (5 largest different):=== 89 0Error logger queue length:===0File descriptors open:=== UDP: 0 TCP: 2 Files: 0 --- Total: 2Number of processes:===89Processes Heap+Stack memory sizes (words) used in the VM (5 largest different):=== 1 46422 1 10958 2 6772 2 4185 2 2586Processes OldHeap memory sizes (words) used in the VM (5 largest different):=== 1 318187 1 75113 1 46422 1 10958 2 2586Process States when crashing (sum): === 1 CONNECTED 2 CONNECTED|BINARY_IO 2 CONNECTED|BINARY_IO|PORT_LOCK 1 Current Process Internal ACT_PRIO_NORMAL | USR_PRIO_NORMAL | PRQ_PRIO_NORMAL | ACTIVE | RUNNING | ACTIVE_SYS 1 Current Process Running 1 Internal ACT_PRIO_HIGH | USR_PRIO_HIGH | PRQ_PRIO_HIGH | OFF_HEAP_MSGQ 3 Internal ACT_PRIO_MAX | USR_PRIO_MAX | PRQ_PRIO_MAX 1 Internal ACT_PRIO_MAX | USR_PRIO_MAX | PRQ_PRIO_MAX | OFF_HEAP_MSGQ 1 Internal ACT_PRIO_MAX | USR_PRIO_MAX | PRQ_PRIO_NORMAL 76 Internal ACT_PRIO_NORMAL | USR_PRIO_NORMAL | PRQ_PRIO_NORMAL 1 Internal ACT_PRIO_NORMAL | USR_PRIO_NORMAL | PRQ_PRIO_NORMAL | ACTIVE | RUNNING | ACTIVE_SYS 6 Internal ACT_PRIO_NORMAL | USR_PRIO_NORMAL | PRQ_PRIO_NORMAL | OFF_HEAP_MSGQ 1 Running 88 Waiting
没排查出问题来,这台机器是8c32g的机器,按道理应该不该出现内存不足的问题。
彩蛋在此
后面开发说RabbitMQ不用了,服务停掉。。。问题消失了。