偶然发现集群中两台机器内存占用存在异常,仔细排查之后,发现是clouera-scm-agent进程作怪。正常的机器里,该进程虚拟内存和物理内存占用,分别是2G和50M左右,而它们两个则分别达到了7G和5G左右。查看问题机器的该进程日志(默认路径/var/log/cloudera-scm-agent/cloudera-scm-agent.log),看到频繁的出现下列提示:
[20/May/2015 14:19:15 +0000] 25686 MainThread agent WARNING Expected user httpfs for /mylogs/hadoop-hdfs/stacks but was hdfs
[20/May/2015 14:19:15 +0000] 25686 MainThread agent WARNING Expected group httpfs for /mylogs/hadoop-hdfs/stacks but was hdfs
[20/May/2015 14:19:15 +0000] 25686 Thread-13 downloader INFO Starting download of: http://cmhost:7180/cmf/parcel/download/CDH-5.1.2-1.cdh5.1.2.p0.3-el6.parcel
[20/May/2015 14:19:15 +0000] 25686 Thread-13 downloader ERROR HTTP error during download
提示中应该反映了两个问题:第一个问题与/mylogs/hadoop-hdfs/有关,第二个问题与CDH-5.1.2-1.cdh5.1.2.p0.3-el6.parcel有关。
先来解决第一个问题。
从提