2014-09-1812:47:05,379INFOorg.apache.hadoop.ipc.Server:IPCServerhandler210on8020caughtanexception
java.nio.channels.ClosedChannelException
atsun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:265)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:474)
atorg.apache.hadoop.ipc.Server.channelWrite(Server.java:2195)
at org.apache.hadoop.ipc.Server.access$2000(Server.java:110)
atorg.apache.hadoop.ipc.Server$Responder.processResponse(Server.java:979)
at org.apache.hadoop.ipc.Server$Responder.doRespond(Server.java:1045)
atorg.apache.hadoop.ipc.Server$Handler.run(Server.java:1798)
用haadmin命令来得到辅namenode状态:
sudo-uhdfshdfshaadmin-getServiceStatenn2
报错,无法得到回应。
2014-09-2510:15:07,972WARNorg.apache.hadoop.ipc.Server:IPCServerResponder,call
org.apache.hadoop.ha.HAServiceProtocol.getServiceStatusfrom172.23.64.24:53267:outputerror
用haadmin命令来得到主namenode状态:
sudo-uhdfshdfshaadmin-getServiceStatenn1
得到正常的回答:
Active
于是重启辅namenode
2014-09-2510:18:04,043INFOorg.apache.hadoop.hdfs.server.namenode.NameNode:STARTUP_MSG:
添加了一批节点信息:
2014-09-2510:18:24,385INFOorg.apache.hadoop.util.HostsFileReader:AddingJXQ-23-64-111.h.chinabank.com.cntothelistofincluded
hosts from/etc/hadoop/conf/allowed_hosts
topo.sh只能一次返回一个结果,这个报错忽略
2014-09-2510:18:42,969ERRORorg.apache.hadoop.net.ScriptBasedMapping:Script/etc/hadoop/conf/topo.shreturned1valueswhen100
were expected.
然后开始loading fsimage:
2014-09-2510:41:51,599INFOorg.apache.hadoop.hdfs.server.namenode.FSImage:Loadedimagefortxid312563139from
/mnt/hdfs/namenode/current/fsimage_0000000000312563139
2014-09-2510:41:51,697INFOorg.apache.hadoop.hdfs.server.namenode.FSImage:Reading
org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream@6a102a08expectingstarttxid#312563140
load 完 fsimage,开始replay edit log:
2014-09-2510:42:23,843INFOorg.apache.hadoop.hdfs.server.namenode.FSEditLogLoader:replayingeditlog:149476/149638transactions
completed.(100%)
因为有不少edit log,所以会花费较长时间,load多个edit log:
好几次的从 1% — 100%
注意,看时间戳,加载fsimage的时间可能会很长,花费20多分钟,注意观察:50070端口的页面,要有耐心:
2014-09-2514:28:15,378INFOorg.apache.hadoop.hdfs.server.namenode.FSImage:Loadingimagefile
/mnt/hdfs/namenode/current/fsimage_0000000000343076632usingnocompression
2014-09-2514:28:15,385INFOorg.apache.hadoop.hdfs.server.namenode.FSImage:Numberoffiles=13682532
2014-09-2514:58:16才加载成功
中途还有加载失败,从journal node取edit log失败:
2014-09-2514:54:49,195ERRORorg.apache.hadoop.hdfs.server.namenode.EditLogInputStream:Goterrorreadingeditloginputstream
http://JXQ-23-64-36.h.chinabank.com.cn:8480/getJournal?jid=wangyin-m6&segmentTxId=343076633&storageInfo=-40%3A994490903%3A0%3ACID-
6c19d051-644a-4a0f-b701-298641102a36;failingovertoeditloghttp://JXQ-23-64-60.h.chinabank.com.cn:8480/getJournal?jid=wangyin-
m6&segmentTxId=343076633&storageInfo=-40%3A994490903%3A0%3ACID-6c19d051-644a-4a0f-b701-298641102a36
进入transaction:
最终加载成功,进入safe mode,然后必须等到有Datanode汇报节点的文件块达到总块数的99.9%的时候才会离开safe mode.
2014-09-2514:58:10,411INFOorg.apache.hadoop.hdfs.StateChange:STATE*SafemodeON.
The reportedblocks0needsadditional14004072blockstoreachthethreshold0.9990oftotalblocks14018089.Safemodewillbe
turnedoffautomatically.
翻回头说primary namenode:
居然在13:09分,second namenode正在加载fsimage和transaction中崩溃了,幸亏当时有人及时发现,
当即重启,13:15分重启:
显然这时候已经发生脑裂了,因为主辅的状态都不对,所以zkfc服务不停的试图扶正辅namenode(变成主)
这时候判断辅namenode由于报错已久,好几天了,metadata估计都是不太正常的,主辅的meta所占磁盘空间也相差比较大。所以杀掉了辅namenode,备份了好的主namenode的metadata,然后单独启动了主namenode
悲剧就从这时开始了,主namenode启动后处于standby模式,safe mode is on。
sudo -u hdfs hdfs haadmin的命令都不可用,总是连接辅namenode的8020,得不到回应而失败。
注意以下这条命令,他是直接把nn1 变成主,而不执行autofailover
sudo -u hdfs hdfs haadmin haadmin –transitionToActive nn1
没办法:按一下方法处理,才解决:
①先停掉主辅namenode,备份好meata数据文件
②杀掉所有的Datanode上面的Datanode进程,避免读写文件,产生干扰。
③停掉主辅namenode上面的zkfc进程,同时修改hdfs-site.xml文件:
<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>false</value>
</property>
可以同时修改threshold-pct,使得永远不会自动脱出safe mode,必须手动脱出:
<property>
<name>dfs.namenode.safemode.threshold-pct</name>
<value>1.999</value>
</property>
④修改/etc/hadoop/conf/hadoop-env.sh,加大-Xmx为50G(缺省的是5000m,5G)
hadoop-env.sh:exportHADOOP_NAMENODE_OPTS="-Xmx50000m
/usr/java/jdk1.7.0_51/bin/java-Dproc_namenode-Xmx50000m-Djava.net.preferIPv4Stack=true-Dhadoop.log.dir=/var/log/hadoop-hdfs-
Dhadoop.log.file=hadoop.log-Dhadoop.home.dir=/usr/lib/hadoop-Dhadoop.id.str=hdfs-Dhadoop.root.logger=INFO,console-
Djava.library.path=/usr/lib/hadoop/lib/native-Dhadoop.policy.file=hadoop-policy.xml-Djava.net.preferIPv4Stack=true-
Djava.net.preferIPv4Stack=true-Djava.net.preferIPv4Stack=true-Dhadoop.log.dir=/var/log/hadoop-hdfs-Dhadoop.log.file=hadoop-hdfs-
namenode-JXQ-23-64-12.h.chinabank.com.cn.log-Dhadoop.home.dir=/usr/lib/hadoop-Dhadoop.id.str=hdfs-Dhadoop.root.logger=INFO,RFA-
Djava.library.path=/usr/lib/hadoop/lib/native-Dhadoop.policy.file=hadoop-policy.xml-Djava.net.preferIPv4Stack=true-Xmx50000m-
Dhadoop.security.logger=INFO,RFAS-Dhdfs.audit.logger=INFO,NullAppender-Xmx50000m-Dhadoop.security.logger=INFO,RFAS-
Dhdfs.audit.logger=INFO,NullAppender-Xmx50000m-Dhadoop.security.logger=INFO,RFAS-Dhdfs.audit.logger=INFO,NullAppender-
Dhadoop.security.logger=INFO,RFASorg.apache.hadoop.hdfs.server.namenode.NameNode
注意上面,如果一行里有多个-Xmx参数,是最后一个起作用的。
⑤重启主、辅namonode,观察:50070的页面,看是否都进入standby safe mode.
⑥在主namenode上,切换状态,从standby safe mode切到Active safe mode.
sudo -u hdfs hdfs haadmin -failover –forceactive nn2 nn1
注意:这条命令指定了failove,会调用failover函数,最后指定的是nn1是活状态,nn2是备用。
⑦主namenode处于active状态后,启动所有Datanode上面的Datanode的进程,看:50070页面lost block的变化,等到就绪。
⑧如果有问题,可以hadoop fsck / -blocks,进行检查。直到就绪
⑨主namenode执行命令,离开safe mode,开始提供服务:
sudo -u hdfs hdfs dfsadmin -safemode leave
⑩看看辅namenode的状态,看是否能追上来,并修复,如果不能修复的话可以重做:
sudo -u hdfs hdfs namenode -bootstrapStandby
⑪注意haadmin的几条命令:
transition要慎用,它根本不通知其他的namenode,是单台执行的。
而failover会通知其他的namenode。
[-transitionToActive<serviceId>]
[-transitionToStandby<serviceId>]
[-failover[--forcefence][--forceactive]<serviceId><serviceId>]
[-getServiceState<serviceId>]
[-checkHealth<serviceId>]
[-help<command>]
由于Hadoop脑裂,且ganglia无报警,灭有办法,只能手工加上一层报警了。
#!/bin/sh
isNNActive=$(curl-shttp://172.23.64.12:50070/dfshealth.jsp|grep "(active)")
if["X${isNNActive}"="X"];then
echo"Nameservice is down!!!!"|/usr/local/bin/email-s"Primary Namenode is Down!"zhangrr@chinabank.com.cn-no-encoding
fi
实际过程中却发现整个集群无法访问对应邮件服务器的25端口,反倒是可以访问80端口。
没办法,只好用php中转一下了,否则整个集群都开25端口,权限过多了。
这也引出一个命题,就是运维是否可以只开放80端口,用xml-rpc来调用那台机器上的各种服务,比如yum源,mail服务,短信服务,收据收集等等…..,这样网络权限就会很简单,不至于一堆重复的规则。
mail.php的内容,很简单,自用,就不多做检查了:
<?php
$from
=
$_REQUEST
[
"from"
]
;
$to
=
$_REQUEST
[
"to"
]
;
$subject
=
$_REQUEST
[
"subject"
]
;
$message
=
$_REQUEST
[
"message"
]
;
mail
(
$to
,
$subject
,
$message
,
"From: $from"
)
;
//echo "from:$from to:$to subject:$subject message:$message";
echo
"OK"
;
?>
修改过的shell脚本:
#!/bin/sh
isNNActive
=
$
(
curl
-
s
http
:
//172.23.64.12:50070/dfshealth.jsp|grep "(active)")
if
[
"X${isNNActive}"
=
"X"
]
;
then
#echo "Nameservice is down!" | /usr/local/bin/email -s "Primary Namenode is Down!" zhangrr@chinabank.com.cn -no-encoding
curl
-
d
"from=root@alert"
-
d
"to=zhangrr@chinabank.com.cn"
--
data
-
urlencode
"subject=Nameservice is down!"
--
data
-
urlencode
"message=Primary Namenode is Down!"
http
:
//172.23.64.99/mail.php
fi
注意curl -d和–data-urlencode的区别,一个是原生态,一个是经过urlencode编码,会把空格转译为%20,某些情况下是必须转一下的。
自打hadoop集群发生脑裂后,惶惶不可终日。
先是加了curl的探测namenode是否存活的脚本,而后又想再备一份namenode的文件夹到别的服务器上:
由于是按需同步,所以要用到inodify+rsync。
先普及一下rsync,rsync src dst,即同步文件从src到dst
这里无论src或者dst都可以是rsync的服务daemon所在服务器。
再有src目录如果是/data/rsync,最后不带/,则同步整个个rsync目录过去。
如果src目录是/data/rsync/,最后带个/,则只同步rsync目录下的文件过去,不同步这个文件夹。
好,那么这个方向下:
![tt01](http://www.linuxboy.net/wp-content/uploads/2014/09/tt01.jpg)
rsyncd.conf
pid
file
=
/
var
/
run
/
rsyncd
.
pid
lock
file
=
/
var
/
run
/
rsync
.
lock
log
file
=
/
var
/
log
/
rsyncd
.
log
hosts
allow
=
192.168.100.1
[
rsync
]
path
=
/
export
/
hadoop
/
uid
=
root
read
only
=
no
list
=
no
执行:rsync –daemon
在Namenode(192.168.100.1)上面建立可执行脚本ino.sh:
#!/bin/sh
rmodule
=
hadoop
ddir
=
/
mnt
/
hdfs
/
namenode
/
/
usr
/
bin
/
inotifywait
-
m
-
r
-
q
--
exclude
".*(swp|swx|~)"
-
e
close_write
,
delete
,
create
,
attrib
$
ddir
|
while
read
file
do
rsync
-
arzq
--
ignore
-
errors
--
bwlimit
=
500
--
delete
$
ddir
192.168.100.239
::
$
rmodule
done
>>
/
dev
/
null
2
>
&
1
&
注意上面,限速了500kb,500kb*8=4M,所以是限制了网速最大为4M,以免过多占用Namenode带宽。
执行:./ino.sh
检查hadoop的secondarynamenode的checkpoint是否正常运行的脚本
#!/bin/sh
CHECK_POINT="/mnt/hdfs/namenode/current/fsimage.ckpt*"
chk_files=`find$CHECK_POINT-cmin-120`
case$chk_files in
*fsimage*)
echo"OK - checkpoint is healthy"
exit0
;;
*)
echo"CRITICAL - checkpoint is not operating!>"
exit2
Fuse-hdfs 文件系统不支持 O_RDWR and O_EXCL, 所以用rsync同步文件到fuse_hdfs中的时候会得到 EIO 错误.
必须修正rsync源代码:
diff -r rsync-3.0.8.no_excl/syscall.c rsync-3.0.8/syscall.c
234a235,252
> #if defined HAVE_SECURE_MKSTEMP && defined HAVE_FCHMOD && (!defined HAVE_OPEN64 || defined HAVE_MKSTEMP64)
> {
> int fd = mkstemp(template);
> if (fd == -1)
> return -1;
> if (fchmod(fd, perms) != 0 && preserve_perms) {
> int errno_save = errno;
> close(fd);
> unlink(template);
> errno = errno_save;
> return -1;
> }
> #if defined HAVE_SETMODE && O_BINARY
> setmode(fd, O_BINARY);
> #endif
> return fd;
> }
> #else
237c255,256
< return do_open(template, O_WRONLY|O_CREAT, perms);
---
> return do_open(template, O_RDWR|O_EXCL|O_CREAT, perms);
> #endif