Zookeeper CancelledKeyException

最新推荐文章于 2022-09-15 22:12:29 发布
最新推荐文章于 2022-09-15 22:12:29 发布
阅读量847 收藏
点赞数
文章标签： 大数据 java python
原文链接：https://my.oschina.net/yulongblog/blog/1506168
版权

2019独角兽企业重金招聘Python工程师标准>>> hot3.png

随着生产集群里应用的增多，zookeeper的压力越来越大，resourcemanager出现了异常挂起。

延迟问题

首先是“fsync-ing the write ahead log in SyncThread:3 took 1606ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide”

这个问题主要是日志同步延迟，一般是磁盘性能不好，log和data目录同磁盘，分开两个目录，设置一些参数可能有点用。

tickTime=4000
# The number of ticks that the initial 
# synchronization phase can take
initLimit=20
# The number of ticks that can pass between 
# sending a request and getting an acknowledgement
syncLimit=10

CancelledKeyException

再次是zookeeper的CancelledKeyException，这个bug在CDH-5.11及之前(笔者写这篇文章的时候5.11是最新的版本)版本的zookeeper中并没有进行修复。

这个bug在在zookeeper集群负载较高时，可能导致其他使用zookeeper的服务(包括yarn、storm、kafka等)出现失去连接挂起的状态，所以需要打patch的。

https://issues.apache.org/jira/browse/ZOOKEEPER-1237

diff -uwp zookeeper-3.4.5/src/java/main/org/apache/zookeeper/server/NIOServerCnxn.java.ZK1237 zookeeper-3.4.5/src/java/main/org/apache/zookeeper/server/NIOServerCnxn.java
--- zookeeper-3.4.5/src/java/main/org/apache/zookeeper/server/NIOServerCnxn.java.ZK1237    2012-09-30 10:53:32.000000000 -0700
+++ zookeeper-3.4.5/src/java/main/org/apache/zookeeper/server/NIOServerCnxn.java    2013-08-07 13:20:19.227152865 -0700
@@ -150,7 +150,8 @@ public class NIOServerCnxn extends Serve
                 // We check if write interest here because if it is NOT set,
                 // nothing is queued, so we can try to send the buffer right
                 // away without waking up the selector
-                if ((sk.interestOps() & SelectionKey.OP_WRITE) == 0) {
+                if (sk.isValid() &&
+                    (sk.interestOps() & SelectionKey.OP_WRITE) == 0) {
                     try {
                         sock.write(bb);
                     } catch (IOException e) {
@@ -214,14 +215,18 @@ public class NIOServerCnxn extends Serve

                 return;
             }
-            if (k.isReadable()) {
+            if (k.isValid() && k.isReadable()) {
                 int rc = sock.read(incomingBuffer);
                 if (rc < 0) {
-                    throw new EndOfStreamException(
+                    if (LOG.isDebugEnabled()) {
+                        LOG.debug(
                             "Unable to read additional data from client sessionid 0x"
                             + Long.toHexString(sessionId)
                             + ", likely client has closed socket");
                 }
+                    close();
+                    return;
+                }
                 if (incomingBuffer.remaining() == 0) {
                     boolean isPayload;
                     if (incomingBuffer == lenBuffer) { // start of next request
@@ -242,7 +247,7 @@ public class NIOServerCnxn extends Serve
                     }
                 }
             }
-            if (k.isWritable()) {
+            if (k.isValid() && k.isWritable()) {
                 // ZooLog.logTraceMessage(LOG,
                 // ZooLog.CLIENT_DATA_PACKET_TRACE_MASK
                 // "outgoingBuffers.size() = " +

日志

zookeeper日志

2017-06-15 03:09:03,098 [myid:3] - WARN  [SyncThread:3:FileTxnLog@321] - fsync-ing the write ahead log in SyncThread:3 took 1506ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
2017-06-15 03:09:05,098 [myid:3] - WARN  [SyncThread:3:FileTxnLog@321] - fsync-ing the write ahead log in SyncThread:3 took 1606ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
2017-06-15 03:09:05,744 [myid:3] - INFO  [ProcessThread(sid:3 cport:-1)::PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x35c9bd7493c22c0 type:create cxid:0x2 zxid:0x4f003006ef txntype:-1 reqpath:n/a Error Path:/hive_zookeeper_namespace/zb_ods Error:KeeperErrorCode = NodeExists for /hive_zookeeper_namespace/zb_ods
2017-06-15 03:09:09,952 [myid:3] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@349] - caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x25c9bd75af70091, likely client has closed socket
        at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
        at java.lang.Thread.run(Thread.java:745)
2017-06-15 03:09:12,076 [myid:3] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1001] - Closed socket connection for client /133.0.91.42:46579 which had sessionid 0x25c9bd75af70091
2017-06-15 03:09:12,076 [myid:3] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@349] - caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x35c9bd7493c22ac, likely client has closed socket
        at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
        at java.lang.Thread.run(Thread.java:745)
2017-06-15 03:09:12,076 [myid:3] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1001] - Closed socket connection for client /133.0.91.41:53351 which had sessionid 0x35c9bd7493c22ac
2017-06-15 03:09:12,076 [myid:3] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@349] - caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x35c9bd7493c2268, likely client has closed socket
        at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
        at java.lang.Thread.run(Thread.java:745)
2017-06-15 03:09:12,075 [myid:3] - INFO  [ProcessThread(sid:3 cport:-1)::PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x25c9bd75af70091 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:null Error:KeeperErrorCode = Session moved
2017-06-15 03:09:12,083 [myid:3] - INFO  [ProcessThread(sid:3 cport:-1)::PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x35c9bd7493c2268 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:null Error:KeeperErrorCode = Session moved
2017-06-15 03:09:12,083 [myid:3] - INFO  [ProcessThread(sid:3 cport:-1)::PrepRequestProcessor@574] - Got user-level KeeperException when processing sessionid:0x35c9bd7493c2268 type:multi cxid:0x7e06 zxid:0x4f003006f2 txntype:-1 reqpath:n/a aborting remaining multi ops. Error Path:null Error:KeeperErrorCode = Session moved
2017-06-15 03:09:12,087 [myid:3] - WARN  [SyncThread:3:FileTxnLog@321] - fsync-ing the write ahead log in SyncThread:3 took 6988ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
2017-06-15 03:09:14,136 [myid:3] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1001] - Closed socket connection for client /133.0.91.42:46580 which had sessionid 0x35c9bd7493c2268
2017-06-15 03:09:14,136 [myid:3] - ERROR [CommitProcessor:3:NIOServerCnxn@180] - Unexpected Exception:
java.nio.channels.CancelledKeyException
        at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
        at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
        at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:153)
        at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1076)
        at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:170)
        at org.apache.zookeeper.server.quorum.Leader$ToBeAppliedRequestProcessor.processRequest(Leader.java:634)
        at org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:74)
2017-06-15 03:09:14,137 [myid:3] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /133.0.91.42:47513
2017-06-15 03:09:14,341 [myid:3] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@832] - Client attempting to renew session 0x35c9bd7493c2268 at /133.0.91.42:47513
2017-06-15 03:09:14,341 [myid:3] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@595] - Established session 0x35c9bd7493c2268 with negotiated timeout 10000 for client /133.0.91.42:47513
2017-06-15 03:09:14,342 [myid:3] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@863] - got auth packet /133.0.91.42:47513
2017-06-15 03:09:14,342 [myid:3] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@897] - auth success /133.0.91.42:47513
2017-06-15 03:09:14,342 [myid:3] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@354] - Exception causing close of session 0x35c9bd7493c2268 due to java.io.IOException: Len error 1673753
2017-06-15 03:09:14,342 [myid:3] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1001] - Closed socket connection for client /133.0.91.42:47513 which had sessionid 0x35c9bd7493c2268
2017-06-15 03:09:14,341 [myid:3] - ERROR [CommitProcessor:3:NIOServerCnxn@180] - Unexpected Exception:
java.nio.channels.CancelledKeyException
        at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
        at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
        at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:153)
        at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1076)
        at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:404)
        at org.apache.zookeeper.server.quorum.Leader$ToBeAppliedRequestProcessor.processRequest(Leader.java:634)
        at org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:74)
2017-06-15 03:09:14,433 [myid:3] - INFO  [ProcessThread(sid:3 cport:-1)::PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x35c9bd7493c22c0 type:create cxid:0x5 zxid:0x4f003006fd txntype:-1 reqpath:n/a Error

 

resourcemanager日志：

2017-06-15 03:09:11,943 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected.
2017-06-15 03:09:11,946 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced...
2017-06-15 03:09:11,947 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a036265681203726d32
2017-06-15 03:09:11,947 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it.
2017-06-15 03:09:11,947 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /yarn-leader-election/beh/ActiveBreadCrumb to indicate that the local node is the most recent active...
2017-06-15 03:09:11,953 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x35c9bd7493c2268, likely server has closed socket, closing socket connection and attempting reconnect
2017-06-15 03:09:12,051 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo: checking for deactivate...
2017-06-15 03:09:12,361 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server had02.hadoop/133.0.91.42:2181. Will not attempt to authenticate using SASL (unknown error)
2017-06-15 03:09:12,361 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to had02.hadoop/133.0.91.42:2181, initiating session
2017-06-15 03:09:12,387 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server had02.hadoop/133.0.91.42:2181, sessionid = 0x35c9bd7493c2268, negotiated timeout = 10000
2017-06-15 03:09:12,392 WARN org.apache.zookeeper.ClientCnxn: Session 0x35c9bd7493c2268 for server had02.hadoop/133.0.91.42:2181, unexpected error, closing socket connection and attempting reconnect
java.io.IOException: Broken pipe
        at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
        at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
        at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
        at sun.nio.ch.IOUtil.write(IOUtil.java:65)
        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:117)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:355)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1075)
2017-06-15 03:09:12,534 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo: checking for deactivate...
2017-06-15 03:09:12,678 INFO org.apache.hadoop.conf.Configuration: found resource yarn-site.xml at file:/opt/beh/core/hadoop/etc/hadoop/yarn-site.xml
2017-06-15 03:09:12,681 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=hadoop   OPERATION=refreshAdminAcls      TARGET=AdminService     RESULT=SUCCESS
2017-06-15 03:09:12,681 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Already in active state
2017-06-15 03:09:12,681 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=hadoop   OPERATION=refreshQueues TARGET=AdminService     RESULT=SUCCESS
2017-06-15 03:09:12,681 INFO org.apache.hadoop.conf.Configuration: found resource yarn-site.xml at file:/opt/beh/core/hadoop/etc/hadoop/yarn-site.xml
2017-06-15 03:09:12,683 INFO org.apache.hadoop.util.HostsFileReader: Setting the includes file to
2017-06-15 03:09:12,683 INFO org.apache.hadoop.util.HostsFileReader: Setting the excludes file to
2017-06-15 03:09:12,683 INFO org.apache.hadoop.util.HostsFileReader: Refreshing hosts (include/exclude) list
2017-06-15 03:09:12,683 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=hadoop   OPERATION=refreshNodes  TARGET=AdminService     RESULT=SUCCESS
2017-06-15 03:09:12,683 INFO org.apache.hadoop.conf.Configuration: found resource core-site.xml at file:/opt/beh/core/hadoop/etc/hadoop/core-site.xml
2017-06-15 03:09:12,685 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=hadoop   OPERATION=refreshSuperUserGroupsConfiguration   TARGET=AdminService     RESULT=SUCCESS
2017-06-15 03:09:12,685 INFO org.apache.hadoop.conf.Configuration: found resource core-site.xml at file:/opt/beh/core/hadoop/etc/hadoop/core-site.xml
2017-06-15 03:09:12,685 INFO org.apache.hadoop.security.Groups: clearing userToGroupsMap cache
2017-06-15 03:09:12,685 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=hadoop   OPERATION=refreshUserToGroupsMappings   TARGET=AdminService     RESULT=SUCCESS
2017-06-15 03:09:12,685 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=hadoop   OPERATION=transitionToActive    TARGET=RMHAProtocolService      RESULT=SUCCESS
2017-06-15 03:09:12,876 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server had03.hadoop/133.0.91.43:2181. Will not attempt to authenticate using SASL (unknown error)
2017-06-15 03:09:12,877 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to had03.hadoop/133.0.91.43:2181, initiating session
2017-06-15 03:09:13,535 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1497466013139_0394_000001 (auth:SIMPLE)
2017-06-15 03:09:13,539 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: AM registration appattempt_1497466013139_0394_000001
2017-06-15 03:09:13,539 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=hadoop   IP=133.0.91.43  OPERATION=Register App Master   TARGET=ApplicationMasterService RESULT=SUCCESS  APPID=application_1497466013139_0394    APPATTEMPTID=appattempt_1497466013139_0394_000001
2017-06-15 03:09:14,920 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server had03.hadoop/133.0.91.43:2181, sessionid = 0x35c9bd7493c2268, negotiated timeout = 10000
2017-06-15 03:09:14,927 WARN org.apache.zookeeper.ClientCnxn: Session 0x35c9bd7493c2268 for server had03.hadoop/133.0.91.43:2181, unexpected error, closing socket connection and attempting reconnect
java.io.IOException: Broken pipe
        at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
        at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
        at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
        at sun.nio.ch.IOUtil.write(IOUtil.java:65)
        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:117)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:355)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1075)
2017-06-15 03:09:15,028 INFO org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore: Waiting for zookeeper to be connected, retry no. + 2
2017-06-15 03:09:15,077 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server had01.hadoop/133.0.91.41:2181. Will not attempt to authenticate using SASL (unknown error)
2017-06-15 03:09:15,078 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to had01.hadoop/133.0.91.41:2181, initiating session
2017-06-15 03:09:15,100 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server had01.hadoop/133.0.91.41:2181, sessionid = 0x35c9bd7493c2268, negotiated timeout = 10000
2017-06-15 03:09:15,105 WARN org.apache.zookeeper.ClientCnxn: Session 0x35c9bd7493c2268 for server had01.hadoop/133.0.91.41:2181, unexpected error, closing socket connection and attempting reconnect
java.io.IOException: Broken pipe
        at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
        at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
        at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
        at sun.nio.ch.IOUtil.write(IOUtil.java:65)
        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:117)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:355)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1075)
2017-06-15 03:09:15,588 INFO org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Allocated new applicationId: 397
2017-06-15 03:09:15,772 INFO org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Allocated new applicationId: 398
2017-06-15 03:09:16,091 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server had02.hadoop/133.0.91.42:2181. Will not attempt to authenticate using SASL (unknown error)
2017-06-15 03:09:16,092 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to had02.hadoop/133.0.91.42:2181, initiating session
2017-06-15 03:09:16,118 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server had02.hadoop/133.0.91.42:2181, sessionid = 0x35c9bd7493c2268, negotiated timeout = 10000
2017-06-15 03:09:16,124 WARN org.apache.zookeeper.ClientCnxn: Session 0x35c9bd7493c2268 for server had02.hadoop/133.0.91.42:2181, unexpected error, closing socket connection and attempting reconnect
java.io.IOException: Broken pipe
        at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
        at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
        at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
        at sun.nio.ch.IOUtil.write(IOUtil.java:65)
        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:117)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:355)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1075)
2017-06-15 03:09:16,315 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server had03.hadoop/133.0.91.43:2181. Will not attempt to authenticate using SASL (unknown error)
2017-06-15 03:09:16,315 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to had03.hadoop/133.0.91.43:2181, initiating session
2017-06-15 03:09:16,339 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server had03.hadoop/133.0.91.43:2181, sessionid = 0x35c9bd7493c2268, negotiated timeout = 10000
2017-06-15 03:09:16,344 WARN org.apache.zookeeper.ClientCnxn: Session 0x35c9bd7493c2268 for server had03.hadoop/133.0.91.43:2181, unexpected error, closing socket connection and attempting reconnect
java.io.IOException: Broken pipe
        at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
        at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
        at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
        at sun.nio.ch.IOUtil.write(IOUtil.java:65)
        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:117)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:355)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1075)
2017-06-15 03:09:17,176 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server had01.hadoop/133.0.91.41:2181. Will not attempt to authenticate using SASL (unknown error)
2017-06-15 03:09:17,177 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to had01.hadoop/133.0.91.41:2181, initiating session
2017-06-15 03:09:17,201 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server had01.hadoop/133.0.91.41:2181, sessionid = 0x35c9bd7493c2268, negotiated timeout = 10000
2017-06-15 03:09:17,206 WARN org.apache.zookeeper.ClientCnxn: Session 0x35c9bd7493c2268 for server had01.hadoop/133.0.91.41:2181, unexpected error, closing socket connection and attempting reconnect
java.io.IOException: Broken pipe
        at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
        at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
        at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
        at sun.nio.ch.IOUtil.write(IOUtil.java:65)
        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:117)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:355)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1075)
2017-06-15 03:09:17,307 INFO org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore: Waiting for zookeeper to be connected, retry no. + 3
2017-06-15 03:09:17,329 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server had02.hadoop/133.0.91.42:2181. Will not attempt to authenticate using SASL (unknown error)

转载于:https://my.oschina.net/yulongblog/blog/1506168

weixin_34270606
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
【zookeeper】fsync-ing the write ahead log in took which will adversely effect operation latency
九师兄
08-26 617
在这篇博客中出现了如下。我想知道这个是在干嘛，会不会对我上面的那个问题造成影响。因此去查看了一下。
Zookeeper运维小结--CancelledKeyException
weixin_34121282的博客
02-24 539
https://www.jianshu.com/p/73eec030db86 项目中用到storm+kafka+zookeeper，在实际应用中zk和kafka常出问题，这里记录下在使用zk过程中的问题。 注：zk版本是3.4.8，kafka是0.8.2.0。zk、storm和kafka都是运行在同一个集群的三台机器上。 CancelledKeyException 在开发环境测试的时候，一直没...
java.nio.channels.CancelledKeyException
weixin_33953384的博客
11-30 368
使用nio来进行多线程编程，服务器端缓存了每个客户端连接的 key，当客户端网络断开/离线，非正常退出时，服务器端仍然用缓存的 key 来进行操作，就产生了这个异常。 完整异常信息： org.apache.catalina.core.StandardWrapperValve invoke 严重: Servlet.service() for servle...
ZooKeeper的配置文件优化性能（转）
angou6476的博客
09-10 571
一、前言 ZooKeeper的功能特性通过ZooKeeper配置文件来进行控制管理（zoo.cfg配置文件）。 ZooKeeper这样的设计其实是有它自身的原因的。通过前面对ZooKeeper的配置可以看出，对ZooKeeper集群进行配置的时候，它的配置文档是完全相同的（对于集群伪分布模式来说，只有很少的部分是不同的）。这样的配置方使得在部署ZooKeeper服务的时候非常地方便。...
zookeeper3.5.5镜像
06-29
zookeeper3.5.5镜像
zookeeper连接工具
07-05
zookeeper连接工具,可视化数据查看
ZooKeeper安装部署
01-06
zookeeper安装与配置 以下是Zookeeper的安装和配置步骤： 1. 首先，你需要从Zookeeper官方网站（https://zookeeper.apache.org/releases.html）下载Zookeeper的二进制发行版。选择合适的版本并下载压缩包。 2. 将...
Zookeeper-基础
12-19
Zookeeper-基础
zookeeper集群安装
08-24
ZooKeeper 集群安装 ZooKeeper 是一个开源的分布式协调服务，用于提供配置管理、名字服务、分布式同步和组服务等功能。下面是 ZooKeeper 集群安装的详细步骤和相关知识点。 Step 1: 安装 Java JDK 在安装 ...
【zookeeper】zookeeper 性能优化与配置详解 一些踩坑建议
九师兄
09-15 5106
ZooKeeper的功能特性通过ZooKeeper配置文件来进行控制管理（ zoo.cfg配置文件）。ZooKeeper这样的设计其实是有它自身的原因的。通过前面对ZooKeeper的配置可以看出，对ZooKeeper集群进行配置的时候，它的配置文档是完全相同的（对于集群伪分布模式来说，只有很少的部分是不同的）。这样的配置方使得在部署ZooKeeper服务的时候非常地方便。另外，如果服务器使用不同的配置文件，必须要确保不同配置文件中的服务器列表相匹配。
zookeeper频繁异常问题分析
热门推荐
09-02 10万+
1.1            操作系统信息 1.1.1          CPU信息 从上述可以看出在问题点CPU使用率并不高，故排除CPU问题。 1.1.2          内存信息   CM监控界面并无内存信息，无法提供图片。 但问题期间，监控机器内存，发现还有剩余，故排除内存问题。 1.2            ZKFAILOVER日志信息   2017-09-0
Zookeeper：fsync超时导致实例异常
weixin_30563319的博客
02-20 514
一、问题描述 2019-02-19 08:44左右，实时计算服务重启，报错显示找不到zk集群的leader节点，同时ZooKeeper集群有告警显示连接超时： 　　指标[连接耗时(ms)=18221]符合告警规则[连接耗时(ms)>=3000] 二、排查过程 查看当前集群状态，集群状态正常，查看报“连接超时”的zk节点日志，发现有shutdown情况，表明作为...
一次生产zookeeper sessionMovedException排查分析
trntaken的博客
06-07 1151
生产问题 1.生产zk集群采用5节点3机房 2-2-1架构部署,zookeeper客户端使用的curator框架,网络抖动情况下客户端发生了session moved异常,这个异常只要出现就会不停的报session moved异常，异常信息如下： 16:56:05ERROR [soa_roomviewers_s10_1_175_111_16024_PullJob-Quartz-worker]c.v.s.j.b.AbstractElasticJob- [PullJob] msg=org....
ZooKeeper运行报错KeeperErrorCode = NoNode for .....
椰子真的是只猫
08-22 9万+
最近遇到一个很尴尬的报错，org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode KeeperErrorCode = NoNode for
对Zookeeper的一些分析
weixin_30381793的博客
06-08 275
zookeeper不是为高可用性设计的由于要跨机房容灾，很多系统实际上是需要跨机房部署的。出于性价比的考虑我们通常会让多个机房同时工作，而不会搭建N倍的冗余。也就是说单个机房肯定撑不住全流量（你能设想谷歌在全球只剩下一个机房在干活吗）。由于zookeeper集群只能有一个master，因此一旦机房之间连接出现故障，zookeeper master就只能照顾一个机房，其他机房运行的业务模块由于没...
zookeeper节点启动失败
gongmf的专栏
10-12 4384
环境：zookeeper版本号：3.4.6，三个节点。 有一台服务器重启了，用zkServer.sh  start 启动是出错，信息如下： [2016-10-11 10:47:49,715] ERROR Unable to load database on disk (org.apache.zookeeper.server.quorum.QuorumPeer) java.io.IOEx
Zookeeper源码阅读（三）数据存储-日志
a814333256的博客
09-17 372
前言 接着上篇的内容，上一篇主要说了下Zookeeper内部的数据存储基本单位datanode和内存中维护的Datatree，还有和配额有关的三个实体类。具体在zk初始化时是怎么生成datatree等等后面会说。这一篇主要说下事务日志，后面也有用到。 事务日志 关于zookeeper事务日志的格式，可以参考下 zk事务日志，其实简单说也就是header+body两个部分，header有版...
zookeeper 启动
最新发布
04-05
ZooKeeper是一个开源的分布式协调服务，它提供了一个简单的接口，用于处理分布式应用程序中的一些关键任务，如统一命名服务、配置管理、分布式锁和分布式队列等。下面是ZooKeeper启动的步骤： 1. 下载和安装ZooKeeper：首先，你需要从ZooKeeper官方网站（https://zookeeper.apache.org/）下载ZooKeeper的二进制文件，并解压到你的机器上。 2. 配置ZooKeeper：在解压后的目录中，找到`conf`文件夹，并复制`zoo_sample.cfg`文件并重命名为`zoo.cfg`。然后，编辑`zoo.cfg`文件，根据你的需求进行配置，例如设置数据目录、客户端端口等。 3. 启动ZooKeeper服务器：在命令行中，进入ZooKeeper的安装目录，并执行以下命令启动ZooKeeper服务器： ``` bin/zkServer.sh start ``` 4. 检查ZooKeeper状态：执行以下命令可以检查ZooKeeper服务器的状态： ``` bin/zkServer.sh status ``` 如果输出显示为`Mode: leader`或`Mode: follower`，则表示ZooKeeper服务器已成功启动。 5. 连接到ZooKeeper：你可以使用ZooKeeper提供的命令行客户端或编程语言的ZooKeeper库来连接到ZooKeeper服务器，并执行相应的操作。

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值