zookeeper提示包太大问题

现象描述

beeline访问zookeeper报错

beeline报错
(base) -bash-4.2$ ./beeline -u 'jdbc:hive2://10.169.49.131:2181/default?spark.yarn.queue=szopam-test;username=leon'
Connecting to jdbc:hive2://10.169.49.131:2181/default?spark.yarn.queue=szopam-test;username=leon
21/11/12 16:51:44 INFO jdbc.Utils: Supplied authorities: 10.169.49.131:2181
21/11/12 16:51:44 INFO jdbc.Utils: Resolved authority: 10.169.49.131:2181
21/11/12 16:51:44 WARN jdbc.HiveConnection: Failed to connect to 10.169.49.131:2181
Unexpected end of file when reading from HS2 server. The root cause might be too many concurrent connections. Please ask the administrator to check the number of active connections, and adjust hive.server2.thrift.max.worker.threads if applicable.
Error: Could not open client transport with JDBC Uri: jdbc:hive2://10.169.49.131:2181/default?spark.yarn.queue=szopam-test;username=leon: null (state=08S01,code=0)
Beeline version 2.3.7 by Apache Hive
beeline>

zookeeper日志

java.io.IOException: Len error. A message from /10.128.151.5:36498 with advertised length of 16777216 is either a malformed message or too large to process (length is greater than jute.maxbuffer=1048575)
    at org.apache.zookeeper.server.NIOServerCnxn.readLength(NIOServerCnxn.java:549)
    at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:339)
    at org.apache.zookeeper.server.NIOServerCnxnFactory$IOWorkRequest.doWork(NIOServerCnxnFactory.java:508)
    at org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:154)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

 

原因分析 

客户端发送的包太大，超过jute.maxbuffer的设置，默认大小为1048575。

解决方法

修改jute.maxbuffer的配置，设置成50M

bin/zkServer.sh

JVMFLAGS="$JVMFLAGS -Djute.maxbuffer=50000000"