Byteman!Byteman!

最近生产应用报错：
23-Oct-2019 13:59:08.592 严重 [http-nio-7001-exec-9] org.apache.coyote.http11.Http11Processor.endRequest Error finishing response
java.nio.BufferOverflowException
at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:183)
at org.apache.coyote.http11.Http11OutputBuffer.write(Http11OutputBuffer.java:454)
at org.apache.coyote.http11.Http11OutputBuffer.write(Http11OutputBuffer.java:439)
at org.apache.coyote.http11.Http11OutputBuffer.sendHeader(Http11OutputBuffer.java:399)
at org.apache.coyote.http11.Http11Processor.prepareResponse(Http11Processor.java:1277)
at org.apache.coyote.AbstractProcessor.action(AbstractProcessor.java:263)
at org.apache.coyote.http11.Http11Processor.endRequest(Http11Processor.java:1457)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:823)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:789)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1455)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)

响应头的buffer不够，导致BufferOverflow了，经排查应该是应用某个页面响应头设置超过8KB导致，但是哪个页面呢？有方法跟踪吗？答案是Byteman

Byteman是JBOSS的一个开源程序注入、跟踪工具，目前最新版为4.0.8，运行JDK6以上JVM
以下为我跟踪Tomcat的例子（TOMCAT版本为8.5.8），进程ID为41275：
1、安装Byteman代理至一个正运行的JAVA程序：

[root@localhost bin]# bminstall.sh -b -Dorg.jboss.byteman.transform.all 41275
/mnt/byteman-download-4.0.8/lib/byteman-install.jar:/usr/java/jdk1.7.0_67/lib/tools.jar:/mnt/byteman-download-4.0.8/lib/byteman.jar
byteman jar is /mnt/byteman-download-4.0.8/lib/byteman.jar

安装后日志中显示：

Setting org.jboss.byteman.transform.all=

注意：此方法安装的Byteman默认监听9091端口，也可以指定自定义监听端口，如指定端口55000，指定后，下面提交规则文件也需要跟-p 55000 ：
bminstall.sh -b -Dorg.jboss.byteman.transform.all -p 55000 41275
另外注意：可以直接在程序运行时增加agent的方式来安装Byteman，如增加参数：
-javaagent:${BYTEMAN_HOME}/lib/byteman.jar=listener:true
此方式默认监听端口号是9091
2、写注入规则文件，以下规则指定跟踪org.apache.coyote.Response类和org.apache.coyote.http11.Http11OutputBuffer类对应的相关方法，并输出
vim showlocalvar.btm
RULE trace http buffer size
CLASS org.apache.coyote.http11.Http11OutputBuffer
METHOD write
AT ENTRY
IF $0.headerBuffer.position() >= 0
DO traceln("*** RESPONSE HEADER SIZE: " + $0.headerBuffer.capacity() + " POSITION:" + $0.headerBuffer.position() + " LIMIT:" + $0.headerBuffer.limit() + “***”)
ENDRULE

RULE trace send headers begin
CLASS org.apache.coyote.Response
METHOD sendHeaders
AT ENTRY
IF TRUE
DO traceln("*** BEGIN SEND RESPONSE HEADERS … Request_URL: " + $0.getRequest().getHeader(“Referer”) + " ***")
ENDRULE

RULE trace send headers end
CLASS org.apache.coyote.Response
METHOD sendHeaders
AT EXIT
IF TRUE
DO traceln("*** END SEND RESPONSE HEADERS ***")
ENDRULE

3、检查规则文件

[root@localhost ~]# bmcheck.sh -cp /home/apache-tomcat-8.5.8-MJSC/lib/tomcat-coyote.jar showlocalvar.btm
/mnt/byteman-download-4.0.8/lib/byteman.jar:/home/apache-tomcat-8.5.8-MJSC/lib/tomcat-coyote.jar
Checking rule trace http buffer size against class org.apache.coyote.http11.Http11OutputBuffer
Parsed rule “trace http buffer size” for class org.apache.coyote.http11.Http11OutputBuffer
ERROR : Failed to check rule “trace http buffer size” loaded from showlocalvar.btm line 5 against method (org.apache.coyote.Response,int) void
java.lang.NoClassDefFoundError: Lorg/apache/tomcat/util/res/StringManager;

TestScript: 1 total errors
0 total warnings
0 parse errors
1 type errors
0 type warnings

-cp参数指出bmcheck.sh所需的包含org.apache.coyote.Response类和org.apache.coyote.http11.Http11OutputBuffer类的jar包位置及名字，否则bmcheck.sh找不到类所在位置
报错，找不到另一个类，该类在tomcat-util.jar包，因此

[root@localhost ~]# bmcheck.sh -cp /home/apache-tomcat-8.5.8-MJSC/lib/tomcat-coyote.jar:/home/apache-tomcat-8.5.8-MJSC/lib/tomcat-util.jar:/home/apache-tomcat-8.5.8-MJSC/bin/tomcat-juli.jar showlocalvar.btm
/mnt/byteman-download-4.0.8/lib/byteman.jar:/home/apache-tomcat-8.5.8-MJSC/lib/tomcat-coyote.jar:/home/apache-tomcat-8.5.8-MJSC/lib/tomcat-util.jar:/home/apache-tomcat-8.5.8-MJSC/bin/tomcat-juli.jar
Checking rule trace http buffer size against class org.apache.coyote.http11.Http11OutputBuffer
Parsed rule “trace http buffer size” for class org.apache.coyote.http11.Http11OutputBuffer
Type checked rule “trace http buffer size”
Parsed rule “trace http buffer size” for class org.apache.coyote.http11.Http11OutputBuffer
Type checked rule “trace http buffer size”
Parsed rule “trace http buffer size” for class org.apache.coyote.http11.Http11OutputBuffer
Type checked rule “trace http buffer size”
Parsed rule “trace http buffer size” for class org.apache.coyote.http11.Http11OutputBuffer
Type checked rule “trace http buffer size”

TestScript: no errors

4、提交规则文件并注入

[root@localhost ~]# bmsubmit.sh -l showlocalvar.btm
install rule trace http buffer size

再次执行bmsubmit.sh会显示规则文件在程序中的注入情况

[root@localhost ~]# bmsubmit.sh
# File showlocalvar.btm line 5
RULE trace http buffer size
CLASS org.apache.coyote.http11.Http11OutputBuffer
METHOD write
NOCOMPILE
AT ENTRY
IF TRUE
DO traceln("*** transfer value is : " + $0.headerBuffer.capacity() + " ***")
ENDRULE

5、卸载注入规则

[root@localhost ~]# bmsubmit.sh -u showlocalvar.btm
uninstall RULE trace http buffer size

[root@localhost ~]# bmsubmit.sh
no rules installed

跟踪结果：

[root@localhost logs]# tail -f catalina.out
*** BEGIN SEND RESPONSE HEADERS … Request_URL: http://10.191.60.254:17001/wcis/cis/workflow/console/workflowconsole.jsp ***
*** RESPONSE HEADER SIZE: 8192 POSITION:0 LIMIT:8192***
*** RESPONSE HEADER SIZE: 8192 POSITION:9 LIMIT:8192***
*** RESPONSE HEADER SIZE: 8192 POSITION:15 LIMIT:8192***
*** RESPONSE HEADER SIZE: 8192 POSITION:15 LIMIT:8192***
*** RESPONSE HEADER SIZE: 8192 POSITION:29 LIMIT:8192***
*** RESPONSE HEADER SIZE: 8192 POSITION:29 LIMIT:8192***
*** RESPONSE HEADER SIZE: 8192 POSITION:61 LIMIT:8192***
*** RESPONSE HEADER SIZE: 8192 POSITION:61 LIMIT:8192***
*** RESPONSE HEADER SIZE: 8192 POSITION:80 LIMIT:8192***
*** RESPONSE HEADER SIZE: 8192 POSITION:80 LIMIT:8192***
*** RESPONSE HEADER SIZE: 8192 POSITION:89 LIMIT:8192***
*** RESPONSE HEADER SIZE: 8192 POSITION:89 LIMIT:8192***
*** RESPONSE HEADER SIZE: 8192 POSITION:95 LIMIT:8192***
*** RESPONSE HEADER SIZE: 8192 POSITION:95 LIMIT:8192***
*** RESPONSE HEADER SIZE: 8192 POSITION:126 LIMIT:8192***
*** RESPONSE HEADER SIZE: 8192 POSITION:126 LIMIT:8192***
*** RESPONSE HEADER SIZE: 8192 POSITION:138 LIMIT:8192***
*** RESPONSE HEADER SIZE: 8192 POSITION:138 LIMIT:8192***
*** END SEND RESPONSE HEADERS ***

参考：https://developer.jboss.org/wiki/ABytemanTutorial#top
https://developer.jboss.org/thread/278603
https://developer.jboss.org/thread/266891