jmh使用笔记

pom.xml

pom文件可以通过archtype来生成，如果是java11这样的高版本需要加入javax.annotation-api，最终得到的代码如下

<!--
Copyright (c) 2005, 2014, Oracle and/or its affiliates. All rights reserved.
DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.

This code is free software; you can redistribute it and/or modify it
under the terms of the GNU General Public License version 2 only, as
published by the Free Software Foundation.  Oracle designates this
particular file as subject to the "Classpath" exception as provided
by Oracle in the LICENSE file that accompanied this code.

This code is distributed in the hope that it will be useful, but WITHOUT
ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
version 2 for more details (a copy is included in the LICENSE file that
accompanied this code).

You should have received a copy of the GNU General Public License version
2 along with this work; if not, write to the Free Software Foundation,
Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA.

Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA
or visit www.oracle.com if you need additional information or have any
questions.
-->

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>qbit</groupId>
    <artifactId>jmh</artifactId>
    <version>1.0-SNAPSHOT</version>
    <packaging>jar</packaging>

    <name>Auto-generated JMH benchmark</name>

    <prerequisites>
        <maven>3.0</maven>
    </prerequisites>

    <dependencies>
        <dependency>
            <groupId>org.openjdk.jmh</groupId>
            <artifactId>jmh-core</artifactId>
            <version>${jmh.version}</version>
        </dependency>
        <dependency>
            <groupId>org.openjdk.jmh</groupId>
            <artifactId>jmh-generator-annprocess</artifactId>
            <version>${jmh.version}</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>javax.annotation</groupId>
            <artifactId>javax.annotation-api</artifactId>
            <version>1.3.2</version>
        </dependency>
    </dependencies>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <jmh.version>1.0</jmh.version>
        <javac.target>1.6</javac.target>
        <uberjar.name>benchmarks</uberjar.name>
    </properties>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.1</version>
                <configuration>
                    <compilerVersion>${javac.target}</compilerVersion>
                    <source>11</source>
                    <target>11</target>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>2.2</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <finalName>${uberjar.name}</finalName>
                            <transformers>
                                <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                    <mainClass>org.openjdk.jmh.Main</mainClass>
                                </transformer>
                            </transformers>
                            <filters>
                                <filter>
                                    <!--
                                        Shading signed JARs will fail without this.
                                        http://stackoverflow.com/questions/999489/invalid-signature-file-when-attempting-to-run-a-jar
                                    -->
                                    <artifact>*:*</artifact>
                                    <excludes>
                                        <exclude>META-INF/*.SF</exclude>
                                        <exclude>META-INF/*.DSA</exclude>
                                        <exclude>META-INF/*.RSA</exclude>
                                    </excludes>
                                </filter>
                            </filters>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
        <pluginManagement>
            <plugins>
                <plugin>
                    <artifactId>maven-clean-plugin</artifactId>
                    <version>2.5</version>
                </plugin>
                <plugin>
                    <artifactId>maven-deploy-plugin</artifactId>
                    <version>2.8.1</version>
                </plugin>
                <plugin>
                    <artifactId>maven-install-plugin</artifactId>
                    <version>2.5.1</version>
                </plugin>
                <plugin>
                    <artifactId>maven-jar-plugin</artifactId>
                    <version>2.4</version>
                </plugin>
                <plugin>
                    <artifactId>maven-javadoc-plugin</artifactId>
                    <version>2.9.1</version>
                </plugin>
                <plugin>
                    <artifactId>maven-resources-plugin</artifactId>
                    <version>2.6</version>
                </plugin>
                <plugin>
                    <artifactId>maven-site-plugin</artifactId>
                    <version>3.3</version>
                </plugin>
                <plugin>
                    <artifactId>maven-source-plugin</artifactId>
                    <version>2.2.1</version>
                </plugin>
                <plugin>
                    <artifactId>maven-surefire-plugin</artifactId>
                    <version>2.17</version>
                </plugin>
            </plugins>
        </pluginManagement>
    </build>

</project>

java

测试方法

然后写一个java文件，把各种要对比的情况分别写一个方法，入参一般是Blackhole这样可以防止jvm的优化，一个最佳实践是写一个baseline方法用来考察测试代码本身的消耗，baseline的声明一般如下：

	@Benchmark
    @Threads(1)
    public void baseline(Blackhole bh) 

main方法

用一个main方法来进行配置并运行测试，一般来说代码如下：

public static void main(String[] args) throws RunnerException {
        Options options = new OptionsBuilder().include(QbitBenchmarks.class.getName()).warmupIterations(1)
                .warmupForks(1).measurementIterations(2).forks(1).build();
        new Runner(options).run();
    }

可以看到里面有很多参数但是有时却不生效，有空我再研究下

类注解

@State(Scope.Benchmark)
@OutputTimeUnit(TimeUnit.SECONDS)
@BenchmarkMode(Mode.AverageTime)

运行

使用maven的package后再target下得到benchmarks.jar，运行即可。如果日志太多可以写文件：

java -jar target/benchmarks.jar >benchmarks.log

结果解析

运行后会得到类似下面的日志

# VM invoker: /usr/lib/jvm/java-11-openjdk-amd64/bin/java
# VM options: <none>
# Warmup: 20 iterations, 1 s each
# Measurement: 20 iterations, 1 s each
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Average time, time/op
# Benchmark: qbit.QbitBenchmarks.baseline

# Run progress: 0.00% complete, ETA 00:20:00
# Fork: 1 of 10
# Warmup Iteration   1: 0.111 s/op
......
# Warmup Iteration  20: 0.025 s/op
Iteration   1: 0.019 s/op
......
Iteration  20: 0.031 s/op

# Run progress: 3.33% complete, ETA 00:23:24
# Fork: 2 of 10
# Warmup Iteration   1: 0.141 s/op
# Warmup Iteration   2: 0.059 s/op
......
# Warmup Iteration  20: 0.023 s/op
Iteration   1: 0.027 s/op
Iteration   2: 0.015 s/op
......
Iteration  20: 0.015 s/op

......

Result: 0.022 ±(99.9%) 0.002 s/op [Average]
  Statistics: (min, avg, max) = (0.014, 0.022, 0.053), stdev = 0.008
  Confidence interval (99.9%): [0.020, 0.024]


# VM invoker: /usr/lib/jvm/java-11-openjdk-amd64/bin/java
# VM options: <none>
# Warmup: 20 iterations, 1 s each
# Measurement: 20 iterations, 1 s each
# Threads: 1 thread, will synchronize iterations
......

上面是对测试方法逐个运行，每个运行10Fork,然后每个Fork里进行20次Warmup Iteration和20次Iteration.最后在Result里对结果进行了汇总，会有min，avg，max，分别表示最小，平均，最大值。更进一步还有stdev表示标准差，Confidence interval表示置信区间。在所有的测试方法运行完毕后会有一次汇总，汇总信息要简要些，主要是平均值Score，另外Score error看上去像方差。

troubleshooting

javax.annotation.Generated

jdk11及其以上移除了该包,可以使用jar替代，具体参见上面pom

一般来说jmh无法直接在IDE里运行，必须先使用maven打成jar然后执行

connect failed: Connection refused

无法通过IDE运行，需要maven打包并在命令行运行

Address already in use

问题原因：Java的 Socket.close() 并不能立即释放连接，会造成一段时间内 TIME_WAIT 的状态，操作系统可供使用的连接数有默认限制，很快会被占满
解决办法

1. Tomcat中Connector配置里增加辅助参数（经验证无效）
2. 切换为linux
3. 修改操作系统注册表（经验证能够解决问题）

• 打开注册表：regedit
• HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\ Services\TCPIP\Parameters
• 新建 DWORD值，name：TcpTimedWaitDe，value：30（十进制） –> 设置为30秒，默认是240秒
• 新建 DWORD值，name：MaxUserPort，value：65534（十进制） –> 设置最大连接数65534
• 重启系统

maven clean报错

由于是在命令行运行，在windows下运行的时候Ctrl+C并不能关闭正在运行的测试，即使把Intellij关闭也只是让其后台运行，所以benchamrks.jar是无法删除的，解决方案只能是在任务管理器里杀死。另外这个问题也会导致有的时候jar删除了但是执行java -jar后运行的仍然是老的jar