你曾经希望能够将Java对象object快速地转化为字节流,就像原生语言C/C++?如果你使用标准的Java Serialization
,你会对它的性能感到非常失望。Java Serialization
针对不同的使用目的,而不仅仅是快速紧凑地序列化对象。
那为什么需要快速紧凑的序列化?我们遇到的大多数系统时分布式的,我们需要在处理任务之间有效地通过传递状态(state)进行交流,这个状态在我们的对象中。所碰到的许多系统,相当大的一部分代价是在byte buffers序列化状态(state)。目前已有一大批协议和机制实现了这种需求,其中的一部分协议使用简单,但是非常低效,如Java Serialisation
, XML 和JSON。另一部分采用二进制协议,性能高效,但是需要深入理解其技术。
这篇文章,将会使用简单的二进制协议和Java的相关技术,达到与原生语言 C 或者 C++ 一样的性能。
三种方法:
- Java
Serialization
:实现Serializable接口; - Binary via
ByteBuffer
: 使用ByteBuffer API以二进制的方式写入对象的field;这也是好的二进制编码方法的基准。 - Binary via
Unsafe
:引入Unsafe类及其方法,使用direct memory。
import sun.misc.Unsafe;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.ObjectInputStream;
import java.io.ObjectOutputStream;
import java.io.Serializable;
import java.lang.reflect.Field;
import java.nio.ByteBuffer;
import java.util.Arrays;
@SuppressWarnings("restriction")
public final class TestSerialisationPerf {
public static final int REPETITIONS = 1 * 1000 * 1000;
private static ObjectToBeSerialised ITEM = new ObjectToBeSerialised(1010L, true, 777, 99,
new double[] { 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0 },
new long[] { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 });
public static void main(final String[] arg) throws Exception {
for (final PerformanceTestCase testCase : testCases) {
for (int i = 0; i < 5; i++) {
testCase.performTest();
System.out.format("%d %s\twrite=%,dns read=%,dns total=%,dns\n", i, testCase.getName(),
testCase.getWriteTimeNanos(), testCase.getReadTimeNanos(),
testCase.getWriteTimeNanos() + testCase.getReadTimeNanos());
if (!ITEM.equals(testCase.getTestOutput())) {
throw new IllegalStateException("Objects do not match");
}
System.gc();
Thread.sleep(3000);
}
}
}
private static final PerformanceTestCase[] testCases = {
new PerformanceTestCase("Serialisation", REPETITIONS, ITEM) {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
@Override
public void testWrite(ObjectToBeSerialised item) throws Exception {
for (int i = 0; i < REPETITIONS; i++) {
baos.reset();
ObjectOutputStream oos = new ObjectOutputStream(baos);
oos.writeObject(item);
oos.close();
}
}
@Override
public ObjectToBeSerialised testRead() throws Exception {
ObjectToBeSerialised object = null;
for (int i = 0; i < REPETITIONS; i++) {
ByteArrayInputStream bais = new ByteArrayInputStream(baos.toByteArray());
ObjectInputStream ois = new ObjectInputStream(bais);
object = (ObjectToBeSerialised) ois.readObject();
}
return object;
}
},
new PerformanceTestCase("ByteBuffer", REPETITIONS, ITEM) {
ByteBuffer byteBuffer = ByteBuffer.allocate(1024);
@Override
public void testWrite(ObjectToBeSerialised item) throws Exception {
for (int i = 0; i < REPETITIONS; i++) {
byteBuffer.clear();
item.write(byteBuffer);
}
}
@Override
public ObjectToBeSerialised testRead() throws Exception {
ObjectToBeSerialised object = null;
for (int i = 0; i < REPETITIONS; i++) {
byteBuffer.flip();
object = ObjectToBeSerialised.read(byteBuffer);
}
return object;
}
},
new PerformanceTestCase("UnsafeMemory", REPETITIONS, ITEM) {
UnsafeMemory buffer = new UnsafeMemory(new byte[1024]);
@Override
public void testWrite(ObjectToBeSerialised item) throws Exception {
for (int i = 0; i < REPETITIONS; i++) {
buffer.reset();
item.write(buffer);
}
}
@Override
public ObjectToBeSerialised testRead() throws Exception {
ObjectToBeSerialised object = null;
for (int i = 0; i < REPETITIONS; i++) {
buffer.reset();
object = ObjectToBeSerialised.read(buffer);
}
return object;
}
}
};
}
abstract class PerformanceTestCase {
private final String name;
private final int repetitions;
private final ObjectToBeSerialised testInput;
private ObjectToBeSerialised testOutput;
private long writeTimeNanos;
private long readTimeNanos;
public PerformanceTestCase(final String name, final int repetitions, final ObjectToBeSerialised testInput) {
this.name = name;
this.repetitions = repetitions;
this.testInput = testInput;
}
public String getName() {
return name;
}
public ObjectToBeSerialised getTestOutput() {
return testOutput;
}
public long getWriteTimeNanos() {
return writeTimeNanos;
}
public long getReadTimeNanos() {
return readTimeNanos;
}
public void performTest() throws Exception {
final long startWriteNanos = System.nanoTime();
testWrite(testInput);
writeTimeNanos = (System.nanoTime() - startWriteNanos) / repetitions;
final long startReadNanos = System.nanoTime();
testOutput = testRead();
readTimeNanos = (System.nanoTime() - startReadNanos) / repetitions;
}
public abstract void testWrite(ObjectToBeSerialised item) throws Exception;
public abstract ObjectToBeSerialised testRead() throws Exception;
}
class ObjectToBeSerialised implements Serializable {
private static final long serialVersionUID = 10275539472837495L;
private final long sourceId;
private final boolean special;
private final int orderCode;
private final int priority;
private final double[] prices;
private final long[] quantities;
public ObjectToBeSerialised(final long sourceId, final boolean special, final int orderCode, final int priority,
final double[] prices, final long[] quantities) {
this.sourceId = sourceId;
this.special = special;
this.orderCode = orderCode;
this.priority = priority;
this.prices = prices;
this.quantities = quantities;
}
public void write(final ByteBuffer byteBuffer) {
byteBuffer.putLong(sourceId);
byteBuffer.put((byte) (special ? 1 : 0));
byteBuffer.putInt(orderCode);
byteBuffer.putInt(priority);
byteBuffer.putInt(prices.length);
for (final double price : prices) {
byteBuffer.putDouble(price);
}
byteBuffer.putInt(quantities.length);
for (final long quantity : quantities) {
byteBuffer.putLong(quantity);
}
}
public static ObjectToBeSerialised read(final ByteBuffer byteBuffer) {
final long sourceId = byteBuffer.getLong();
final boolean special = 0 != byteBuffer.get();
final int orderCode = byteBuffer.getInt();
final int priority = byteBuffer.getInt();
final int pricesSize = byteBuffer.getInt();
final double[] prices = new double[pricesSize];
for (int i = 0; i < pricesSize; i++) {
prices[i] = byteBuffer.getDouble();
}
final int quantitiesSize = byteBuffer.getInt();
final long[] quantities = new long[quantitiesSize];
for (int i = 0; i < quantitiesSize; i++) {
quantities[i] = byteBuffer.getLong();
}
return new ObjectToBeSerialised(sourceId, special, orderCode, priority, prices, quantities);
}
public void write(final UnsafeMemory buffer) {
buffer.putLong(sourceId);
buffer.putBoolean(special);
buffer.putInt(orderCode);
buffer.putInt(priority);
buffer.putDoubleArray(prices);
buffer.putLongArray(quantities);
}
public static ObjectToBeSerialised read(final UnsafeMemory buffer) {
final long sourceId = buffer.getLong();
final boolean special = buffer.getBoolean();
final int orderCode = buffer.getInt();
final int priority = buffer.getInt();
final double[] prices = buffer.getDoubleArray();
final long[] quantities = buffer.getLongArray();
return new ObjectToBeSerialised(sourceId, special, orderCode, priority, prices, quantities);
}
@Override
public boolean equals(final Object o) {
if (this == o) {
return true;
}
if (o == null || getClass() != o.getClass()) {
return false;
}
final ObjectToBeSerialised that = (ObjectToBeSerialised) o;
if (orderCode != that.orderCode) {
return false;
}
if (priority != that.priority) {
return false;
}
if (sourceId != that.sourceId) {
return false;
}
if (special != that.special) {
return false;
}
if (!Arrays.equals(prices, that.prices)) {
return false;
}
if (!Arrays.equals(quantities, that.quantities)) {
return false;
}
return true;
}
}
@SuppressWarnings("restriction")
class UnsafeMemory {
private static final Unsafe unsafe;
static {
try {
Field field = Unsafe.class.getDeclaredField("theUnsafe");
field.setAccessible(true);
unsafe = (Unsafe) field.get(null);
} catch (Exception e) {
throw new RuntimeException(e);
}
}
private static final long byteArrayOffset = unsafe.arrayBaseOffset(byte[].class);
private static final long longArrayOffset = unsafe.arrayBaseOffset(long[].class);
private static final long doubleArrayOffset = unsafe.arrayBaseOffset(double[].class);
private static final int SIZE_OF_BOOLEAN = 1;
private static final int SIZE_OF_INT = 4;
private static final int SIZE_OF_LONG = 8;
private int pos = 0;
private final byte[] buffer;
public UnsafeMemory(final byte[] buffer) {
if (null == buffer) {
throw new NullPointerException("buffer cannot be null");
}
this.buffer = buffer;
}
public void reset() {
this.pos = 0;
}
public void putBoolean(final boolean value) {
unsafe.putBoolean(buffer, byteArrayOffset + pos, value);
pos += SIZE_OF_BOOLEAN;
}
public boolean getBoolean() {
boolean value = unsafe.getBoolean(buffer, byteArrayOffset + pos);
pos += SIZE_OF_BOOLEAN;
return value;
}
public void putInt(final int value) {
unsafe.putInt(buffer, byteArrayOffset + pos, value);
pos += SIZE_OF_INT;
}
public int getInt() {
int value = unsafe.getInt(buffer, byteArrayOffset + pos);
pos += SIZE_OF_INT;
return value;
}
public void putLong(final long value) {
unsafe.putLong(buffer, byteArrayOffset + pos, value);
pos += SIZE_OF_LONG;
}
public long getLong() {
long value = unsafe.getLong(buffer, byteArrayOffset + pos);
pos += SIZE_OF_LONG;
return value;
}
public void putLongArray(final long[] values) {
putInt(values.length);
long bytesToCopy = values.length << 3;
unsafe.copyMemory(values, longArrayOffset, buffer, byteArrayOffset + pos, bytesToCopy);
pos += bytesToCopy;
}
public long[] getLongArray() {
int arraySize = getInt();
long[] values = new long[arraySize];
long bytesToCopy = values.length << 3;
unsafe.copyMemory(buffer, byteArrayOffset + pos, values, longArrayOffset, bytesToCopy);
pos += bytesToCopy;
return values;
}
public void putDoubleArray(final double[] values) {
putInt(values.length);
long bytesToCopy = values.length << 3;
unsafe.copyMemory(values, doubleArrayOffset, buffer, byteArrayOffset + pos, bytesToCopy);
pos += bytesToCopy;
}
public double[] getDoubleArray() {
int arraySize = getInt();
double[] values = new double[arraySize];
long bytesToCopy = values.length << 3;
unsafe.copyMemory(buffer, byteArrayOffset + pos, values, doubleArrayOffset, bytesToCopy);
pos += bytesToCopy;
return values;
}
}
结果:
0 Serialisation write=3,131ns read=9,111ns total=12,242ns
1 Serialisation write=4,081ns read=12,723ns total=16,804ns
2 Serialisation write=4,337ns read=16,147ns total=20,484ns
3 Serialisation write=4,274ns read=11,520ns total=15,794ns
4 Serialisation write=4,392ns read=12,414ns total=16,806ns
0 ByteBuffer write=525ns read=564ns total=1,089ns
1 ByteBuffer write=459ns read=642ns total=1,101ns
2 ByteBuffer write=436ns read=638ns total=1,074ns
3 ByteBuffer write=435ns read=649ns total=1,084ns
4 ByteBuffer write=429ns read=637ns total=1,066ns
0 UnsafeMemory write=96ns read=220ns total=316ns
1 UnsafeMemory write=74ns read=217ns total=291ns
2 UnsafeMemory write=78ns read=216ns total=294ns
3 UnsafeMemory write=79ns read=213ns total=292ns
4 UnsafeMemory write=77ns read=214ns total=291ns
分析
使用Java Serialization
写入和读取单个相对比较小的对象需要花费大约15,000ns,然而使用Unsafe
能够下降到300ns。有许多原因可以解释为什么Java Serialisation
代价高昂,如每个对象写入类的全名和版本信息,而且ObjectOutputStream保留了一系列写入对象,当调用close()
方法才进行合并。对每个样本对象,Java Serialisation
需要340字节,而使用二进制的方式只需要185字节。Java Serialisation
的格式细节参考这里。如果在样本的绝大多数不使用数组,那么序列化后的对象会远大于Java Serialisation
,因为字段名称。
使用Unsafe
的方法获取了性能的提升,因为在Hotspot和其他的JVMs中,优化器对待这些操作作为指令,替换成汇编指令调用执行内存的操作。
结论
Java序列化实现像原生C/C++一样的性能是有可能的。上面的UnsafeMemory已提供基本框架的实现,可以进行扩充封装这种行为,这样就可以使用这种锋利的工具解决一些潜在的问题。
原文:Native C/C++ Like Performance For Java Object Serialization