【Flume】flume中Avro Sink到Avro Source的性能测试，是否压缩，是否加密

从官方下载的flume-ng的源码里有单元测试的代码，本文就通过单元测试来体验下flume中avro的性能如何

本文中的一个event的body内容大小是1KB，读者可自行组织文本，达到1KB即可，方便测试时，性能的计算

先把我的代码贴出来

package flume.performance.test;

import org.apache.flume.Channel;
import org.apache.flume.Context;
import org.apache.flume.Event;
import org.apache.flume.EventDeliveryException;
import org.apache.flume.Transaction;
import org.apache.flume.channel.MemoryChannel;
import org.apache.flume.conf.Configurables;
import org.apache.flume.event.EventBuilder;
import org.apache.flume.sink.AvroSink;

import com.google.common.base.Charsets;

public class AvroTest {

	private static final String hostname = "192.168.11.176";
	private static final Integer port = 9520;
	private static int batchSize = 100;
	private static int batch = 1000;
	
	private static AvroSink sink;
	private static Channel channel;
	private static Context context;
	
	
	public static void setUp() {
		if (sink != null) {
			throw new RuntimeException("double setup");
		}
		sink = new AvroSink();
		channel = new MemoryChannel();
		context = new Context();

		context.put("hostname", hostname);
		context.put("port", String.valueOf(port));
		context.put("batch-size", String.valueOf(batchSize));
		context.put("connect-timeout", String.valueOf(2000L));
		context.put("request-timeout", String.valueOf(3000L));
	}
	
	public static void testWithNone() {
		dataTransport();
	}
	
	public static void testWithCompress() {
		context.put("compression-type", "deflate");
		dataTransport();
	}
	
	public static void testWithSSL() {
		context.put("ssl", "true");
		context.put("truststore", "/home/flume/keystore/chiwei.keystore");
		context.put("truststore-password", "123456");
		context.put("truststore-type", "JKS");
		dataTransport();
	}
	
	public static void testWithCompressAndSSL() {
		context.put("compression-type", "deflate");
		context.put("ssl", "true");
		context.put("truststore", "<span style="font-family: Arial, Helvetica, sans-serif;">/home/flume/keystore/chiwei.keystore</span>");
		context.put("truststore-password", "123456");
		context.put("truststore-type", "JKS");
		dataTransport();
	}
	
	public static void main(String args[]) {
		if(args.length<3) {
			System.out.println("Please Input 3 args: type[none,compress,ssl,comssl] batch(int) batchSize(int)");
			return;
		}
		String str = args[0];
		batchSize = Integer.parseInt(args[2]);
		batch = Integer.parseInt(args[1]);
		
		setUp();
		
		if("none".equals(str)) {
			testWithNone();
		}else if("compress".equals(str)) {
			testWithCompress();
		}else if("ssl".equals(str)) {
			testWithSSL();
		}else if("comssl".equals(str)) {
			testWithCompressAndSSL();
		}
		System.exit(0);
	}
	
	private static void dataTransport() {
		Configurables.configure(sink, context);
		Configurables.configure(channel, context);
		sink.setChannel(channel);
		Event event = EventBuilder
				.withBody("XXXXXX",Charsets.UTF_8);
		sink.start();
		long allBegin = System.currentTimeMillis();
		int a;
		for (a = 0; a < batch; a++) {
			Transaction transaction = channel.getTransaction();

			transaction.begin();
			int i;
			for (i = 0; i < batchSize; i++) {
				channel.put(event);
			}
			transaction.commit();
			transaction.close();
			try {
				sink.process();
			} catch (EventDeliveryException e) {
				e.printStackTrace();
			}
		}
		System.out.println("Sum Time = "
				+ (System.currentTimeMillis() - allBegin)+" ms");
		System.out.println("How many batch ? --> " + batch);
		System.out.println("How many events in one batch ? --> " + batchSize);
	}
	
}

请自行组织文本内容1KB。

在对avro进行测试的时候，分为四种情况测试

本人本机到远程flume是通过vpn连上的，网络可能是一个原因

下面将四中情况的测试结果数据展示下（1000批数据，每批100条，总共104M数据）

1、不加密，不压缩

148秒，性能=0.70M/S

2、不加密，压缩

171秒，性能=0.61M/S

3、加密，不压缩

168秒，性能=0.62M/S

4、加密，压缩

141秒，性能=0.73M/S

下面再给出一个同一内网中的测试数据，性能明显提升，看来我的VPN不给力啊：

1、不加密，不压缩

java -jar avroTest-1.0.jar none 1000 100

Sum Time = 79968 ms
How many batch ? --> 1000
How many events in one batch ? --> 100

1.3M/S

2、不加密，压缩

Sum Time = 76740 ms
How many batch ? --> 1000
How many events in one batch ? --> 100

1.35M/S

3、加密，不压缩

Sum Time = 88234 ms
How many batch ? --> 1000
How many events in one batch ? --> 100

1.18M/S

4、加密，压缩

Sum Time = 77825 ms
How many batch ? --> 1000
How many events in one batch ? --> 100

1.33M/S

注意：

以上测试前提是事务容量100，因为测试的可变参数太多，希望大家自己尝试调整参数进行测试。

以上测试结果数据，只限于本次测试，多次测试，不同网络环境的测试，对结果都是有影响的，在此只是为了提供一个测试的方法而已，望大家不吝指教！！！