從官方下載的flume-ng的原始碼裡有單元測試的程式碼，本文就通過單元測試來體驗下flume中avro的效能如何

本文中的一個event的body內容大小是1KB，讀者可自行組織文字，達到1KB即可，方便測試時，效能的計算

先把我的程式碼貼出來

package flume.performance.test;

import org.apache.flume.Channel;
import org.apache.flume.Context;
import org.apache.flume.Event;
import org.apache.flume.EventDeliveryException;
import org.apache.flume.Transaction;
import org.apache.flume.channel.MemoryChannel;
import org.apache.flume.conf.Configurables;
import org.apache.flume.event.EventBuilder;
import org.apache.flume.sink.AvroSink;

import com.google.common.base.Charsets;

public class AvroTest {

	private static final String hostname = "192.168.11.176";
	private static final Integer port = 9520;
	private static int batchSize = 100;
	private static int batch = 1000;
	
	private static AvroSink sink;
	private static Channel channel;
	private static Context context;
	
	
	public static void setUp() {
		if (sink != null) {
			throw new RuntimeException("double setup");
		}
		sink = new AvroSink();
		channel = new MemoryChannel();
		context = new Context();

		context.put("hostname", hostname);
		context.put("port", String.valueOf(port));
		context.put("batch-size", String.valueOf(batchSize));
		context.put("connect-timeout", String.valueOf(2000L));
		context.put("request-timeout", String.valueOf(3000L));
	}
	
	public static void testWithNone() {
		dataTransport();
	}
	
	public static void testWithCompress() {
		context.put("compression-type", "deflate");
		dataTransport();
	}
	
	public static void testWithSSL() {
		context.put("ssl", "true");
		context.put("truststore", "/home/flume/keystore/chiwei.keystore");
		context.put("truststore-password", "123456");
		context.put("truststore-type", "JKS");
		dataTransport();
	}
	
	public static void testWithCompressAndSSL() {
		context.put("compression-type", "deflate");
		context.put("ssl", "true");
		context.put("truststore", "<span style="font-family: Arial, Helvetica, sans-serif;">/home/flume/keystore/chiwei.keystore</span>");
		context.put("truststore-password", "123456");
		context.put("truststore-type", "JKS");
		dataTransport();
	}
	
	public static void main(String args[]) {
		if(args.length<3) {
			System.out.println("Please Input 3 args: type[none,compress,ssl,comssl] batch(int) batchSize(int)");
			return;
		}
		String str = args[0];
		batchSize = Integer.parseInt(args[2]);
		batch = Integer.parseInt(args[1]);
		
		setUp();
		
		if("none".equals(str)) {
			testWithNone();
		}else if("compress".equals(str)) {
			testWithCompress();
		}else if("ssl".equals(str)) {
			testWithSSL();
		}else if("comssl".equals(str)) {
			testWithCompressAndSSL();
		}
		System.exit(0);
	}
	
	private static void dataTransport() {
		Configurables.configure(sink, context);
		Configurables.configure(channel, context);
		sink.setChannel(channel);
		Event event = EventBuilder
				.withBody("XXXXXX",Charsets.UTF_8);
		sink.start();
		long allBegin = System.currentTimeMillis();
		int a;
		for (a = 0; a < batch; a++) {
			Transaction transaction = channel.getTransaction();

			transaction.begin();
			int i;
			for (i = 0; i < batchSize; i++) {
				channel.put(event);
			}
			transaction.commit();
			transaction.close();
			try {
				sink.process();
			} catch (EventDeliveryException e) {
				e.printStackTrace();
			}
		}
		System.out.println("Sum Time = "
				+ (System.currentTimeMillis() - allBegin)+" ms");
		System.out.println("How many batch ? --> " + batch);
		System.out.println("How many events in one batch ? --> " + batchSize);
	}
	
}
 

在對avro進行測試的時候，分為四種情況測試

本人本機到遠端flume是通過vpn連上的，網路可能是一個原因

下面將四中情況的測試結果資料展示下（1000批資料，每批100條，總共104M資料）

1、不加密，不壓縮

148秒，效能=0.70M/S

2、不加密，壓縮

171秒，效能=0.61M/S

3、加密，不壓縮

168秒，效能=0.62M/S

4、加密，壓縮

141秒，效能=0.73M/S

下面再給出一個同一內網中的測試資料，效能明顯提升，看來我的VPN不給力啊：

1、不加密，不壓縮

java -jar avroTest-1.0.jar none 1000 100

Sum Time = 79968 ms
How many batch ? --> 1000
How many events in one batch ? --> 100
 

2、不加密，壓縮

Sum Time = 76740 ms
How many batch ? --> 1000
How many events in one batch ? --> 100

Sum Time = 88234 ms
How many batch ? --> 1000
How many events in one batch ? --> 100

4、加密，壓縮

Sum Time = 77825 ms
How many batch ? --> 1000
How many events in one batch ? --> 100
 

注意：

以上測試前提是事務容量100，因為測試的可變引數太多，希望大家自己嘗試調整引數進行測試。

以上測試結果資料，只限於本次測試，多次測試，不同網路環境的測試，對結果都是有影響的，在此只是為了提供一個測試的方法而已，望大家不吝指教！！！