好久沒寫部落格了。最近在研究storm、flume和kafka。今天給大夥寫下我測試flume failover以及load balance的場景以及一些結論；

測試環境包含5個配置檔案，也就是5個agent。

一個主的配置檔案，也就是我們配置failover以及load balance關係的配置檔案（flume-sink.properties），這個檔案在下面的場景

會變動，所以這裡就不列舉出來了，會在具體的場景中寫明；

其他4個配置檔案類似：

#Name the compents on this agent
a2.sources = r1
a2.sinks = k1
a2.channels = c1

#Describe/configure the source
a2.sources.r1.type = avro
a2.sources.r1.channels = c1
a2.sources.r1.bind = 192.168.220.159
a2.sources.r1.port = 44411

#Describe the sink
a2.sinks.k1.type = logger
a2.sinks.k1.channel = c1

#Use a channel which buffers events in memory
a2.channels.c1.type = memory
a2.channels.c1.capacity = 1000
a2.channels.c1.transactionCapacity = 100
 

另外三個只需修改agent名稱即可。

場景1：---想法（flume failover以及load balance同時應用，兼顧負載和容錯，使環境更加有保障）

配置檔案（flume-sink.properties）：

#Name the compents on this agent
a1.sources = r1
a1.sinks = k1 k2 k3
a1.channels = c1

#Describe the sinkgroups
a1.sinkgroups = g1 g2
a1.sinkgroups.g1.sinks = k1 k2
a1.sinkgroups.g1.processor.type = failover
a1.sinkgroups.g1.processor.priority.k1 = 10
a1.sinkgroups.g1.processor.priority.k2 = 5
a1.sinkgroups.g1.processor.maxpenalty = 10000

a1.sinkgroups.g2.sinks = k1 k3
a1.sinkgroups.g2.processor.type = load_balance
a1.sinkgroups.g2.processor.backoff = true
a1.sinkgroups.g2.processor.selector = round_robin

#Describe/config the source
a1.sources.r1.type = syslogtcp
a1.sources.r1.port = 5140
a1.sources.r1.host = localhost
a1.sources.r1.channels = c1

#Describe the sink
a1.sinks.k1.type = avro
a1.sinks.k1.channel = c1
a1.sinks.k1.hostname = 192.168.220.159
a1.sinks.k1.port = 44411

a1.sinks.k2.type = avro
a1.sinks.k2.channel = c1
a1.sinks.k2.hostname = 192.168.220.159
a1.sinks.k2.port = 44422

a1.sinks.k4.type = avro
a1.sinks.k4.channel = c1
 

測試結果：

發現K1，K2，K3都可以接收到資料，但是這裡K2的優先級別是最低的，原本不應該獲得資料的；除非K1掛掉了。

這個測試之前最開始的時候我測試了，但是忘記配置k3的聲明瞭，結果給我報了好多錯誤，讓我以為sinkgroups的sink不能共享，

然後我就跑去社群發郵件問，是不是sinkgroups的sink不能共享，結果有個哥們跟我說是。現在知道我和那哥們都錯了。哈哈

社群郵件地址：http://mail-archives.apache.org/mod_mbox/flume-user/201408.mbox/browser

其實上面我還少配置了一個東西，大夥可以看上面的配置檔案，K1做了容錯，但是K3沒有。所以這裡需要新增以下K3的容錯節點K4。

這點是@晨色星空跟我說的，我一下明白了，哈；

#Describe the sinkgroups
a1.sinkgroups = g1 g2 g3
a1.sinkgroups.g1.sinks = k1 k2
a1.sinkgroups.g1.processor.type = failover
a1.sinkgroups.g1.processor.priority.k1 = 10
a1.sinkgroups.g1.processor.priority.k2 = 5
a1.sinkgroups.g1.processor.maxpenalty = 10000

a1.sinkgroups.g2.sinks = k1 k3
a1.sinkgroups.g2.processor.type = load_balance
a1.sinkgroups.g2.processor.backoff = true
a1.sinkgroups.g2.processor.selector = round_robin

a1.sinkgroups.g3.sinks = k3 k4
a1.sinkgroups.g3.processor.type = failover
a1.sinkgroups.g3.processor.priority.k3 = 10
a1.sinkgroups.g3.processor.priority.k4 = 5
a1.sinkgroups.g3.processor.maxpenalty = 10000

那我們把g3和k4加上，然後傳送資料看怎麼樣；

好把。竟然4個都接收到了。

接下來我們把K1斷掉，看什麼結果；

結果這裡K2,K3,K4都接收到了資料，真的有點詭異；K4是做K3的容錯的不應該接收到資料才對。K2接收到資料倒是在理；

我們再把K1啟起來。然後我們來斷掉K3看看。

結果和上面的情況類似；我只能說測試結果有點奇怪；哈哈

場景2：---想法（failover和load balance分開不同sink）

配置檔案（flume-sink.properties）：

#Name the compents on this agent
a1.sources = r1
a1.sinks = k1 k2 k3 k4
a1.channels = c1

#Describe the sinkgroups
a1.sinkgroups = g1 g2
a1.sinkgroups.g1.sinks = k1 k2
a1.sinkgroups.g1.processor.type = failover
a1.sinkgroups.g1.processor.priority.k1 = 10
a1.sinkgroups.g1.processor.priority.k2 = 5
a1.sinkgroups.g1.processor.maxpenalty = 10000

a1.sinkgroups.g2.sinks = k3 k4
a1.sinkgroups.g2.processor.type = load_balance
a1.sinkgroups.g2.processor.backoff = true
a1.sinkgroups.g2.processor.selector = round_robin

#Describe/config the source
a1.sources.r1.type = syslogtcp
a1.sources.r1.port = 5140
a1.sources.r1.host = localhost
a1.sources.r1.channels = c1

#Describe the sink
a1.sinks.k1.type = avro
a1.sinks.k1.channel = c1
a1.sinks.k1.hostname = 192.168.220.159
a1.sinks.k1.port = 44411

a1.sinks.k2.type = avro
a1.sinks.k2.channel = c1
a1.sinks.k2.hostname = 192.168.220.159
a1.sinks.k2.port = 44422

a1.sinks.k3.type = avro
a1.sinks.k3.channel = c1
a1.sinks.k3.hostname = 192.168.220.159
a1.sinks.k3.port = 44433

a1.sinks.k4.type = avro
a1.sinks.k4.channel = c1
a1.sinks.k4.hostname = 192.168.220.159
a1.sinks.k4.port = 44444
#Use a channel which butters events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

測試結果：

這裡可以看到K1，K3，K4是可以接收到資料的，這個還是正常的。這裡相當於K2做的容錯，K1，K3，K4做的負載均衡分擔下發的資料。

然後接下來我們把K1斷開，再測試看看是什麼情況。

這裡主agent會報錯，原因是因為我們把K1斷開了。

我們再來看接收情況

可以看到，這裡K2，K3，K4做了負載。K1掛掉之後，K2和K3，K4一起分擔資料。

測試結果正常。K1，K4可以接收到資料，K3斷開了之後，K1，K4做了負載

場景3：---想法（單獨使用failover配置，多個同級別高階的sink，一個低階的sink）

配置檔案（flume-sink.properties）：

#Name the compents on this agent
a1.sources = r1
a1.sinks = k1 k2 k3 k4
a1.channels = c1

#Describe the sinkgroups
a1.sinkgroups = g1 g2
a1.sinkgroups.g1.sinks = k1 k2 k3 k4
a1.sinkgroups.g1.processor.type = failover
a1.sinkgroups.g1.processor.priority.k1 = 10
a1.sinkgroups.g1.processor.priority.k3 = 10
a1.sinkgroups.g1.processor.priority.k4 = 10
a1.sinkgroups.g1.processor.priority.k2 = 5
a1.sinkgroups.g1.processor.maxpenalty = 10000

#Describe/config the source
a1.sources.r1.type = syslogtcp
a1.sources.r1.port = 5140
a1.sources.r1.host = localhost
a1.sources.r1.channels = c1

#Describe the sink
a1.sinks.k1.type = avro
a1.sinks.k1.channel = c1
a1.sinks.k1.hostname = 192.168.220.159
a1.sinks.k1.port = 44411

a1.sinks.k2.type = avro
a1.sinks.k2.channel = c1
a1.sinks.k2.hostname = 192.168.220.159
a1.sinks.k2.port = 44422

a1.sinks.k3.type = avro
a1.sinks.k3.channel = c1
a1.sinks.k3.hostname = 192.168.220.159
a1.sinks.k3.port = 44433

a1.sinks.k4.type = avro
a1.sinks.k4.channel = c1
a1.sinks.k4.hostname = 192.168.220.159
a1.sinks.k4.port = 44444
#Use a channel which butters events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

我們來發資料看看。

這裡我們可以看到K3接收到了資料，但是我不管發多少資料都被K3接收了。原本的想法是想多個同等級的sink是否會做負載，結果測試失敗，不行；

那麼接下來我們把K3斷掉看怎麼樣；

K3斷掉了結果K2接收到了資料，K2的級別是最低的；如果這樣的話我們來把K2斷掉看看。

結果K1，K4都接收不到資料了。我們來看下日誌

org.apache.flume.EventDeliveryException: All sinks failed to process, nothing left to failover to
        at org.apache.flume.sink.FailoverSinkProcessor.process(FailoverSinkProcessor.java:191)
        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
        at java.lang.Thread.run(Thread.java:745)
2014-07-08 12:12:26,574 (SinkRunner-PollingRunner-FailoverSinkProcessor) [INFO - org.apache.flume.sink.AbstractRpcSink.createConnection(AbstractRpcSink.java:206)] Rpc sink k2: Building RpcClient with hostname: 192.168.220.159, port: 44422
2014-07-08 12:12:26,575 (SinkRunner-PollingRunner-FailoverSinkProcessor) [INFO - org.apache.flume.sink.AvroSink.initializeRpcClient(AvroSink.java:126)] Attempting to create Avro Rpc client.
2014-07-08 12:12:26,577 (SinkRunner-PollingRunner-FailoverSinkProcessor) [WARN - org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:620)] Using default maxIOWorkers
2014-07-08 12:12:26,595 (SinkRunner-PollingRunner-FailoverSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:160)] Unable to deliver event. Exception follows.

這裡寫了，All sinks failed to process。從上面的測試可以表明，在配置failover的時候，sink的級別不能配置相同；如果配置多個相同級別的sink，只有一個生效；

我發了一封郵件到社群，郵件地址是：http://mail-archives.apache.org/mod_mbox/flume-user/201408.mbox/browser，大家可以去看看。

Each sink needs to have a different priority. If multiple sinks have 
same priority, only one of them will be used.

這是一個叫cloudera的外國哥們回答的。意思是說我配置多個sinks的時候，sink等級必須不同；

測試4：--想法：測試下上面那哥們說的，哈哈

配置檔案（flume-sink.properties）

#Name the compents on this agent
a1.sources = r1
a1.sinks = k1 k2 k3 k4
a1.channels = c1

#Describe the sinkgroups
a1.sinkgroups = g1 g2
a1.sinkgroups.g1.sinks = k1 k2 k3 k4
a1.sinkgroups.g1.processor.type = failover
a1.sinkgroups.g1.processor.priority.k1 = 10
a1.sinkgroups.g1.processor.priority.k3 = 9
a1.sinkgroups.g1.processor.priority.k4 = 9
a1.sinkgroups.g1.processor.priority.k2 = 5
a1.sinkgroups.g1.processor.maxpenalty = 10000

#Describe/config the source
a1.sources.r1.type = syslogtcp
a1.sources.r1.port = 5140
a1.sources.r1.host = localhost
a1.sources.r1.channels = c1

#Describe the sink
a1.sinks.k1.type = avro
a1.sinks.k1.channel = c1
a1.sinks.k1.hostname = 192.168.220.159
a1.sinks.k1.port = 44411

a1.sinks.k2.type = avro
a1.sinks.k2.channel = c1
a1.sinks.k2.hostname = 192.168.220.159
a1.sinks.k2.port = 44422

a1.sinks.k3.type = avro
a1.sinks.k3.channel = c1
a1.sinks.k3.hostname = 192.168.220.159
a1.sinks.k3.port = 44433

a1.sinks.k4.type = avro
a1.sinks.k4.channel = c1
a1.sinks.k4.hostname = 192.168.220.159
a1.sinks.k4.port = 44444
#Use a channel which butters events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

第四個場景就留給大家去測試吧。測試結果正常。

訊息由高級別的sink接收。依次關閉，其次等級接收；

轉載請寫明源地址：

http://blog.csdn.net/weijonathan/article/details/38557959

http://www.51studyit.com/html/notes/20140814/1055.html