Flume的一些元件（如Spooling Directory Source、File Channel）能夠保證agent掛掉後不丟失資料。

1、負載均衡

1）Load balancing Sink Processor

source裡的event流經channel，進入sink組，在sink組內部根據負載演算法（round_robin、random）選擇sink，後續可以選擇不同機器上的agent實現負載均衡。

配置如下：

# Name the components on this agent
a1.sources = r1
a1.sinks = k1 k2
a1.channels
 
 = c1

# Describe/configure the source
a1.sources.r1.type = exec
a1.sources.r1.channels=c1
a1.sources.r1.command=tail -F /home/flume/xx.log 

#define sinkgroups
a1.sinkgroups=g1
a1.sinkgroups.g1.sinks=k1 k2
a1.sinkgroups.g1.processor.type=load_balance
a1.sinkgroups.g1.processor.backoff=true
a1.sinkgroups
 
.g1.processor.selector=round_robin

#define the sink 1
a1.sinks.k1.type=avro
a1.sinks.k1.hostname=192.168.1.112
a1.sinks.k1.port=9876  

#define the sink 2
a1.sinks.k2.type=avro
a1.sinks.k2.hostname=192.168.1.113
a1.sinks.k2.port=9876


# Use a channel which buffers events in memory
a1.channels.c1.type
 
 = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
a1.sinks.k2.channel=c1

2）Load balancing Log4J Appender
不同的agent處理同一個client產生的資料。

log4j.rootLogger=INFO,flume
log4j.appender.flume = org.apache.flume.clients.log4jappender.LoadBalancingLog4jAppender
log4j.appender.flume.Hosts = 192.168.1.111:41414 192.168.1.111:41414

2、故障轉移

Failover Sink Processor

配置一組sink，這組sink組成一個Failover Sink Processor，當有一個sink處理失敗，Flume將這個sink放到一個地方，等待冷卻時間，可以正常處理event時再拿回來。

event通過通過一個channel流向一個sink組，在sink組內部根據優先順序選擇具體的sink，一個失敗後再轉向另一個sink，流程圖如下：

# Name the components on this agent
a1.sources = r1
a1.sinks = k1 k2
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = exec
a1.sources.r1.channels=c1
a1.sources.r1.command=tail -F /home/flume/xx.log

#define sinkgroups
a1.sinkgroups=g1
a1.sinkgroups.g1.sinks=k1 k2
a1.sinkgroups.g1.processor.type=failover
a1.sinkgroups.g1.processor.priority.k1=10
a1.sinkgroups.g1.processor.priority.k2=5
a1.sinkgroups.g1.processor.maxpenalty=10000

#define the sink 1
a1.sinks.k1.type=avro
a1.sinks.k1.hostname=192.168.1.112
a1.sinks.k1.port=9876

#define the sink 2
a1.sinks.k2.type=avro
a1.sinks.k2.hostname=192.168.1.113
a1.sinks.k2.port=9876


# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
a1.sinks.k2.channel=c1

備註：

在Sink中的兩個資料為優先順序設定預設為5，數字越大越優先

a1.sinkgroups.g1.processor.priority.k1=10
a1.sinkgroups.g1.processor.priority.k2=5