Flume是Cloudera提供的一個高可用的，高可靠的，分散式的海量日誌採集、聚合和傳輸的系統，Flume支援在日誌系統中定製各類資料傳送方，用於收集資料；同時，Flume提供對資料進行簡單處理，並寫到各種資料接受方（可定製）的能力。 當前Flume有兩個版本Flume 0.9X版本的統稱Flume-og，Flume1.X版本的統稱Flume-ng。由於Flume-ng經過重大重構，與Flume-og有很大不同，使用時請注意區分。 日誌收集 Flume最早是Cloudera提供的日誌收集系統，目前是Apache下的一個孵化專案，Flume支援在日誌系統中定製各類資料傳送方，用於收集資料。

資料處理

Flume提供對資料進行簡單處理，並寫到各種資料接受方（可定製）的能力 Flume提供了從console（控制檯）、RPC（Thrift-RPC）、text（檔案）、tail（UNIX tail）、syslog（syslog日誌系統，支援TCP和UDP等2種模式），exec（命令執行）等資料來源上收集資料的能力。
省去長篇介紹........
下載安裝：
wget http://mirror.bit.edu.cn/apache/flume/1.7.0/apache-flume-1.7.0-bin.tar.gz
tar -zxvf apache-flume-1.7.0-bin.tar.gz

測試例子：
cd conf/
touch example.conf
vim  example.conf

# example.conf: A single-node Flume configuration

# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1


# Describe/configure the source
a1.sources.r1.type = netcat
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444


# Describe the sink
a1.sinks.k1.type = logger


# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100


# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1


啟動:
$ bin/flume-ng agent --conf conf --conf-file conf/example.conf --name a1 -Dflume.root.logger=INFO,console


 
yum install telnet


telnet localhost 44444


helloworld 測試(a->b傳送資料)


安裝avro （cmark）
wget https://cmake.org/files/v3.8/cmake-3.8.1.tar.gz

1: wget http://mirrors.hust.edu.cn/apache/avro/avro-1.7.7/c/avro-c-1.7.7.tar.gz
2:tar -zxvf avro-src-1.7.7.tar.gz
3:cd avro-c-1.7.7
4:mkdir build
5:cd build
6:cmake .. -DCMAKE_INSTALL_PREFIX=$PREFIX -DCMAKE_BUILD_TYPE=RelWithDebinfo
7:make
8:make install


關閉防火牆
service iptables stop

-------------測試 HTTP Source--------------------------
conf：httppost.conf


# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1


# Describe/configure the source
a1.sources.r1.type = org.apache.flume.source.http.HTTPSource
a1.sources.r1.port = 5140
a1.sources.r1.channels = c1


# Describe the sink
a1.sinks.k1.type = logger
a1.sinks.k1.channel = c1


# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100


啟動flume agent al
bin/flume-ng agent --conf conf --conf-file conf/httppost.conf --name a1 -Dflume.root.logger=INFO,console


傳送一個請求
curl -XPOST  -d '[{"headers" : {"timestamp" : "434324343","host" : "random_host.example.com"},"body" : "random_body"},{"headers" : {"namenode" : "namenode.example.com","datanode" : "random_datanode.example.com"},"body" : "really_random_body" }]' http://localhost:5140


結果：控制檯會打印出來   OK-----


-------------測試 avro Source--------------------------
conf：avrotest.conf


# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1


# Describe/configure the source
a1.sources.r1.type = avro
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444
a1.sources.r1.channels = c1


# Describe the sink
a1.sinks.k1.type = logger
a1.sinks.k1.channel = c1


# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100


啟動flume agent al
bin/flume-ng agent --conf conf --conf-file conf/avrotest.conf --name a1 -Dflume.root.logger=INFO,console




echo "hello world" > test.log


傳送檔案到flume
bin/flume-ng avro-client -c conf -H 192.168.254.129 -p 44444 -F /home/rui/log/test.log


-------------測試 Spooling Directory Source--------------------------
conf：directorytest.conf


# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1


# Describe/configure the source
a1.sources.r1.type = spooldir
a1.sources.r1.spoolDir = /home/rui/log/flumespool
a1.sources.r1.fileHeader = true
a1.sources.r1.channels = c1


# Describe the sink
a1.sinks.k1.type = logger
a1.sinks.k1.channel = c1


# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100


啟動flume agent al
bin/flume-ng agent --conf conf --conf-file conf/directorytest.conf --name a1 -Dflume.root.logger=INFO,console


測試:
cd /home/rui/log/flumespool
echo "hello world" > test.log

-------------測試 avro sink-----------a1服務--send-->a2服---------------
接收端：touch receive.conf
# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1


# Describe/configure the source
a1.sources.r1.type = avro
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44445
a1.sources.r1.channels = c1


# Describe the sink
a1.sinks.k1.type = logger
a1.sinks.k1.channel = c1


# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100


啟動flume agent al
bin/flume-ng agent --conf conf --conf-file conf/receive.conf --name a1 -Dflume.root.logger=INFO,console


傳送端：send.conf

# Name the components on this agent
a2.sources = r1
a2.sinks = k1
a2.channels = c1


# Describe/configure the source
#或者目錄監聽方式
a2.sources.r1.type = syslogtcp
a2.sources.r1.bind = localhost
a2.sources.r1.port = 44446
a2.sources.r1.channels = c1


# Describe the sink !!!!!!!!!!!!!!!!!!!!The point is here
a2.sinks.k1.type = avro  
a2.sinks.k1.channel = c1
a2.sinks.k1.hostname = 127.0.0.1
a2.sinks.k1.port = 44445


# Use a channel which buffers events in memory
a2.channels.c1.type = memory
a2.channels.c1.capacity = 1000
a2.channels.c1.transactionCapacity = 100


啟動flume agent al
bin/flume-ng agent --conf conf --conf-file conf/send.conf --name a2 -Dflume.root.logger=INFO,console


tcp方式測試：
echo "<37>hello via avro sink" | nc localhost 44446


如果沒有nc 就 yum install nc


44445埠控制檯列印現來：<37>hello via avro sink, 就成功了！