flume+kafka+hdfs詳解
阿新 • • 發佈：2017-11-03
utf-8 conf prop nts command format ext sink 1.4
flume架構圖
單節點flume配置
flume-1.4.0 啟動flume
bin/flume-ng agent --conf ./conf -f conf/flume-conf.properties -Dflume.root.logger=DEBUG,console -n agent
-n表示配置文件中agent的名字
[java] view plain copy- agent.sources = r1
- agent.sinks = s1
- agent.channels = c1
- agent.sources.r1.channels = c1
- agent.sinks.s1.channel = c1
- #Describe/configure the sources
- agent.sources.r1.type = exec
- agent.sources.r1.command = tail -F /home/flume/loginfo
- #Use a channel which buffers events in memory
- agent.channels.c1.type = memory
- agent.channels.c1.capacity = 1000 #Event
- agent.channels.c1.transactionCapacity = 100
- agent.sinks.s1.type = logger
flume-1.4.0 + kafka-0.7.2+hdfs flume配置
[xml] view plain copy
- agent.sources = r1
- agent.sinks = s_kafka s_hdfs
- agent.channels = c_kafka c_hdfs
- agent.sources.r1.channels = c_kafka c_hdfs
- agent.sources.r1.type = exec
- #下面這個腳本tail某個日誌
- agent.sources.r1.command = tail -F /home/flume/loginfo
- agent.channels.c_kafka.type = memory
- agent.channels.c_hdfs.type = memory
- agent.sinks.s_kafka.type = com.sink.FirstkafkaSink
- agent.sinks.s_kafka.channel = c_kafka
- #kafka需要連接zk，寫入broker數據
- agent.sinks.s_kafka.zkconnect = localhost:2181
- agent.sinks.s_kafka.topic = test
- agent.sinks.s_kafka.serializer.class = kafka.serializer.StringEncoder
- agent.sinks.s_kafka.metadata.broker.list = localhost:9092 #配置文件server.properties
- agent.sinks.s_kafka.custom.encoding = UTF-8
- agent.sinks.s_hdfs.type = hdfs
- agent.sinks.s_hdfs.channel = c_hdfs
- #默認端口8020
- agent.sinks.s_hdfs.hdfs.path = hdfs://localhost:9000/root/source
- agent.sinks.s_hdfs.hdfs.filePrefix = events-
- agent.sinks.s_hdfs.hdfs.fileType = DataStream
- agent.sinks.s_hdfs.hdfs.writeFormat = Text
- agent.sinks.s_hdfs.hdfs.rollCount = 30 #達到某一數值記錄生成文件
- agent.sinks.s_hdfs.hdfs.rollSize = 0
- agent.sinks.s_hdfs.hdfs.rollInterval = 0
- agent.sinks.s_hdfs.hdfs.useLocalTimeStamp = true
- agent.sinks.s_hdfs.hdfs.idleTimeout = 51
- agent.sinks.s_hdfs.hdfs.threadsPoolSize = 2
Flume內置channel,source,sink匯總
http://www.iteblog.com/archives/948
本文出自 “點滴積累” 博客，請務必保留此出處http://tianxingzhe.blog.51cto.com/3390077/1700049
flume+kafka+hdfs詳解