flume將資料匯入hbase
阿新 • • 發佈:2019-01-06
1 將hbase的lib目錄下jar拷貝到flume的lib目錄下;
2 在hbase中建立儲存資料的表
hbase(main):
002
:
0
> create
'test_idoall_org'
,
'uid'
,
'name'
3 建立flume配置檔案 vi.conf
a1.sources = r1
a1.sinks = k1
a1.channels = c1
# Describe/configure the source
a1.sources.r1.type = exec
a1.sources.r1.command = tail -F /home/hadoop/data.txt
a1.sources.r1.port =
5555
a1.sources.r1.host =
master
a1.sources.r1.channels = c1
# Describe the sink
a1.sinks.k1.type = logger
a1.sinks.k1.type = hbase
a1.sinks.k1.table = test_idoall_org
a1.sinks.k1.columnFamily = name
a1.sinks.k1.column = idoall
a1.sinks.k1.serializer = org.apache.flume.sink.hbase.RegexHbaseEventSerializer
a1.sinks.k1.channel = memoryChannel
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity =
1000
a1.channels.c1.transactionCapacity =
100
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
4 執行flume
flume-ng agent -c . -f conf/vi.conf -n a1 -Dflume.root.logger=INFO,console
5 建立匯入檔案
/home/hadoop/data.txt