1. 程式人生 > >flume將資料匯入到hbase中

flume將資料匯入到hbase中

安裝flume:

[[email protected] ~]$ tar -zxvf flume-ng-1.6.0-cdh5.5.2.tar.gz
修改 flume-env.sh 配置檔案,主要是JAVA_HOME變數設定
[[email protected] apache-flume-1.6.0-cdh5.5.2-bin]$ cp conf/flume-env.sh.template conf/flume-env.sh
[[email protected] apache-flume-1.6.0-cdh5.5.2-bin]$ vi conf/flume-env.sh
新增:
export JAVA_HOME=/usr/jdk1.7.0_25

(這裡新增的是你的Java安裝目錄,我這裡安裝的是jdk1.7.0_25,注意Java版本不要過低)

驗證是否安裝成功:
[[email protected] apache-flume-1.6.0-cdh5.5.2-bin]$ bin/flume-ng version

Flume 1.6.0-cdh5.5.2
Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git
Revision: f65a722cd2d1e7ceeda972570a5a6ee01c3a0a3d
Compiled by jenkins on Mon Jan 25 16:38:11 PST 2016
From source with checksum 028a2c6b035a03df1dfa91a3feda3424

[[email protected] ~]$ cd hbase-1.0.0-cdh5.5.2/lib/
然後將以下檔案複製到flume中:
[[email protected] lib]$ cp protobuf-java-2.5.0.jar /home/hadoop/apache-flume-1.6.0-cdh5.5.2-bin/lib/
[[email protected] lib]$ cp hbase-protocol-1.0.0-cdh5.5.2.jar /home/hadoop/apache-flume-1.6.0-cdh5.5.2-bin/lib/
[[email protected]
lib]$ cp hbase-client-1.0.0-cdh5.5.2.jar /home/hadoop/apache-flume-1.6.0-cdh5.5.2-bin/lib/
[[email protected] lib]$ cp hbase-common-1.0.0-cdh5.5.2.jar /home/hadoop/apache-flume-1.6.0-cdh5.5.2-bin/lib/
[hado[email protected] lib]$ cp hbase-server-1.0.0-cdh5.5.2.jar /home/hadoop/apache-flume-1.6.0-cdh5.5.2-bin/lib/
[[email protected] lib]$ cp hbase-hadoop2-compat-1.0.0-cdh5.5.2.jar /home/hadoop/apache-flume-1.6.0-cdh5.5.2-bin/lib/
[[email protected] lib]$ cp hbase-hadoop-compat-1.0.0-cdh5.5.2.jar /home/hadoop/apache-flume-1.6.0-cdh5.5.2-bin/lib/
[[email protected] lib]$ cp htrace-core-3.2.0-incubating.jar /home/hadoop/apache-flume-1.6.0-cdh5.5.2-bin/lib/
(也可以直接把hbase-1.0.0-cdh5.5.2/lib下的jar包全部複製到flume的lib目錄下)

確保test_idoall_org表在hbase中已經存在

hbase(main):002:0> create 'test_idoall_org','uid','name'
0 row(s) in 0.6730 seconds

=> Hbase::Table - test_idoall_org
hbase(main):003:0> put 'test_idoall_org','10086','name:idoall','idoallvalue'
0 row(s) in 0.0960 seconds

[[email protected] apache-flume-1.6.0-cdh5.5.2-bin]$ vi conf/hbase_simple.conf
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = exec
a1.sources.r1.command = tail -F /home/hadoop/data.txt
a1.sources.r1.port = 44444
a1.sources.r1.host = 192.168.8.71
a1.sources.r1.channels = c1

# Describe the sink
a1.sinks.k1.type = logger
a1.sinks.k1.type = hbase
a1.sinks.k1.table = test_idoall_org
a1.sinks.k1.columnFamily = name
a1.sinks.k1.serializer = org.apache.flume.sink.hbase.RegexHbaseEventSerializer
a1.sinks.k1.channel = memoryChannel

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

啟動flume agent:
[[email protected] apache-flume-1.6.0-cdh5.5.2-bin]$ bin/flume-ng agent -c . -f conf/hbase_simple.conf -n a1 -Dflume.root.logger=INFO,console
。。。。。。。。。(前面省略,太多了)
:/home/hadoop/hive-1.1.0-cdh5.5.2/lib/logredactor-1.0.3.jar:/home/hadoop/hive-1.1.0-cdh5.5.2/lib/commons-dbcp-1.4.jar:/home/hadoop/hive-1.1.0-cdh5.5.2/lib/jcommander-1.32.jar
12/12/13 00:21:08 INFO zookeeper.ZooKeeper: Client environment:java.library.path=:/home/hadoop/hadoop-2.6.0-cdh5.5.2/lib/native
12/12/13 00:21:08 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
12/12/13 00:21:08 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
12/12/13 00:21:08 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
12/12/13 00:21:08 INFO zookeeper.ZooKeeper: Client environment:os.arch=i386
12/12/13 00:21:08 INFO zookeeper.ZooKeeper: Client environment:os.version=2.6.18-194.el5
12/12/13 00:21:08 INFO zookeeper.ZooKeeper: Client environment:user.name=hadoop
12/12/13 00:21:08 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/hadoop
12/12/13 00:21:08 INFO zookeeper.ZooKeeper: Client environment:user.dir=/home/hadoop/apache-flume-1.6.0-cdh5.5.2-bin
12/12/13 00:21:08 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=hconnection-0x1d630160x0, quorum=localhost:2181, baseZNode=/hbase
12/12/13 00:21:09 INFO zookeeper.ClientCnxn: Opening socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
12/12/13 00:21:09 INFO zookeeper.ClientCnxn: Socket connection established, initiating session, client: /127.0.0.1:47668, server: 127.0.0.1/127.0.0.1:2181
12/12/13 00:21:09 INFO zookeeper.ClientCnxn: Session establishment complete on server 127.0.0.1/127.0.0.1:2181, sessionid = 0x3b8fd59e830000, negotiated timeout = 90000
12/12/13 00:21:10 INFO Configuration.deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available
12/12/13 00:21:10 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SINK, name: k1: Successfully registered new MBean.
12/12/13 00:21:10 INFO instrumentation.MonitoredCounterGroup: Component type: SINK, name: k1 started
(啟動成功。。。)

產生資料:
[[email protected] ~]$ touch data.txt
[[email protected] ~]$ echo "hello idoall.org from flume" >> data.txt

這時登入到hbase中,可以發現新資料已經插入:
hbase(main):005:0> scan 'test_idoall_org'
ROW                                                          COLUMN+CELL                                                                                                                                                                     
 10086                                                       column=name:idoall, timestamp=1355329032253, value=idoallvalue                                                                                                                  
 1355329550628-0EZpfeEvxG-0                                  column=name:payload, timestamp=1355329383396, value=hello idoall.org from flume                                                                                                 
2 row(s) in 0.0140 seconds