楔子

學習瞭解HBase，使用系統環境是CentOS6.9，Hadoop等版本是CDH5.3.6

配置了Hadoop、HBase等環境變數，yarn可以直接使用。以下基於這些配置

1.1 檢視HBase執行MapReduce所依賴的Jar包

[[email protected] hbase0986]$ bin/hbase mapredcp
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/module/hbase0986/lib/slf4j-log4j12-1.7.5.jar!
 
/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/hadoop250/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
 

2018-10-28 14:32:07,428 WARN  [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
/opt/module/hbase0986/lib/hbase-client-0.98.6-cdh5.3.6.jar:/opt/module/hbase0986/lib/hbase-server-0.98.6-cdh5.3.6.jar:/opt/module/hbase0986/lib/htrace-core-2.04.jar:/opt/module/hbase0986/lib/netty-3.6.6.Final.jar:/opt/module/hbase0986/lib/hbase-common-0.98.6-cdh5.3.6.jar:/opt/module/hbase0986/lib/high-scale-lib-1.1.1.jar:/opt/module/hbase0986/lib/zookeeper-3.4.5-cdh5.3.6.jar:/opt/module/hbase0986/lib/guava-12.0.1.jar:/opt/module/hbase0986/lib/protobuf-java-2.5.0.jar:/opt/module/hbase0986/lib/hbase-protocol-0.98.6-cdh5.3.6.jar:/opt/module/hbase0986/lib/hbase-hadoop-compat-0.98.6-cdh5.3.6.jar

 

1.2 環境匯入

[[email protected] hbase0986]$ export HADOOP_CLASSPATH=`${HBASE_HOME}/bin/hbase mapredcp`
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/module/hbase0986/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/hadoop250/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2018-10-28 14:34:47,798 WARN  [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[[email protected] hbase0986]$

HADOOP_HOME和HBASE_HOME已經配置為了系統變數，如果沒有還需使用export臨時匯入

1.3 執行官方MapReduce任務

執行官方任務統計，person表示資料行數

[[email protected] hbase0986]$  yarn jar lib/hbase-server-0.98.6-cdh5.3.6.jar rowcounter person

1.4 使用MapRecude將資料匯入到HBASE

1.4.1 準備資料

準備資料並上傳到Hadoop，資料格式是使用tsv格式(\t為分割符號) [[email protected] hbase0986]$ more …/data/fruit.txt

001	Apple	Red
002	Pig	blue
003	Pear	yelllow

1.4.2 建立HBASE表

create 'fruits','info'

1.4.3 HDFS建立資料夾並上傳檔案

1.4.4 執行MapReduce到HBase表中

[[email protected] hbase0986]$ yarn jar lib/hbase-server-0.98.6-cdh5.3.6.jar importtsv -Dimporttsv.columns=HBASE_ROW_KEY,info:name,info:color fruits hdfs://hadoop:9000/input/

檢視結果

hbase(main):001:0> scan 'fruits'
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/module/hbase0986/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/hadoop250/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2018-10-28 14:57:08,685 WARN  [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
ROW                        COLUMN+CELL                                                                
 001                       column=info:color, timestamp=1540707102480, value=Red                      
 001                       column=info:name, timestamp=1540707102480, value=Apple                     
 002                       column=info:color, timestamp=1540707102480, value=blue                     
 002                       column=info:name, timestamp=1540707102480, value=Pig                       
 003                       column=info:color, timestamp=1540707102480, value=yelllow                  
 003                       column=info:name, timestamp=1540707102480, value=Pear                      
3 row(s) in 0.8320 seconds