Hbase表兩種資料備份方法-匯入和匯出示例[未測試]
Hbase表兩種資料備份方法-匯入和匯出示例
本文將提供兩種備份方法 ——
1) 基於Hbase提供的類對hbase中某張表進行備份
2) 基於Hbase snapshot資料快速備份方法
場合：由於線上和測試環境是分離的，無法在測試環境訪問線上庫，所以需要將線上的hbase表匯出一部分到測試環境中的hbase表，這就是本文的由來。
一、基於hbase提供的類對hbase中某張表進行備份
本文使用hbase提供的類把hbase中某張表的資料匯出hdfs，之後再匯出到測試hbase表中。
首先介紹一下相關引數選項：
(1) 從hbase表匯出(# 預設不寫file://的時候就是匯出到hdfs上了 )
HBase資料匯出到HDFS或者本地檔案
hbase org.apache.hadoop.hbase.mapreduce.Export emp file:///Users/a6/Applications/experiment_data/hbase_data/bak
HBase資料匯出到本地檔案
hbase org.apache.hadoop.hbase.mapreduce.Export emp /hbase/emp_bak
(2) 匯入hbase表(# 預設不寫file://的時候就是匯出到hdfs上了 )
將hdfs上的資料匯入到備份目標表中
localhost:bin a6$ hbase org.apache.hadoop.hbase.mapreduce.Driver import emp_bak /hbase/emp_bak/*
將本地檔案上的資料匯入到備份目標表中
hbase org.apache.hadoop.hbase.mapreduce.Driver import emp_bak file:///Users/a6/Applications/experiment_data/hbase_data/bak/*
(3) 匯出時可以限制scanner.batch的大小
如果在hbase中的一個row出現大量的資料，那麼匯出時會報出ScannerTimeoutException的錯誤。這時候需要設定hbase.export.scaaner.batch 這個引數。這樣匯出時的錯誤就可以避免了。
hbase org.apache.hadoop.hbase.mapreduce.Export -Dhbase.export.scanner.batch=2000 emp file:///Users/a6/Applications/experiment_data/hbase_data/bak
(4)為了節省空間可以使用compress選項
hbase的資料匯出的時候，如果不適用compress的選項，資料量的大小可能相差5倍。因此使用compress的選項，備份資料的時候是可以節省不少空間的。
並且本人測試了compress選項的匯出速度，和無此選項時差別不大（幾乎無差別）：
hbase org.apache.hadoop.hbase.mapreduce.Export -Dhbase.export.scanner.batch=2000 -D mapred.output.compress=true emp file:///Users/a6/Applications/experiment_data/hbase_data/bak
通過新增compress選項，最終匯出檔案的大小由335位元組變成了325位元組，
File Output Format Counters File Output Format Counters
Bytes Written=335 Bytes Written=323
(5)匯出指定行鍵範圍和列族
在公司準備要更換資料中心，需要將hbase資料庫中的資料進行遷移。雖然進行hbase資料庫資料遷移時，使用其自帶的工具import和export是很方便的。只不過，在遷移大量資料時，可能需要執行很長的時間，甚至可能出錯。這時，是可以通過指定行鍵範圍和列族，來減少單次export工具的執行時間。可以看出，支援的選項有好幾個。假如，我們想匯出表test的資料，且只要列族Info，行鍵範圍在000到001之間，可以這樣寫：
這樣就可以了，且資料將會儲存在hdfs中。
通過指定列族和行鍵範圍，可以只匯出部分資料，避免export啟動的mapreduce任務執行時間過長。也就是可以分多次匯出資料。
./hbase org.apache.hadoop.hbase.mapreduce.Export -D hbase.mapreduce.scan.column.family=Info -D hbase.mapreduce.scan.row.start=000 -D hbase.mapreduce.scan.row.stop=001 test /test_datas
閒話少敘，例子就來：
查到了HBase自帶的export/import機制可以實現Backup Restore功能。而且可以實現增量備份。
原理都是用了MapReduce來實現的。
1、Export是以表為單位匯出資料的，若想完成整庫的備份需要執行n遍。
2、Export在shell中的呼叫方式類似如下格式：
./hbase org.apache.hadoop.hbase.mapreduce.Export 表名 備份路徑 （版本號） （起始時間戳） （結束時間戳）
括號內為可選項，例如
Usage: Export [-D <property=value>]* <tablename> <outputdir> [<versions> [<starttime> [<endtime>]] [^[regex pattern] or [Prefix] to filter]]
hbase org.apache.hadoop.hbase.mapreduce.Export emp /hbase/emp_bak 1 123456789
備份 emp 這張表到 /hbase/emp_bak 目錄下（最後一級目錄必須由Export自己建立），版本號為1，備份記錄從123456789這個時間戳開始到當前時間內所有的執行過put操作的記錄。
注意：為什麼是所有put操作記錄？因為在備份時是掃描所有表中所有時間戳大於等於123456789這個值的記錄並匯出。如果是delete操作，則表中這條記錄已經刪除，掃描時也無法獲取這條記錄資訊
當不指定時間戳時，備份的就是當前完整表中的資料。
1）、建立hbase表emp
localhost:bin a6$ pwd
/Users/a6/Applications/hbase-1.2.6/bin
localhost:bin a6$ hbase shell
create 'emp','personal data','professional data'
2）、插入資料並檢視資料
將第一行的值插入到emp表如下所示。
hbase(main):005:0> put 'emp','1','personal data:name','raju'
0 row(s) in 0.6600 seconds
hbase(main):006:0> put 'emp','1','personal data:city','hyderabad'
0 row(s) in 0.0410 seconds
hbase(main):007:0> put 'emp','1','professional data:designation','manager'
0 row(s) in 0.0240 seconds
hbase(main):007:0> put 'emp','1','professional data:salary','50000'
0 row(s) in 0.0240 seconds
插入完成整個表格，會得到下面的輸出。
hbase(main):002:0> scan 'emp'
ROW COLUMN+CELL
1 column=personal data:city, timestamp=1526269334560, value=hyderabad
1 column=personal data:name, timestamp=1526269326929, value=raju
1 column=professional data:designation, timestamp=1526269345044, value=manager
1 column=professional data:salary, timestamp=1526269352605, value=50000
1 row(s) in 0.2230 seconds
3）、將hbase表emp的資料匯出到hdfs的路徑/hbase/emp_bak上面
localhost:bin a6$ pwd
/Users/a6/Applications/hbase-1.2.6/bin
localhost:bin a6$ hbase org.apache.hadoop.hbase.mapreduce.Export emp /hbase/emp_bak
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/Users/a6/Applications/hbase-1.2.6/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2018-05-15 17:31:18,340 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-05-15 17:31:18,412 INFO [main] mapreduce.Export: versions=1, starttime=0, endtime=9223372036854775807, keepDeletedCells=false
2018-05-15 17:31:19,224 INFO [main] client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
2018-05-15 17:31:23,325 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x5ed731d0 connecting to ZooKeeper ensemble=localhost:2182
2018-05-15 17:31:23,332 INFO [main] zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2018-05-15 17:31:23,333 INFO [main] zookeeper.ZooKeeper: Client environment:host.name=localhost
2018-05-15 17:31:23,333 INFO [main] zookeeper.ZooKeeper: Client environment:java.version=1.8.0_131
2018-05-15 17:31:23,333 INFO [main] zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
2018-05-15 17:31:23,333 INFO [main] zookeeper.ZooKeeper: Client environment:java.home=/Library/Java/JavaVirtualMachines/jdk1.8.0_131.jdk/Contents/Home/jre
2018-05-15 17:31:23,333 INFO [main] zookeeper.ZooKeeper: Client environment:java.class.path=/Users/a6/Applications/hbase-1.2.6/bin/../conf:/Library/Java/JavaVirtualMachines/jdk1.8.0_131.jdk/Contents/Home/lib/tools.jar:/Users/a6/Applications/hbase-1.2.6/bin/..:/Users/a6/Applications/hbase-1.2.6/bin/../lib/activation-1.1.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/aopalliance-1.0.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/apacheds-i18n-2.0.0-M15.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/apacheds-kerberos-codec-2.0.0-M15.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/api-asn1-api-1.0.0-M20.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/api-util-1.0.0-M20.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/asm-3.1.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/avro-1.7.4.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/commons-beanutils-1.7.0.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/commons-beanutils-core-1.8.0.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/commons-cli-1.2.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/commons-codec-1.9.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/commons-collections-3.2.2.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/commons-compress-1.4.1.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/commons-configuration-1.6.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/commons-daemon-1.0.13.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/commons-digester-1.8.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/commons-el-1.0.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/commons-httpclient-3.1.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/commons-io-2.4.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/commons-lang-2.6.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/commons-logging-1.2.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/commons-math-2.2.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/commons-math3-3.1.1.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/commons-net-3.1.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/disruptor-3.3.0.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/findbugs-annotations-1.3.9-1.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/guava-12.0.1.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/guice-3.0.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/guice-servlet-3.0.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/hadoop-annotations-2.5.1.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/hadoop-auth-2.5.1.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/hadoop-client-2.5.1.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/hadoop-common-2.5.1.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/hadoop-hdfs-2.5.1.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/hadoop-mapreduce-client-app-2.5.1.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/hadoop-mapreduce-client-common-2.5.1.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/hadoop-mapreduce-client-core-2.5.1.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/hadoop-mapreduce-client-jobclient-2.5.1.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/hadoop-mapreduce-client-shuffle-2.5.1.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/hadoop-yarn-api-2.5.1.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/hadoop-yarn-client-2.5.1.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/hadoop-yarn-common-2.5.1.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/hadoop-yarn-server-common-2.5.1.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/hbase-annotations-1.2.6-tests.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/hbase-annotations-1.2.6.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/hbase-client-1.2.6.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/hbase-common-1.2.6-tests.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/hbase-common-1.2.6.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/hbase-examples-1.2.6.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/hbase-external-blockcache-1.2.6.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/hbase-hadoop-compat-1.2.6.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/hbase-hadoop2-compat-1.2.6.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/hbase-it-1.2.6-tests.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/hbase-it-1.2.6.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/hbase-prefix-tree-1.2.6.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/hbase-procedure-1.2.6.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/hbase-protocol-1.2.6.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/hbase-resource-bundle-1.2.6.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/hbase-rest-1.2.6.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/hbase-server-1.2.6-tests.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/hbase-server-1.2.6.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/hbase-shell-1.2.6.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/hbase-thrift-1.2.6.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/htrace-core-3.1.0-incubating.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/httpclient-4.2.5.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/httpcore-4.4.1.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/jackson-core-asl-1.9.13.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/jackson-jaxrs-1.9.13.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/jackson-mapper-asl-1.9.13.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/jackson-xc-1.9.13.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/jamon-runtime-2.4.1.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/jasper-compiler-5.5.23.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/jasper-runtime-5.5.23.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/java-xmlbuilder-0.4.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/javax.inject-1.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/jaxb-api-2.2.2.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/jaxb-impl-2.2.3-1.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/jcodings-1.0.8.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/jersey-client-1.9.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/jersey-core-1.9.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/jersey-guice-1.9.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/jersey-json-1.9.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/jersey-server-1.9.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/jets3t-0.9.0.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/jettison-1.3.3.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/jetty-6.1.26.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/jetty-sslengine-6.1.26.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/jetty-util-6.1.26.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/joni-2.1.2.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/jruby-complete-1.6.8.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/jsch-0.1.42.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/jsp-2.1-6.1.14.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/jsp-api-2.1-6.1.14.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/junit-4.12.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/leveldbjni-all-1.8.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/libthrift-0.9.3.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/log4j-1.2.17.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/metrics-core-2.2.0.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/netty-all-4.0.23.Final.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/paranamer-2.3.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/protobuf-java-2.5.0.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/servlet-api-2.5-6.1.14.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/servlet-api-2.5.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/slf4j-api-1.7.7.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/slf4j-log4j12-1.7.5.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/snappy-java-1.0.4.1.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/spymemcached-2.11.6.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/xmlenc-0.52.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/xz-1.0.jar:/Users/a6/Applications/hbase-1.2.6/bin/../lib/zookeeper-3.4.6.jar:/Users/a6/Applications/hadoop-2.6.5/etc/hadoop:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/jsr305-1.3.9.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/activation-1.1.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/curator-recipes-2.6.0.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/commons-configuration-1.6.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/xz-1.0.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/junit-4.11.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/commons-httpclient-3.1.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/stax-api-1.0-2.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/httpclient-4.2.5.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/mockito-all-1.8.5.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/commons-logging-1.1.3.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/slf4j-api-1.7.5.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/jersey-json-1.9.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/avro-1.7.4.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/log4j-1.2.17.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/commons-cli-1.2.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/commons-digester-1.8.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/servlet-api-2.5.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/hadoop-annotations-2.6.5.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/hadoop-auth-2.6.5.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/xmlenc-0.52.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/jetty-util-6.1.26.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/guava-11.0.2.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/commons-compress-1.4.1.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/htrace-core-3.0.4.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/commons-io-2.4.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/jersey-core-1.9.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/jsp-api-2.1.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/commons-codec-1.4.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/netty-3.6.2.Final.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/curator-framework-2.6.0.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/jetty-6.1.26.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/jersey-server-1.9.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/curator-client-2.6.0.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/paranamer-2.3.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/zookeeper-3.4.6.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/commons-collections-3.2.2.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/jettison-1.1.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/asm-3.2.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/hamcrest-core-1.3.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/commons-net-3.1.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/gson-2.2.4.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/jets3t-0.9.0.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/commons-lang-2.6.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/jsch-0.1.42.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/commons-el-1.0.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/commons-math3-3.1.1.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/httpcore-4.2.5.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/hadoop-common-2.6.5-tests.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/hadoop-nfs-2.6.5.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/hadoop-common-2.6.5.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/hdfs:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/hdfs/lib/guava-11.0.2.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/hdfs/lib/htrace-core-3.0.4.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/hdfs/lib/commons-io-2.4.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/hdfs/lib/asm-3.2.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/hdfs/lib/commons-el-1.0.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/hdfs/hadoop-hdfs-2.6.5-tests.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/hdfs/hadoop-hdfs-2.6.5.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/hdfs/hadoop-hdfs-nfs-2.6.5.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/yarn/lib/jsr305-1.3.9.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/yarn/lib/activation-1.1.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/yarn/lib/aopalliance-1.0.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/yarn/lib/xz-1.0.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/yarn/lib/commons-httpclient-3.1.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/yarn/lib/jline-0.9.94.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/yarn/lib/jersey-json-1.9.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/yarn/lib/log4j-1.2.17.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/yarn/lib/commons-cli-1.2.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/yarn/lib/servlet-api-2.5.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/yarn/lib/guava-11.0.2.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/yarn/lib/commons-io-2.4.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/yarn/lib/jersey-core-1.9.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/yarn/lib/commons-codec-1.4.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/yarn/lib/jetty-6.1.26.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/yarn/lib/jersey-server-1.9.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/yarn/lib/guice-3.0.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/yarn/lib/jersey-client-1.9.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/yarn/lib/commons-collections-3.2.2.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/yarn/lib/jettison-1.1.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/yarn/lib/asm-3.2.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/yarn/lib/commons-lang-2.6.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/yarn/lib/javax.inject-1.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/yarn/hadoop-yarn-common-2.6.5.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.6.5.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.6.5.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.6.5.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/yarn/hadoop-yarn-server-common-2.6.5.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/yarn/hadoop-yarn-client-2.6.5.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/yarn/hadoop-yarn-registry-2.6.5.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.6.5.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.6.5.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.6.5.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/yarn/hadoop-yarn-server-tests-2.6.5.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/yarn/hadoop-yarn-api-2.6.5.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/mapreduce/lib/xz-1.0.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/mapreduce/lib/junit-4.11.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/mapreduce/lib/hadoop-annotations-2.6.5.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/mapreduce/lib/guice-3.0.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/mapreduce/lib/asm-3.2.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/mapreduce/lib/javax.inject-1.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.6.5.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.6.5.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.6.5.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.6.5.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.6.5-tests.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.6.5.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.6.5.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.5.jar:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.6.5.jar:/Users/a6/Applications/hadoop-2.6.5/contrib/capacity-scheduler/*.jar
2018-05-15 17:31:23,336 INFO [main] zookeeper.ZooKeeper: Client environment:java.library.path=/Users/a6/Applications/hadoop-2.6.5/lib/native
2018-05-15 17:31:23,336 INFO [main] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/var/folders/bm/dccwv2v97y75hdshqnh1bbpr0000gn/T/
2018-05-15 17:31:23,336 INFO [main] zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
2018-05-15 17:31:23,336 INFO [main] zookeeper.ZooKeeper: Client environment:os.name=Mac OS X
2018-05-15 17:31:23,336 INFO [main] zookeeper.ZooKeeper: Client environment:os.arch=x86_64
2018-05-15 17:31:23,336 INFO [main] zookeeper.ZooKeeper: Client environment:os.version=10.13.2
2018-05-15 17:31:23,337 INFO [main] zookeeper.ZooKeeper: Client environment:user.name=a6
2018-05-15 17:31:23,337 INFO [main] zookeeper.ZooKeeper: Client environment:user.home=/Users/a6
2018-05-15 17:31:23,337 INFO [main] zookeeper.ZooKeeper: Client environment:user.dir=/Users/a6/Applications/hbase-1.2.6/bin
2018-05-15 17:31:23,338 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2182 sessionTimeout=90000 watcher=hconnection-0x5ed731d00x0, quorum=localhost:2182, baseZNode=/hbase
2018-05-15 17:31:23,360 INFO [main-SendThread(localhost:2182)] zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2182. Will not attempt to authenticate using SASL (unknown error)
2018-05-15 17:31:23,361 INFO [main-SendThread(localhost:2182)] zookeeper.ClientCnxn: Socket connection established to localhost/127.0.0.1:2182, initiating session
2018-05-15 17:31:23,371 INFO [main-SendThread(localhost:2182)] zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2182, sessionid = 0x163615ea3e6000c, negotiated timeout = 40000
2018-05-15 17:31:23,455 INFO [main] util.RegionSizeCalculator: Calculating region sizes for table "emp".
2018-05-15 17:31:23,834 INFO [main] client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
2018-05-15 17:31:23,834 INFO [main] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x163615ea3e6000c
2018-05-15 17:31:23,837 INFO [main] zookeeper.ZooKeeper: Session: 0x163615ea3e6000c closed
2018-05-15 17:31:23,837 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
2018-05-15 17:31:23,933 INFO [main] mapreduce.JobSubmitter: number of splits:1
2018-05-15 17:31:23,955 INFO [main] Configuration.deprecation: io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
2018-05-15 17:31:24,115 INFO [main] mapreduce.JobSubmitter: Submitting tokens for job: job_1526346976211_0003
2018-05-15 17:31:24,513 INFO [main] impl.YarnClientImpl: Submitted application application_1526346976211_0003
2018-05-15 17:31:24,561 INFO [main] mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1526346976211_0003/
2018-05-15 17:31:24,562 INFO [main] mapreduce.Job: Running job: job_1526346976211_0003
2018-05-15 17:31:36,842 INFO [main] mapreduce.Job: Job job_1526346976211_0003 running in uber mode : false
2018-05-15 17:31:36,844 INFO [main] mapreduce.Job: map 0% reduce 0%
2018-05-15 17:31:43,965 INFO [main] mapreduce.Job: map 100% reduce 0%
2018-05-15 17:31:44,980 INFO [main] mapreduce.Job: Job job_1526346976211_0003 completed successfully
2018-05-15 17:31:45,120 INFO [main] mapreduce.Job: Counters: 43
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=139577
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=64
HDFS: Number of bytes written=323
HDFS: Number of read operations=4
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=4842
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=4842
Total vcore-seconds taken by all map tasks=4842
Total megabyte-seconds taken by all map tasks=4958208
Map-Reduce Framework
Map input records=1
Map output records=1
Input split bytes=64
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=73
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0
Total committed heap usage (bytes)=111149056
HBase Counters
BYTES_IN_REMOTE_RESULTS=0
BYTES_IN_RESULTS=210
MILLIS_BETWEEN_NEXTS=517
NOT_SERVING_REGION_EXCEPTION=0
NUM_SCANNER_RESTARTS=0
NUM_SCAN_RESULTS_STALE=0
REGIONS_SCANNED=1
REMOTE_RPC_CALLS=0
REMOTE_RPC_RETRIES=0
ROWS_FILTERED=0
ROWS_SCANNED=1
RPC_CALLS=3
RPC_RETRIES=0
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=323
檢視生成的目錄並檢視匯出到hdfs上的二進位制資料
localhost:bin a6$ hadoop dfs -ls /hbase/emp_bak
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
18/05/15 17:34:29 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 2 items
-rw-r--r-- 1 a6 supergroup 0 2018-05-15 17:31 /hbase/emp_bak/_SUCCESS
-rw-r--r-- 1 a6 supergroup 323 2018-05-15 17:31 /hbase/emp_bak/part-m-00000
localhost:bin a6$ hadoop dfs -cat /hbase/emp_bak/*
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
18/05/15 17:34:37 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
SEQ1org.apache.hadoop.hbase.io.ImmutableBytesWritable%org.apache.hadoop.hbase.client.ResultI~
�F�;�H��[$���1�
,
personal datacity ����,(2 hyderabad
'
personal dataname ����,(2raju
5
1professional data
designation ����,(2manager
.
1professional datasalary ����,(250000 )
將hbase資料備份到本地檔案
localhost:bin a6$ hbase org.apache.hadoop.hbase.mapreduce.Export emp file:///Users/a6/Applications/experiment_data/hbase_data/bak
4）、建立備份到的目標hbase表
create 'emp_bak','personal data','professional data'
5)、將hdfs上的資料匯入到備份目標表中
localhost:bin a6$ hbase org.apache.hadoop.hbase.mapreduce.Driver import emp_bak /hbase/emp_bak/*
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/Users/a6/Applications/hbase-1.2.6/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2018-05-15 17:37:07,154 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-05-15 17:37:08,045 INFO [main] client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
2018-05-15 17:37:09,852 INFO [main] input.FileInputFormat: Total input paths to process : 1
2018-05-15 17:37:09,907 INFO [main] mapreduce.JobSubmitter: number of splits:1
2018-05-15 17:37:10,026 INFO [main] mapreduce.JobSubmitter: Submitting tokens for job: job_1526346976211_0005
2018-05-15 17:37:10,384 INFO [main] impl.YarnClientImpl: Submitted application application_1526346976211_0005
2018-05-15 17:37:10,413 INFO [main] mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1526346976211_0005/
2018-05-15 17:37:10,413 INFO [main] mapreduce.Job: Running job: job_1526346976211_0005
2018-05-15 17:37:18,621 INFO [main] mapreduce.Job: Job job_1526346976211_0005 running in uber mode : false
2018-05-15 17:37:18,622 INFO [main] mapreduce.Job: map 0% reduce 0%
2018-05-15 17:37:25,705 INFO [main] mapreduce.Job: map 100% reduce 0%
2018-05-15 17:37:25,716 INFO [main] mapreduce.Job: Job job_1526346976211_0005 completed successfully
2018-05-15 17:37:25,832 INFO [main] mapreduce.Job: Counters: 30
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=139121
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=436
HDFS: Number of bytes written=0
HDFS: Number of read operations=3
HDFS: Number of large read operations=0
HDFS: Number of write operations=0
Job Counters
Launched map tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=4804
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=4804
Total vcore-seconds taken by all map tasks=4804
Total megabyte-seconds taken by all map tasks=4919296
Map-Reduce Framework
Map input records=1
Map output records=1
Input split bytes=113
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=86
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0
Total committed heap usage (bytes)=112197632
File Input Format Counters
Bytes Read=323
File Output Format Counters
Bytes Written=0
2018-05-15 17:37:25,842 INFO [main] mapreduce.Job: Running job: job_1526346976211_0005
2018-05-15 17:37:25,848 INFO [main] mapreduce.Job: Job job_1526346976211_0005 running in uber mode : false
2018-05-15 17:37:25,849 INFO [main] mapreduce.Job: map 100% reduce 0%
2018-05-15 17:37:25,855 INFO [main] mapreduce.Job: Job job_1526346976211_0005 completed successfully
2018-05-15 17:37:25,862 INFO [main] mapreduce.Job: Counters: 30
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=139121
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=436
HDFS: Number of bytes written=0
HDFS: Number of read operations=3
HDFS: Number of large read operations=0
HDFS: Number of write operations=0
Job Counters
Launched map tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=4804
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=4804
Total vcore-seconds taken by all map tasks=4804
Total megabyte-seconds taken by all map tasks=4919296
Map-Reduce Framework
Map input records=1
Map output records=1
Input split bytes=113
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=86
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0
Total committed heap usage (bytes)=112197632
File Input Format Counters
Bytes Read=323
File Output Format Counters
Bytes Written=0
這樣基本就完成了hbase表中的資料我們可以轉化為mapreduce任務程序開始匯出匯入。當然也可以這麼備份的。
6)、最後我們仔細看一下hbase匯出和匯入的關鍵命令引數
localhost:bin a6$ hbase org.apache.hadoop.hbase.mapreduce.Export
ERROR: Wrong number of arguments: 0
Usage: Export [-D <property=value>]* <tablename> <outputdir> [<versions> [<starttime> [<endtime>]] [^[regex pattern] or [Prefix] to filter]]
Note: -D properties will be applied to the conf used.
For example:
-D mapreduce.output.fileoutputformat.compress=true
-D mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.GzipCodec
-D mapreduce.output.fileoutputformat.compress.type=BLOCK
Additionally, the following SCAN properties can be specified
to control/limit what is exported..
-D hbase.mapreduce.scan.column.family=<familyName>
-D hbase.mapreduce.include.deleted.rows=true
-D hbase.mapreduce.scan.row.start=<ROWSTART>
-D hbase.mapreduce.scan.row.stop=<ROWSTOP>
For performance consider the following properties:
-Dhbase.client.scanner.caching=100
-Dmapreduce.map.speculative=false
-Dmapreduce.reduce.speculative=false
For tables with very wide rows consider setting the batch size as below:
-Dhbase.export.scanner.batch=10
localhost:bin a6$ hbase org.apache.hadoop.hbase.mapreduce.Driver import
ERROR: Wrong number of arguments: 0
Usage: Import [options] <tablename> <inputdir>
By default Import will load data directly into HBase. To instead generate
HFiles of data to prepare for a bulk data load, pass the option:
-Dimport.bulk.output=/path/for/output
To apply a generic org.apache.hadoop.hbase.filter.Filter to the input, use
-Dimport.filter.class=<name of filter class>
-Dimport.filter.args=<comma separated list of args for filter
NOTE: The filter will be applied BEFORE doing key renames via the HBASE_IMPORTER_RENAME_CFS property. Futher, filters will only use the Filter#filterRowKey(byte[] buffer, int offset, int length) method to identify whether the current row needs to be ignored completely for processing and Filter#filterKeyValue(KeyValue) method to determine if the KeyValue should be added; Filter.ReturnCode#INCLUDE and #INCLUDE_AND_NEXT_COL will be considered as including the KeyValue.
To import data exported from HBase 0.94, use
-Dhbase.import.version=0.94
For performance consider the following options:
-Dmapreduce.map.speculative=false
-Dmapreduce.reduce.speculative=false
-Dimport.wal.durability=<Used while writing data to hbase. Allowed values are the supported durability values like SKIP_WAL/ASYNC_WAL/SYNC_WAL/...>
二、基於Hbase snapshot資料快速備份方法
1.Snapshot備份的優點是什麼？
HBase以往資料的備份基於distcp或者copyTable等工具，這些備份機制或多或少對當前的online資料讀寫存在一定的影響，Snapshot提供了一種快速的資料備份方式，無需進行資料copy。
參見下圖
2.HBase資料的備份的方式有幾種？Snapshot包括線上和離線的，他們之間有什麼區別？
Snapshot包括線上和離線的
(1)離線方式是disabletable，由HBase Master遍歷HDFS中的table metadata和hfiles，建立對他們的引用。
(2)線上方式是enabletable，由Master指示region server進行snapshot操作，在此過程中，master和regionserver之間類似兩階段commit的snapshot操作。
HFile是不可變的，只能append和delete， region的split和compact，都不會對snapshot引用的檔案做刪除(除非刪除snapshot檔案)，這些檔案會歸檔到archive目錄下，進而需要重新調整snapshot檔案中相關hfile的引用位置關係。
基於snapshot檔案，可以做clone一個新表，restore，export到另外一個叢集中操作；其中clone生成的新表只是增加元資料，相關的資料檔案還是複用snapshot指定的資料檔案
參見clone新表操作示意圖：
3.snashot的shell的命令都由哪些？如何刪除、檢視快照？如何匯出到另外一個叢集？
snashot相關的操作命令如下：
1)建立快照(檢視快照->檢視快照snapshot命令相關引數->建立快照—>檢視快照)
hbase(main):002:0> list_snapshots
SNAPSHOT TABLE + CREATION TIME
0 row(s) in 0.0290 seconds
=> []
hbase(main):003:0> snapshot
ERROR: wrong number of arguments (0 for 2)
Here is some help for this command:
Take a snapshot of specified table. Examples:
hbase> snapshot 'sourceTable', 'snapshotName'
hbase> snapshot 'namespace:sourceTable', 'snapshotName', {SKIP_FLUSH => true}
hbase(main):004:0> snapshot 'emp','emp_snapshot'
0 row(s) in 0.3730 seconds
hbase(main):005:0> list_snapshots
SNAPSHOT TABLE + CREATION TIME
emp_snapshot emp (Wed May 16 09:44:53 +0800 2018)
1 row(s) in 0.0190 seconds
=> ["emp_snapshot"]
2)刪除並檢視快照
hbase(main):006:0> delete_snapshot 'emp_snapshot'
0 row(s) in 0.0390 seconds
hbase(main):007:0> list_snapshots
SNAPSHOT TABLE + CREATION TIME
0 row(s) in 0.0040 seconds
=> []
3)基於快照，clone一個新表
hbase(main):011:0> clone_snapshot 'emp_snapshot','new_emp'
0 row(s) in 0.5290 seconds
hbase(main):013:0> scan 'new_emp'
ROW COLUMN+CELL
1 column=personal data:city, timestamp=1526269334560, value=hyderabad
1 column=personal data:name, timestamp=1526269326929, value=raju
1 column=professional data:designation, timestamp=1526269345044, value=manager
1 column=professional data:salary, timestamp=1526269352605, value=50000
1 row(s) in 0.1050 seconds
hbase(main):014:0> desc 'new_emp'
Table new_emp is ENABLED
new_emp
COLUMN FAMILIES DESCRIPTION
{NAME => 'personal data', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE
=> '0'}
{NAME => 'professional data', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_S
COPE => '0'}
2 row(s) in 0.0370 seconds
4)基於快照恢復表(原hbase表emp需要刪除)
hbase(main):027:0> list
TABLE
new_emp
t1
test
3 row(s) in 0.0130 seconds
=> ["new_emp", "t1", "test"]
hbase(main):028:0> list_snapshots
SNAPSHOT TABLE + CREATION TIME
emp_snapshot emp (Wed May 16 09:45:25 +0800 2018)
1 row(s) in 0.0130 seconds
=> ["emp_snapshot"]
hbase(main):029:0> restore_snapshot 'emp_snapshot'
0 row(s) in 0.3700 seconds
hbase(main):030:0> list
TABLE
emp
new_emp
t1
test
4 row(s) in 0.0240 seconds
=> ["emp", "new_emp", "t1", "test"]
5)基於快照將資料匯出到另外一個叢集中的本地檔案中
利用mapreduce job將emp_snapshot這個snapshot 匯出到本地目錄/Users/a6/Applications/experiment_data/hbase_data中的bak_emp_snapshot(不存在)
localhost:bin a6$ hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot 'emp_snapshot' -copy-to file:///Users/a6/Applications/experiment_data/hbase_data/bak_emp_snapshot -mappers 16
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/Users/a6/Applications/hbase-1.2.6/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/a6/Applications/hadoop-2.6.5/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2018-05-16 10:21:47,310 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-05-16 10:21:47,633 INFO [main] snapshot.ExportSnapshot: Copy Snapshot Manifest
2018-05-16 10:21:47,922 INFO [main] client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
2018-05-16 10:21:50,233 INFO [main] snapshot.ExportSnapshot: Loading Snapshot 'emp_snapshot' hfile list
2018-05-16 10:21:50,547 INFO [main] mapreduce.JobSubmitter: number of splits:2
2018-05-16 10:21:50,732 INFO [main] mapreduce.JobSubmitter: Submitting tokens for job: job_1526434993990_0001
2018-05-16 10:21:51,182 INFO [main] impl.YarnClientImpl: Submitted application application_1526434993990_0001
2018-05-16 10:21:51,268 INFO [main] mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1526434993990_0001/
2018-05-16 10:21:51,269 INFO [main] mapreduce.Job: Running job: job_1526434993990_0001
2018-05-16 10:22:02,425 INFO [main] mapreduce.Job: Job job_1526434993990_0001 running in uber mode : false
2018-05-16 10:22:02,427 INFO [main] mapreduce.Job: map 0% reduce 0%
2018-05-16 10:22:09,722 INFO [main] mapreduce.Job: map 50% reduce 0%
2018-05-16 10:22:10,731 INFO [main] mapreduce.Job: map 100% reduce 0%
2018-05-16 10:22:10,740 INFO [main] mapreduce.Job: Job job_1526434993990_0001 completed successfully
2018-05-16 10:22:10,848 INFO [main] mapreduce.Job: Counters: 37
File System Counters
FILE: Number of bytes read=9985
FILE: Number of bytes written=291407
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=408
HDFS: Number of bytes written=0
HDFS: Number of read operations=2
HDFS: Number of large read operations=0
HDFS: Number of write operations=0
Job Counters
Launched map tasks=2
Other local map tasks=2
Total time spent by all maps in occupied slots (ms)=9683
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=9683
Total vcore-seconds taken by all map tasks=9683
Total megabyte-seconds taken by all map tasks=9915392
Map-Reduce Framework
Map input records=2
Map output records=0
Input split bytes=408
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=155
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0
Total committed heap usage (bytes)=212860928
org.apache.hadoop.hbase.snapshot.ExportSnapshot$Counter
BYTES_COPIED=9985
BYTES_EXPECTED=9985
BYTES_SKIPPED=0
COPY_FAILED=0
FILES_COPIED=2
FILES_SKIPPED=0
MISSING_FILES=0
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=0
2018-05-16 10:22:10,851 INFO [main] snapshot.ExportSnapshot: Finalize the Snapshot Export
2018-05-16 10:22:10,