Java在Windows下使用Hbase操作Hadoop示例
(1)解決jar包依賴問題,可以使用兩種方式
1、直接建立maven工程,編寫pom檔案,新增依賴的jar包宣告
2、直接下載對應的hbase的linux壓縮包,解壓後拷貝資料夾lib下的所有jar包
注意:
在執行過程中會報錯:java.lang.NoClassDefFoundError: org/hamcrest/SelfDescribing,因為缺少了hadoop的jar包),
可以從解壓hadoop的安裝包中的子目錄獲得(此處我獲取路徑):hadoop-2.9.1\share\hadoop\common\lib\hamcrest-core-1.3.jar
(2)執行報錯問題
1、沒有設定超時情況下,Java會報錯(60s後超時-太長):
java.io.IOException: Failed to get result within timeout, timeout=60000ms
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:206)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)
at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:327)
at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:302)
at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:167)
at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:162)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:797)
at org.apache.hadoop.hbase.MetaTableAccessor.fullScan(MetaTableAccessor.java:602)
at org.apache.hadoop.hbase.MetaTableAccessor.tableExists(MetaTableAccessor.java:366)
at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:406)
at com.donwait.hadoop.HBaseUtil.createTable(HBaseUtil.java:112)
僅僅報超時,都沒有報具體錯誤!為了解決超時問題,設定如下超時:
configuration.set("zookeeper.znode.parent", "/hbase");
configuration.set("hbase.client.retries.number", "3");
configuration.set("hbase.rpc.timeout", "2000");
configuration.set("hbase.client.operation.timeout", "3000");
configuration.set("hbase.client.scanner.timeout.period", "10000");
此時報錯:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=3, exceptions:
Thu Jun 21 09:48:03 CST 2018, RpcRetryingCaller{globalStartTime=1529545680623, pause=100, retries=3}, java.net.UnknownHostException: hadoop2
Thu Jun 21 09:48:03 CST 2018, RpcRetryingCaller{globalStartTime=1529545680623, pause=100, retries=3}, java.net.UnknownHostException: hadoop2
Thu Jun 21 09:48:03 CST 2018, RpcRetryingCaller{globalStartTime=1529545680623, pause=100, retries=3}, java.net.UnknownHostException: hadoop2
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:157)
at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:65)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.UnknownHostException: hadoop2
at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.<init>(AbstractRpcClient.java:315)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.createBlockingRpcChannel(AbstractRpcClient.java:267)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getClient(ConnectionManager.java:1639)
at org.apache.hadoop.hbase.client.ScannerCallable.prepare(ScannerCallable.java:163)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.prepare(ScannerCallableWithReplicas.java:376)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:134)
... 4 more
2、原因是因為hbase配置的Zookeeper列表為主機名,Hbase客戶端與Hbase的主節點通訊後拿到的就是對應配置
這裡hbase-site.xml配置如下:
<configuration>
<property>
<!-- 埠要和Hadoop的fs.defaultFS埠一致-->
<!-- hbase存放資料目錄 -->
<name>hbase.rootdir</name>
<value>hdfs://hadoop1:9000/hbase</value>
</property>
<property>
<!-- 是否分散式部署 -->
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<!-- list of zookooper -->
<name>hbase.zookeeper.quorum</name>
<value>hadoop1,hadoop2,hadoop3</value>
</property>
<property>
<!--zookooper配置、日誌等臨時檔案的儲存位置 -->
<name>hbase.zookeeper.property.dataDir</name>
<value>/home/hadoop/hbase/zookeeper</value>
</property>
<property>
<!--不設定將會使用一個預設埠,可能不是zookeeper使用的2181這些埠-->
<name>hbase.zookeeper.property.clientPort</name>
<value>2181</value>
</property>
</configuration>
可以看到Zookeeper列表為hadoop1、hadoop2和hadoop3(因為我配置了3臺叢集),客戶端獲取到列表後無法解析對應主機,所以windows上執行對應程式碼必須讓本機器能識別對應主機名,故修改本機的hosts檔案,新增上對應主機和ip的對映即可(路徑位置:C:\Windows\System32\drivers\etc\hosts):
具體配置:
# Copyright (c) 1993-2009 Microsoft Corp.
#
# This is a sample HOSTS file used by Microsoft TCP/IP for Windows.
#
# This file contains the mappings of IP addresses to host names. Each
# entry should be kept on an individual line. The IP address should
# be placed in the first column followed by the corresponding host name.
# The IP address and the host name should be separated by at least one
# space.
#
# Additionally, comments (such as these) may be inserted on individual
# lines or following the machine name denoted by a '#' symbol.
#
# For example:
#
# 102.54.94.97 rhino.acme.com # source server
# 38.25.63.10 x.acme.com # x client host
# localhost name resolution is handled within DNS itself.
# 127.0.0.1 localhost
# ::1 localhost
192.168.12.104 hadoop1
192.168.12.105 hadoop2
192.168.12.106 hadoop3
再次執行例項:public class Test {
@org.junit.Test
public void testHBase(){
HBaseUtil hbase = new HBaseUtil();
// 通話記錄,聯絡人
hbase.createTable("CallLog", "Record,Contact");
}
}
具體測試程式碼:
package com.donwait.hadoop;
import java.io.IOException;
public class Test {
/**
* 做多表儲存使用【相對於關係型資料庫中的表通話記錄表、聯絡人表】
* @throws IOException
*/
@org.junit.Test
public void testCreateCallLog() throws IOException{
HBaseUtil hbase = new HBaseUtil();
// 通話記錄,聯絡人
hbase.createTable("CallLog", "Record,Contact");
// 通話記錄 from 指點給 to, from的具體聯絡人詳情Contact(類似於RDBMS中的Record表和Contact表)
hbase.putDataH("CallLog", "row1", "Record", "from", "13556892563");
hbase.putDataH("CallLog", "row1", "Record", "to", "13256894751");
hbase.putDataH("CallLog", "row1", "Contact", "name", "lixx");
hbase.putDataH("CallLog", "row1", "Contact", "addr", "test address");
}
@org.junit.Test
public void testDropCallLog() throws IOException{
HBaseUtil hbase = new HBaseUtil();
// 通話記錄,聯絡人
hbase.dropTable("CallLog");
}
/**
* 做多表儲存使用
* @throws IOException
*/
@org.junit.Test
public void testGetFromCallLog() throws IllegalArgumentException, IOException{
HBaseUtil hbase = new HBaseUtil();
// Record表
System.out.println(hbase.getValueBySeriesH("CallLog", "row1", "Record", "from"));
System.out.println(hbase.getValueBySeriesH("CallLog", "row1", "Record", "to"));
// Contact表
System.out.println(hbase.getValueBySeriesH("CallLog", "row1", "Contact", "name"));
System.out.println(hbase.getValueBySeriesH("CallLog", "row1", "Contact", "addr"));
}
/**
* 做單表儲存使用
* @throws IOException
*/
@org.junit.Test
public void testCreatePerson() throws IOException{
HBaseUtil hbase = new HBaseUtil();
// 通話記錄,聯絡人
hbase.createTable("Person", "name,sex,age");
hbase.putDataH("Person", "id11", "name", null, "lixx");
hbase.putDataH("Person", "id11", "sex", null, "20");
hbase.putDataH("Person", "id11", "age", null, "1");
}
/**
* 做單表儲存使用
* @throws IOException
*/
@org.junit.Test
public void testGetFromPerson() throws IOException{
HBaseUtil hbase = new HBaseUtil();
// 通話記錄,聯絡人
hbase.createTable("Person", "name,sex,age");
System.out.println(hbase.getValueBySeriesH("Person", "id11", "name", null));
System.out.println(hbase.getValueBySeriesH("Person", "id11", "sex", null));
System.out.println(hbase.getValueBySeriesH("Person", "id11", "age", null));
}
}