1. 程式人生 > >Java在Windows下使用Hbase操作Hadoop示例

Java在Windows下使用Hbase操作Hadoop示例

(1)解決jar包依賴問題,可以使用兩種方式

1、直接建立maven工程,編寫pom檔案,新增依賴的jar包宣告

2、直接下載對應的hbase的linux壓縮包,解壓後拷貝資料夾lib下的所有jar包


注意:

    在執行過程中會報錯:java.lang.NoClassDefFoundError: org/hamcrest/SelfDescribing,因為缺少了hadoop的jar包),

可以從解壓hadoop的安裝包中的子目錄獲得(此處我獲取路徑):hadoop-2.9.1\share\hadoop\common\lib\hamcrest-core-1.3.jar

(2)執行報錯問題

1、沒有設定超時情況下,Java會報錯(60s後超時-太長):

java.io.IOException: Failed to get result within timeout, timeout=60000ms
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:206)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)
at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:327)
at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:302)
at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:167)
at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:162)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:797)
at org.apache.hadoop.hbase.MetaTableAccessor.fullScan(MetaTableAccessor.java:602)
at org.apache.hadoop.hbase.MetaTableAccessor.tableExists(MetaTableAccessor.java:366)
at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:406)

at com.donwait.hadoop.HBaseUtil.createTable(HBaseUtil.java:112)

僅僅報超時,都沒有報具體錯誤!為了解決超時問題,設定如下超時:

configuration.set("zookeeper.znode.parent", "/hbase");
configuration.set("hbase.client.retries.number", "3"); 
configuration.set("hbase.rpc.timeout", "2000"); 
configuration.set("hbase.client.operation.timeout", "3000"); 

configuration.set("hbase.client.scanner.timeout.period", "10000"); 

此時報錯:

org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=3, exceptions:
Thu Jun 21 09:48:03 CST 2018, RpcRetryingCaller{globalStartTime=1529545680623, pause=100, retries=3}, java.net.UnknownHostException: hadoop2
Thu Jun 21 09:48:03 CST 2018, RpcRetryingCaller{globalStartTime=1529545680623, pause=100, retries=3}, java.net.UnknownHostException: hadoop2
Thu Jun 21 09:48:03 CST 2018, RpcRetryingCaller{globalStartTime=1529545680623, pause=100, retries=3}, java.net.UnknownHostException: hadoop2


at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:157)
at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:65)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.UnknownHostException: hadoop2
at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.<init>(AbstractRpcClient.java:315)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.createBlockingRpcChannel(AbstractRpcClient.java:267)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getClient(ConnectionManager.java:1639)
at org.apache.hadoop.hbase.client.ScannerCallable.prepare(ScannerCallable.java:163)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.prepare(ScannerCallableWithReplicas.java:376)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:134)
... 4 more

2、原因是因為hbase配置的Zookeeper列表為主機名,Hbase客戶端與Hbase的主節點通訊後拿到的就是對應配置


這裡hbase-site.xml配置如下:

<configuration>
	<property>
		<!-- 埠要和Hadoop的fs.defaultFS埠一致-->
		<!-- hbase存放資料目錄 -->
		<name>hbase.rootdir</name> 
		<value>hdfs://hadoop1:9000/hbase</value>
	</property> 
	<property>
		<!-- 是否分散式部署 -->
		<name>hbase.cluster.distributed</name> 
		<value>true</value> 
	</property> 
	<property> 
		<!-- list of  zookooper -->
		<name>hbase.zookeeper.quorum</name> 
		<value>hadoop1,hadoop2,hadoop3</value> 
	</property>     
	<property>
		<!--zookooper配置、日誌等臨時檔案的儲存位置 -->
		<name>hbase.zookeeper.property.dataDir</name> 
		<value>/home/hadoop/hbase/zookeeper</value>
	</property>
	<property>
		<!--不設定將會使用一個預設埠,可能不是zookeeper使用的2181這些埠-->
		<name>hbase.zookeeper.property.clientPort</name>
		<value>2181</value>
	</property>
</configuration>

可以看到Zookeeper列表為hadoop1、hadoop2和hadoop3(因為我配置了3臺叢集),客戶端獲取到列表後無法解析對應主機,所以windows上執行對應程式碼必須讓本機器能識別對應主機名,故修改本機的hosts檔案,新增上對應主機和ip的對映即可(路徑位置:C:\Windows\System32\drivers\etc\hosts):


具體配置:

# Copyright (c) 1993-2009 Microsoft Corp.
#
# This is a sample HOSTS file used by Microsoft TCP/IP for Windows.
#
# This file contains the mappings of IP addresses to host names. Each
# entry should be kept on an individual line. The IP address should
# be placed in the first column followed by the corresponding host name.
# The IP address and the host name should be separated by at least one
# space.
#
# Additionally, comments (such as these) may be inserted on individual
# lines or following the machine name denoted by a '#' symbol.
#
# For example:
#
#      102.54.94.97     rhino.acme.com          # source server
#       38.25.63.10     x.acme.com              # x client host

# localhost name resolution is handled within DNS itself.
#	127.0.0.1       localhost
#	::1             localhost
192.168.12.104	hadoop1
192.168.12.105	hadoop2
192.168.12.106	hadoop3
再次執行例項:
public class Test {
	@org.junit.Test
	public void testHBase(){
		HBaseUtil hbase = new HBaseUtil();
		// 通話記錄,聯絡人
		hbase.createTable("CallLog", "Record,Contact");
	}
}

具體測試程式碼:

package com.donwait.hadoop;

import java.io.IOException;

public class Test {
	/**
	 * 做多表儲存使用【相對於關係型資料庫中的表通話記錄表、聯絡人表】
	 * @throws IOException
	 */
	@org.junit.Test
	public void testCreateCallLog() throws IOException{
		HBaseUtil hbase = new HBaseUtil();
		// 通話記錄,聯絡人
		hbase.createTable("CallLog", "Record,Contact");
		
		// 通話記錄 from 指點給 to, from的具體聯絡人詳情Contact(類似於RDBMS中的Record表和Contact表)
		hbase.putDataH("CallLog", "row1", "Record", "from", "13556892563");
		hbase.putDataH("CallLog", "row1", "Record", "to", "13256894751");
		hbase.putDataH("CallLog", "row1", "Contact", "name", "lixx");
		hbase.putDataH("CallLog", "row1", "Contact", "addr", "test address");
	}
	
	@org.junit.Test
	public void testDropCallLog() throws IOException{
		HBaseUtil hbase = new HBaseUtil();
		// 通話記錄,聯絡人
		hbase.dropTable("CallLog");
	}
	
	/**
	 * 做多表儲存使用
	 * @throws IOException
	 */
	@org.junit.Test
	public void testGetFromCallLog() throws IllegalArgumentException, IOException{
		HBaseUtil hbase = new HBaseUtil();	
		// Record表
		System.out.println(hbase.getValueBySeriesH("CallLog", "row1", "Record", "from"));
		System.out.println(hbase.getValueBySeriesH("CallLog", "row1", "Record", "to"));
		
		// Contact表
		System.out.println(hbase.getValueBySeriesH("CallLog", "row1", "Contact", "name"));
		System.out.println(hbase.getValueBySeriesH("CallLog", "row1", "Contact", "addr"));
	}
	
	/**
	 * 做單表儲存使用
	 * @throws IOException
	 */
	@org.junit.Test
	public void testCreatePerson() throws IOException{
		HBaseUtil hbase = new HBaseUtil();
		// 通話記錄,聯絡人
		hbase.createTable("Person", "name,sex,age");
		
		hbase.putDataH("Person", "id11", "name", null, "lixx");
		hbase.putDataH("Person", "id11", "sex", null, "20");
		hbase.putDataH("Person", "id11", "age", null, "1");
	}
	
	/**
	 * 做單表儲存使用
	 * @throws IOException
	 */
	@org.junit.Test
	public void testGetFromPerson() throws IOException{
		HBaseUtil hbase = new HBaseUtil();
		// 通話記錄,聯絡人
		hbase.createTable("Person", "name,sex,age");
		
		System.out.println(hbase.getValueBySeriesH("Person", "id11", "name", null));
		System.out.println(hbase.getValueBySeriesH("Person", "id11", "sex", null));
		System.out.println(hbase.getValueBySeriesH("Person", "id11", "age", null));
	}
}