利用hbase api在本地訪問並操作伺服器的hbase資料庫
阿新 • • 發佈：2018-11-04
最近因為實驗室專案需要，開始研究了hbase。然後想一次性往叢集伺服器上寫入大量的資料，並存到hbase中，考慮到在hbase shell下只能單個數據put，這樣對於批量插入資料的要求完全不合適。於是就研究起hbase的java api，然後去大量填充資料到hbase以測試查詢的效能。於是，故事開始了。
首先在eclipse下寫好了，如下的Java程式碼：
<span style="font-size:12px;">import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.hbase.HBaseConfiguration; import org.apache.hadoop.hbase.HColumnDescriptor; import org.apache.hadoop.hbase.HTableDescriptor; import org.apache.hadoop.hbase.TableName; import org.apache.hadoop.hbase.client.Admin; import org.apache.hadoop.hbase.client.Connection; import org.apache.hadoop.hbase.client.ConnectionFactory; public class testCreate { public static void main(String[] args){ Configuration conf = HBaseConfiguration.create(); conf.set("hbase.rootdir", "hdfs://xxx-a.test.com:8020/apps/hbase/data");//使用eclipse時必須新增這個，否則無法定位 conf.set("hbase.zookeeper.quorum", "xxx-a1.test.com,xxx-a2.test.com,xxx-a3.test.com"); conf.set("hbase.zookeeper.property.clientPort", "2181"); conf.set("zookeeper.znode.parent","/hbase-unsecure");//這行程式碼也是後來加上的 try { Connection conn = ConnectionFactory.createConnection(conf); System.out.println("StartConnect..."); Admin hAdmin = conn.getAdmin(); HTableDescriptor hTableDesc = new HTableDescriptor(TableName.valueOf("Customer2")); hTableDesc.addFamily(new HColumnDescriptor("name")); hTableDesc.addFamily(new HColumnDescriptor("contactinfo")); hTableDesc.addFamily(new HColumnDescriptor("address")); hAdmin.createTable(hTableDesc); System.out.println("Table created Successfully..."); } catch (Exception e) { e.printStackTrace(); } } }</span>
<span style="font-size:12px;"><span style="white-space: pre;"> </span>程式碼寫好，一堆錯誤。是因為缺少依賴的jar包。以下是新增的jar截圖。</span>
這樣jar包都是程式執行報錯一點點加進去的，分享給大家可以少走冤枉路。（debug也是一種樂趣，雖然有時候想死的心都有，但是堅持找，肯定會解決的
）
此時還得配置電腦的hosts，在C:\Windows\System32\drivers\etc\目錄，有一個hosts檔案，在其中加上xxx-a.test.com對應的實際ip地址，這樣程式執行才會訪問到正確的地方，格式類似於這樣
xxx-a.test.com X.X.X.X。基於這樣，程式是可以執行的，Java的執行日誌會有如下顯示：
表示連線hbase資料庫成功，建立表成功。皆大歡喜。
下面說一下幾個需要注意的地方：
1、conf.set("hbase.rootdir", "hdfs://xxx-a.test.com:8020/apps/hbase/data");//使用eclipse時必須新增這個，否則無法定位這句話必須要加上，通過這句配置，才能準確的定位hbase實際儲存的hdfs位置，所以這裡面的<span style="font-family: Arial, Helvetica, sans-serif;">xxx-a.test.com換成詳細的ip地址也是可以的。</span> 2、conf.set("hbase.zookeeper.quorum", "xxx-a1.test.com,xxx-a2.test.com,xxx-a3.test.com");<span style="font-family: Helvetica, Tahoma, Arial, sans-serif; font-size: 14px; line-height: 25.2px;">Zookeeper叢集的地址列表，用逗號分割。例如：</span><span style="font-family: Helvetica, Tahoma, Arial, sans-serif; font-size: 14px; line-height: 25.2px;">"xxx-a1.test.com,xxx-a2.test.com,xxx-a3.test.com</span><span style="font-size: 14px; line-height: 25.2px; font-family: Helvetica, Tahoma, Arial, sans-serif;">".預設是localhost,是給偽分散式用的。要修改才能在完全分散式的情況下使用。如果在hbase-env.sh設定了HBASE_MANAGES_ZK，這些ZooKeeper節點就會和HBase一起啟動。</span><span style="font-family: Helvetica, Tahoma, Arial, sans-serif; font-size: 14px; line-height: 25.2px;"> </span>3、conf.set("hbase.zookeeper.property.clientPort", "2181");這句話是設定zookeeper監控連線的客戶端的埠號是2181，這是預設的配置。 4、conf.set("zookeeper.znode.parent","/hbase-unsecure");//這行程式碼也是後來加上的<span style="font-family: Helvetica, Tahoma, Arial, sans-serif; line-height: 25.2px;">ZooKeeper中的HBase的根ZNode。所有的HBase的ZooKeeper會用這個目錄配置相對路徑。預設情況下，所有的HBase的ZooKeeper檔案路徑是用相對路徑，所以他們會都去這個目錄下面。</span><span style="font-family:Helvetica, Tahoma, Arial, sans-serif;"><span style="line-height: 25.2px;">5.沒加上面那句程式碼之前我的報錯資訊一直是：</span></span><span style="font-family:Helvetica, Tahoma, Arial, sans-serif;"><span style="line-height: 25.2px;">org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=35, exceptions: Wed Aug 10 21:29:19 CST 2016, RpcRetryingCaller{globalStartTime=1470835740311, pause=100, retries=35}, org.apache.hadoop.hbase.MasterNotRunningException: org.apache.hadoop.hbase.MasterNotRunningException: Can't get connection to ZooKeeper: KeeperErrorCode = ConnectionLoss for /hbase </span></span><span style="font-family:Helvetica, Tahoma, Arial, sans-serif;"><span style="line-height: 25.2px;">at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:157) <span style="white-space:pre"> </span>at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4212) <span style="white-space:pre"> </span>at org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsyncV2(HBaseAdmin.java:748) <span style="white-space:pre"> </span>at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:669) <span style="white-space:pre"> </span>at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:602) <span style="white-space:pre"> </span>at hbase.testCreate.main(testCreate.java:24) Caused by: org.apache.hadoop.hbase.MasterNotRunningException: org.apache.hadoop.hbase.MasterNotRunningException: Can't get connection to ZooKeeper: KeeperErrorCode = ConnectionLoss for /hbase <span style="white-space:pre"> </span>at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1535) <span style="white-space:pre"> </span>at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1555) <span style="white-space:pre"> </span>at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1706) <span style="white-space:pre"> </span>at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38) <span style="white-space:pre"> </span>at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:134) <span style="white-space:pre"> </span>... 5 more Caused by: org.apache.hadoop.hbase.MasterNotRunningException: Can't get connection to ZooKeeper: KeeperErrorCode = ConnectionLoss for /hbase <span style="white-space:pre"> </span>at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(ConnectionManager.java:907) <span style="white-space:pre"> </span>at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.access$400(ConnectionManager.java:546) <span style="white-space:pre"> </span>at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(ConnectionManager.java:1485) <span style="white-space:pre"> </span>at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1526) <span style="white-space:pre"> </span>... 9 more Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase <span style="white-space:pre"> </span>at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) <span style="white-space:pre"> </span>at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) <span style="white-space:pre"> </span>at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045) <span style="white-space:pre"> </span>at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:221) <span style="white-space:pre"> </span>at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:541) <span style="white-space:pre"> </span>at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(ConnectionManager.java:896) <span style="white-space:pre"> </span>... 12 more </span></span><span style="font-family:Helvetica, Tahoma, Arial, sans-serif;"><span style="line-height: 25.2px;">加上</span></span><span style="line-height: 25.2px; font-family: Helvetica, Tahoma, Arial, sans-serif;">conf.set("zookeeper.znode.parent","/hbase-unsecure");</span><span style="line-height: 25.2px; font-family: Helvetica, Tahoma, Arial, sans-serif;">就可以成功訪問並操作hbase資料庫。</span><span style="font-family:Helvetica, Tahoma, Arial, sans-serif;"><span style="line-height: 25.2px;">以上是個人實際學習過程的一些經驗，在此記錄一下，歡迎大家一起學習交流hbase相關技術！</span></span>
