Author:謝

本文介紹在2個節點上安裝配置HadoopDB方法。各節點的名稱分別為Cluster01(192.168.0.1)，Cluster02(192.168.0.2)。

以下若是用 [email protected]:~ 代表 Cluster01 ~ Cluster02 都要做的.

1  首先在各節點上安裝Hadoop-0.20.2， 參考此文 。

2  各節點安裝設定 PostgreSQL：

• 安裝併為資料庫建立 hadoop 帳號，假定使用密碼為 1234
• [email protected]:~$ sudo apt-get install postgresql
• [email protected]
  :~$ sudo vim /etc/postgresql/8.4/main/pg_hba.conf
  • #local   all         all                               ident
    local   all         all                               trust
    # IPv4 local connections:
    #host    all         all         127.0.0.1/32          md5
    host    all         all         127.0.0.1/32          password
    host    all         all         192.168.0.1/24          password            # 加上Cluster 機器 IP 範圍
    # IPv6 local connections:
    #host    all         all         ::1/128               md5
    host    all         all         ::1/128               password
• [email protected]:~$ sudo /etc/init.d/postgresql-8.4 restart
• [email protected]:~$ sudo su - postgres
• [email protected]:~$ createuser hadoop
  • Shall the new role be a superuser? (y/n) y
    [email protected]:~$ psql
    psql (8.4.2)
    Type "help" for help.
    
    postgres=# alter user hadoop with password '1234';
    ALTER ROLE
    postgres=# /q
• 測試其他機器可否連線
  • [email protected]:~$ createdb testdb
  • [email protected]:~$ psql -h Cluster01 testdb 
    
    • 如果能夠連線上，則出現以下提示：
      • 口令:
        psql (8.4.2)
        SSL connection (cipher: DHE-RSA-AES256-SHA, bits: 256)
        Type "help" for help.
        
        testdb=#

3 設定HadoopDB 

• core-site.xml 檔案中新增以下內容：
  <property>
  <name>hadoopdb.config.file</name>
  <value>HadoopDB.xml</value>
  <description>The name of the HadoopDB cluster configuration file</description>
  </property>
  
  <property>
  <name>hadoopdb.fetch.size</name>
  <value>1000</value>
  <description>The number of records fetched from JDBC ResultSet at once</description>
  </property>
  
  <property>
  <name>hadoopdb.config.replication</name>
  <value>false</value>
  <description>Tells HadoopDB Catalog whether replication is enabled.
  Replica locations need to be specified in the catalog.
  False causes replica information to be ignored.</description>
  </property>

• 192.168.0.1
• 192.168.0.2

• #Properties for Catalog Generation
  ##################################
  nodes_file=nodes.txt
  # Relations Name and Table Name are the same
  relations_unchunked=raw
  relations_chunked=poi
  catalog_file=HadoopDB.xml
  ##
  #DB Connection Parameters
  ##
  port=5432
  username=hadoop
  password=1234
  driver=org.postgresql.Driver
  url_prefix=jdbc/:postgresql/://
  ##
  #Chunking properties
  ##
  # the number of databases on a node
  chunks_per_node=2
  # for udb0 ,udb1 ( 2 nodes = 0 ~ 1 )
  unchunked_db_prefix=udb
  # for cdb0 ,cdb1, ... , cdb5 (2 nodes x 3 chunks = 0~5 )
  chunked_db_prefix=cdb
  ##
  #Replication Properties
  ##
  dump_script_prefix=/root/dump_
  replication_script_prefix=/root/load_replica_
  dump_file_u_prefix=/mnt/dump_udb
  dump_file_c_prefix=/mnt/dump_cdb
  ##
  #Cluster Connection
  ##
  ssh_key=id_rsa-gsg-keypair

• 產生的 HadoopDB.xml 類似下面：
• <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
  
      <Nodes Password="1234" Username="hadoop" Driver="org.postgresql.Driver" Location=" 192.168.0.1 ">
          <Relations id="raw">
              <Partitions url="jdbc:postgresql:// 192.168.0.1 :5432/udb0" id="0"/>
          </Relations>
          <Relations id="poi">
              <Partitions url="jdbc:postgresql:// 192.168.0.1 :5432/cdb0" id="0"/>
              <Partitions url="jdbc:postgresql:// 192.168.0.1 :5432/cdb1" id="1"/>
          </Relations>
      </Nodes>
      <Nodes Password="1234" Username="hadoop" Driver="org.postgresql.Driver" Location=" 192.168.0.2 ">
          <Relations id="raw">
              <Partitions url="jdbc:postgresql:// 192.168.0.2 :5432/udb1" id="1"/>
          </Relations>
          <Relations id="poi">
              <Partitions url="jdbc:postgresql:// 192.168.0.2 :5432/cdb2" id="2"/>
              <Partitions url="jdbc:postgresql:// 192.168.0.2 :5432/cdb3" id="3"/>
          </Relations>
      </Nodes>
  </DBClusterConfiguration>

4 資料載入

• [email protected]:~$ createdb udb0
  [email protected]:~$ createdb udb1

• [email protected]:~$ psql udb0
  udb0=# 
  CREATE TABLE raw (
  ID int,
  NAME varchar(300)
  );
• 同理如 Cluster02

5 SMS安裝配置

    第3步下載的hadoopdb解壓後包含SMS_dist.tar.gz資料夾。

    SMS的安裝方法很簡單：首先將其解壓，然後開啟bin目錄下的hive-config.sh檔案，再最後加上一句：

    export HADOOP_HOME="此處寫hadoop的安裝路徑"。

5  以上設定完後，即可在一臺機器上(Ex: Cluster1) 執行 $ SMS_dist/bin/hive 看看成果

• [email protected]:~ $ SMS_dist/bin/hive
  hive> show tables;
  hive> select name from raw;