CentOS7上Apache impala2.7整合apache kudu1.3

阿新 • • 發佈：2018-11-14

轉自：https://blog.csdn.net/jssg_tzw/article/details/73549284

Apache impala整合apache kudu

1.環境準備

1.1.伺服器配置

ID	IP address	Hostname	Notes
1	192.168.56.181	apollo.dt.com	Hadoop Master 2. Spark Master 3.Kudu Master 4. Impala Master 5. JDK 6. Scala
2	192.168.56.182	artemis.dt.com	Hadoop Slave 2. Spark Slave 3.Kudu tserver 4. Impalad 5. JDK 6. Scala
3	192.168.56.183	uranus.dt.com	Hadoop Slave 2. Spark Slave 3.Kudu tserver 4. Impalad 5. JDK 6. Scala

1.2.環境配置

1.有關CentOS7.0安裝配置JDK請參考CentOS7安裝配置JDK1.8
2.有關Hadoop2.7.3 叢集配置請參考CentOS7上Hadoop叢集的搭建
3.有關Spark 2.10 叢集搭建請參考CentOS7上Spark叢集拱建
4.有關Scala 安裝配置請參考CentOS7上安裝配置Scala
5.有關Kudu 1.3.1叢集搭建請參考CentOS7上安裝配置Kudu叢集

1.3. impala 安裝包下載

下載impala安裝包，下載地址：http://archive.cloudera.com/beta/impala-kudu/redhat/7/x86_64/impala-kudu/0/RPMS/x86_64/
下載 bigtop-utils，下載地址：http://archive.cloudera.com/cdh5/redhat/7/x86_64/cdh/5.9.0/RPMS/noarch/

2.下載安裝依賴包

[[email protected] impala2.7]# yum -y install cyrus-sasl-plain lsb ntp

3.安裝impala

3.1.安裝 bigtop-utils (主從機都要安裝)

[[email protected] impala2.7]# rpm -ivh bigtop-utils-0.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.30.el7.noarch.rpm
warning: bigtop-utils-0.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.30.el7.noarch.rpm: Header V4 DSA/SHA1 Signature, key ID e8f86acd: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:bigtop-utils-0.7.0+cdh5.9.0+0-1.c################################# [100%]

3.2.安裝 impala-kudu-2.7.0 (主從機都要安裝)

# 要加上選項--nodeps否則安裝會因為缺少依賴包無法通過
[[email protected] impala2.7]# rpm -ivh impala-kudu-2.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.11.el7.x86_64.rpm --nodeps
warning: impala-kudu-2.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.11.el7.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID e8f86acd: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:impala-kudu-2.7.0+cdh5.9.0+0-1.cd################################# [100%]

3.3.安裝 impala-kudu-catalog (主機安裝)

[[email protected] impala2.7]# rpm -ivh impala-kudu-catalog-2.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.11.el7.x86_64.rpm
warning: impala-kudu-catalog-2.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.11.el7.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID e8f86acd: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:impala-kudu-catalog-2.7.0+cdh5.9.################################# [100%]

3.4.安裝 impala-kudu-state (主機安裝)

[[email protected] impala2.7]# rpm -ivh impala-kudu-state-store-2.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.11.el7.x86_64.rpm
warning: impala-kudu-state-store-2.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.11.el7.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID e8f86acd: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:impala-kudu-state-store-2.7.0+cdh################################# [100%]

3.5.主機安裝 impala-kudu-server (主從機都要安裝)

[[email protected] impala2.7]# rpm -ivh impala-kudu-server-2.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.11.el7.x86_64.rpm
warning: impala-kudu-server-2.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.11.el7.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID e8f86acd: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:impala-kudu-server-2.7.0+cdh5.9.0################################# [100%]

3.6.安裝impala-kudu-shell (主從機都要安裝)

[[email protected] impala2.7]# rpm -ivh impala-kudu-shell-2.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.11.el7.x86_64.rpm
warning: impala-kudu-shell-2.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.11.el7.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID e8f86acd: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:impala-kudu-shell-2.7.0+cdh5.9.0+################################# [100%]

3.7.安裝 impala-kudu-udf-devel(主從機都要安裝)

[[email protected] impala2.7]# rpm -ivh impala-kudu-udf-devel-2.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.11.el7.x86_64.rpm
warning: impala-kudu-udf-devel-2.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.11.el7.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID e8f86acd: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:impala-kudu-udf-devel-2.7.0+cdh5.################################# [100%]

4.配置impala (主從機都要配置)

4.1. 配置 vim /etc/default/bigtop-utils裡的JDK

# export JAVA_HOME
export JAVA_HOME=/usr/java/jdk1.8.0_77

4.2.配置vim /etc/default/impala (主從機都要配置)

IMPALA_CATALOG_SERVICE_HOST=apollo.dt.com
IMPALA_STATE_STORE_HOST=apollo.dt.com

5.配置三臺機器時間同步（主從機都要配置)

#主機：apollo.dt.com
[[email protected] kudu]# systemctl restart ntpd
#從機：artemis.dt.com
[[email protected] kudu]# systemctl restart ntpd
#從機：uranus.dt.com
[[email protected] kudu]# systemctl restart ntpd

#主機apollo.dt.com設定開機啟動
[[email protected] kudu]# systemctl enable ntpd
#從機artemis.dt.com設定開機啟動
[[email protected] kudu]# systemctl enable ntpd
#從機uranus.dt.com設定開機啟動
[[email protected] kudu]# systemctl enable ntpd

6.把hadoop配置檔案core-site.xml和hdfs-site.xml拷到/etc/impala/conf.dist/目錄下並作如下改動

6.1 拷貝core-site.xml和hdfs-site.xml (主從機都要配置)

#主機apollo.dt.com
[[email protected] .ssh]# cp $HADOOP_HOME/etc/hadoop/core-site.xml /etc/impala/conf.dist/
[[email protected] .ssh]# cp $HADOOP_HOME/etc/hadoop/hdfs-site.xml /etc/impala/conf.dist/
#從機artemis.dt.com
[[email protected] .ssh]# cp $HADOOP_HOME/etc/hadoop/core-site.xml /etc/impala/conf.dist/
[[email protected] .ssh]# cp $HADOOP_HOME/etc/hadoop/hdfs-site.xml /etc/impala/conf.dist/

#從機uranus.dt.com
[[email protected] .ssh]# cp $HADOOP_HOME/etc/hadoop/core-site.xml /etc/impala/conf.dist/
[[email protected] .ssh]# cp $HADOOP_HOME/etc/hadoop/hdfs-site.xml /etc/impala/conf.dist/

6.2.具體配置（在hadoop原有內容上增加以下內容）

core-site.xml

<property>
        <name>dfs.client.read.shortcircuit</name>
        <value>true</value>
</property>
<property>
        <name>dfs.client.read.shortcircuit.skip.checksum</name>
        <value>false</value>
</property>
<property>
        <name>dfs.datanode.hdfs-blocks-metadata.enabled</name>
        <value>true</value>
</property>

hdfs-site.xml

<!--impala configuration -->
<property>
        <name>dfs.datanode.hdfs-blocks-metadata.enabled</name>
        <value>true</value>
</property>
<property>
        <name>dfs.block.local-path-access.user</name>
        <value>impala</value>
</property>
<property>
        <name>dfs.client.file-block-storage-locations.timeout.millis</name>
        <value>60000</value>
</property>

7. 設定scoket path

在目錄/var/run/目錄下新建目錄hadoop-hdfs (主從機都要配置)。
注意：該資料夾可能已經存在，應當確認用impala是否有許可權進行讀寫。
如果已經存在，將使用者impala加入該檔案所屬的組，並修改該檔案組的許可權即: chown -R 775 hadoop-hdfs/

#主機：apollo.dt.com
[[email protected] .ssh]# cd /var/run/ #進入目錄/var/run/
[[email protected] run]# mkdir hadoop-hdfs #建立目錄hadoop-hdfs
[[email protected] run]# chown -R 775 hadoop-hdfs/ #設定許可權

#從機：artemis.dt.com
[[email protected] .ssh]# cd /var/run/ #進入目錄/var/run/
[[email protected] run]# mkdir hadoop-hdfs #建立目錄hadoop-hdfs
[[email protected] run]# chown -R 775 hadoop-hdfs/ #設定許可權

#從機：uranus.dt.com
[[email protected] .ssh]# cd /var/run/ #進入目錄/var/run/
[[email protected] run]# mkdir hadoop-hdfs #建立目錄hadoop-hdfs
[[email protected] run]# chown -R 775 hadoop-hdfs/ #設定許可權

8. 許可權配置（所有機器都應當配置）

如果想要 impala 和 YARN 合作，需要把 impala 使用者加入 hdfs 組，相關的可以瞭解的是Llama專案.
impala 在執行 DROP TABLE 操作時，需要把檔案移到到 hdfs 的回收站，所以你需要建立一個 hdfs 的目錄 /user/impala，並將其設定為impala 使用者可寫。同樣的，impala 需要讀取 hive 資料倉庫下的資料，故需要把 impala 使用者加入 hive 組。

#主機：apollo.dt.com
[[email protected] hadoop]# usermod -G hdfs,hadoop impala
[[email protected] hadoop]# groups impala
impala : impala hdfs hadoop

#從機：artemis.dt.com
[[email protected] hadoop]# usermod -G hdfs,hadoop impala
[[email protected] hadoop]# groups impala
impala : impala hdfs hadoop

#從機：uranus.dt.com
[[email protected] hadoop]# usermod -G hdfs,hadoop impala
[[email protected] hadoop]# groups impala
impala : impala hdfs hadoop

Extension: 建立impala在hdfs上的目錄並設定許可權

[[email protected] sbin]$ hadoop fs -mkdir -p /user/impala
[[email protected] sbin]$ hadoop fs -chown impala /user/impala

8.啟動impala

#主機apollo.dt.com
[[email protected] impala]# service impala-state-store restart --kudu_master_hosts=apollo.dt.com:7051
[[email protected] impala]# service impala-catalog restart --kudu_master_hosts=apollo.dt.com:7051
[[email protected] impala]# service impala-server restart --kudu_master_hosts=apollo.dt.com:7051
#從機：artemis.dt.com
[[email protected] .ssh]# service impala-server restart --kudu_master_hosts=apollo.dt.com:7051
#從機：uranus.dt.com
[[email protected] .ssh]# service impala-server restart --kudu_master_hosts=apollo.dt.com:7051

9. 登入impala-shell

[[email protected] impala]# impala-shell
Starting Impala Shell without Kerberos authentication
Connected to apollo.dt.com:21000
Server version: impalad version 2.7.0-cdh5-IMPALA_KUDU-cdh5 RELEASE (build 48f1ad385382cd90dbaed53b174965251d91d088)
***********************************************************************************
Welcome to the Impala shell. Copyright (c) 2015 Cloudera, Inc. All rights reserved.
(Impala Shell v2.7.0-cdh5-IMPALA_KUDU-cdh5 (48f1ad3) built on Thu Aug 18 12:15:44 PDT 2016)

Run the PROFILE command after a query has finished to see a comprehensive summary
of all the performance and diagnostic information that Impala gathered for that
query. Be warned, it can be very long!
***********************************************************************************
[apollo.dt.com:21000] >

10.在impala-shell端建立表

[apollo.dt.com:21000] >  CREATE TABLE dept (
 id BIGINT,
 name STRING
 ) distribute by range(name) split rows(('it'),('op'),('hr'))
 TBLPROPERTIES(
   'storage_handler' = 'com.cloudera.kudu.hive.KuduStorageHandler',
   'kudu.table_name' = 'dept',
   'kudu.master_addresses' = 'apollo.dt.com:7051',
   'kudu.key_columns' = 'id,name'
 );

11.在kudu控制檯檢視impala-shell建立的表

這裡寫圖片描述