2. 程式人生 > 其它 >Hadoop-Hive元件部署

Hadoop-Hive元件部署

阿新 • • 發佈：2022-04-09
目錄

一、基礎環境

Hive 元件需要基於之前已部署完畢的 Hadoop 全分佈系統，在 master 節點上實現 Hive 元件安裝。

1、Hadoop-全分散式配置

（全部配置）
2、Hadoop-叢集執行（部分配置）

# 檢視之前部署的環境
[root@master ~]# su - hadoop

[hadoop@master ~]$ jps
47967 Jps

[hadoop@master hadoop]$ start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
Starting namenodes on [master]
master: starting namenode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-namenode-master.example.com.out
192.168.100.30: starting datanode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-datanode-slave2.example.com.out
192.168.100.20: starting datanode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-datanode-slave1.example.com.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-secondarynamenode-master.example.com.out
starting yarn daemons
starting resourcemanager, logging to /usr/local/src/hadoop/logs/yarn-hadoop-resourcemanager-master.example.com.out
192.168.100.30: starting nodemanager, logging to /usr/local/src/hadoop/logs/yarn-hadoop-nodemanager-slave2.example.com.out
192.168.100.20: starting nodemanager, logging to /usr/local/src/hadoop/logs/yarn-hadoop-nodemanager-slave1.example.com.out

[hadoop@master hadoop]$ jps
44678 NameNode
45030 ResourceManager
45289 Jps
44874 SecondaryNameNode

[hadoop@master hadoop]$ exit
logout

要確保 Hadoop 系統能夠正常執行

二、安裝MySQL

注意：使用 root 使用者登入

1、傳輸需要的包到/opt目錄下

[root@master ~]# cd /opt/software/
[root@master software]# ls
apache-hive-2.0.0-bin.tar.gz  jdk-8u152-linux-x64.tar.gz  mysql-connector-java-5.1.46.jar
hadoop-2.7.1.tar.gz           mysql-5.7.18.zip

2、解除安裝自帶的MariaDB

[root@master software]# rpm -qa | grep mariadb
mariadb-libs-5.5.52-1.el7.x86_64
[root@master software]# rpm -e --nodeps mariadb-libs-5.5.52-1.el7.x86_64

3、解壓mysql-5.7.18.zip

[root@master software]# ls
apache-hive-2.0.0-bin.tar.gz  jdk-8u152-linux-x64.tar.gz  mysql-connector-java-5.1.46.jar
hadoop-2.7.1.tar.gz           mysql-5.7.18.zip

[root@master software]# unzip mysql-5.7.18.zip 
Archive:  mysql-5.7.18.zip
   creating: mysql-5.7.18/
  inflating: mysql-5.7.18/mysql-community-client-5.7.18-1.el7.x86_64.rpm  
  inflating: mysql-5.7.18/mysql-community-common-5.7.18-1.el7.x86_64.rpm  
  inflating: mysql-5.7.18/mysql-community-devel-5.7.18-1.el7.x86_64.rpm  
  inflating: mysql-5.7.18/mysql-community-libs-5.7.18-1.el7.x86_64.rpm  
  inflating: mysql-5.7.18/mysql-community-server-5.7.18-1.el7.x86_64.rpm

4、依次安裝MySQL資料庫

[root@master software]# cd mysql-5.7.18/
[root@master mysql-5.7.18]# ls
mysql-community-client-5.7.18-1.el7.x86_64.rpm  mysql-community-libs-5.7.18-1.el7.x86_64.rpm
mysql-community-common-5.7.18-1.el7.x86_64.rpm  mysql-community-server-5.7.18-1.el7.x86_64.rpm
mysql-community-devel-5.7.18-1.el7.x86_64.rpm

[root@master mysql-5.7.18]# rpm -ivh mysql-community-common-5.7.18-1.el7.x86_64.rpm
warning: mysql-community-common-5.7.18-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:mysql-community-common-5.7.18-1.e################################# [100%]
   
[root@master mysql-5.7.18]# rpm -ivh mysql-community-libs-5.7.18-1.el7.x86_64.rpm
warning: mysql-community-libs-5.7.18-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:mysql-community-libs-5.7.18-1.el7################################# [100%]
   
[root@master mysql-5.7.18]# rpm -ivh mysql-community-client-5.7.18-1.el7.x86_64.rpm
warning: mysql-community-client-5.7.18-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:mysql-community-client-5.7.18-1.e################################# [100%]
   
[root@master mysql-5.7.18]# rpm -ivh mysql-community-server-5.7.18-1.el7.x86_64.rpm
warning: mysql-community-server-5.7.18-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:mysql-community-server-5.7.18-1.e################################# [100%]

5、修改配置

新增配置資訊到/etc/my.cnf 檔案 symbolic-links=0 的下方

[root@master mysql-5.7.18]# vi /etc/my.cnf

default-storage-engine=innodb
innodb_file_per_table
collation-server=utf8_general_ci
init-connect='SET NAMES utf8'
character-set-server=utf8

6、啟動MySQL

[root@master mysql-5.7.18]# systemctl start mysqld
[root@master mysql-5.7.18]# systemctl status mysqld
● mysqld.service - MySQL Server
   Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2022-04-09 10:12:27 CST; 44s ago
     Docs: man:mysqld(8)
           http://dev.mysql.com/doc/refman/en/using-systemd.html
  Process: 45577 ExecStart=/usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid $MYSQLD_OPTS (code=exited, status=0/SUCCESS)
  Process: 45499 ExecStartPre=/usr/bin/mysqld_pre_systemd (code=exited, status=0/SUCCESS)
 Main PID: 45580 (mysqld)
   CGroup: /system.slice/mysqld.service
           └─45580 /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid

Apr 09 10:12:17 master.example.com systemd[1]: Starting MySQL Server...
Apr 09 10:12:27 master.example.com systemd[1]: Started MySQL Server.

7、查詢MySQL預設密碼

[root@master ~]# cat /var/log/mysqld.log | grep password
2022-04-09T02:12:24.671882Z 1 [Note] A temporary password is generated for root@localhost: /RrHz+J-t1fj

預設密碼為/RrHz+J-t1fj

8、初始化MySQL

是否更改 root 使用者密碼n
是否使用設定的密碼繼續y
是否刪除匿名使用者n
是否拒絕 root 使用者遠端登入n
是否刪除測試資料庫n
是否重新載入授權表n

[root@master ~]# mysql_secure_installation

Securing the MySQL server deployment.

Enter password for user root: 

The existing password for the user account root has expired. Please set a new password.

New password: 

Re-enter new password: 
The 'validate_password' plugin is installed on the server.
The subsequent steps will run with the existing configuration
of the plugin.
Using existing password for root.

Estimated strength of the password: 100 
Change the password for root ? ((Press y|Y for Yes, any other key for No) : n

Estimated strength of the password: 100 
Do you wish to continue with the password provided?(Press y|Y for Yes, any other key for No) : y
By default, a MySQL installation has an anonymous user,
allowing anyone to log into MySQL without having to have
a user account created for them. This is intended only for
testing, and to make the installation go a bit smoother.
You should remove them before moving into a production
environment.

Remove anonymous users? (Press y|Y for Yes, any other key for No) : n

 ... skipping.


Normally, root should only be allowed to connect from
'localhost'. This ensures that someone cannot guess at
the root password from the network.

Disallow root login remotely? (Press y|Y for Yes, any other key for No) : n

 ... skipping.
By default, MySQL comes with a database named 'test' that
anyone can access. This is also intended only for testing,
and should be removed before moving into a production
environment.


Remove test database and access to it? (Press y|Y for Yes, any other key for No) : n

 ... skipping.
Reloading the privilege tables will ensure that all changes
made so far will take effect immediately.

Reload privilege tables now? (Press y|Y for Yes, any other key for No) : n

 ... skipping.
All done!

9、進入資料庫，新增許可權

新增 root 使用者從本地和遠端訪問 MySQL 資料庫表單的授權

[root@master ~]# mysql -uroot -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 12
Server version: 5.7.18 MySQL Community Server (GPL)

Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> grant all privileges on *.* to root@'localhost' identified by 'Wangzhigang123!';
Query OK, 0 rows affected, 1 warning (0.00 sec)

mysql> grant all privileges on *.* to root@'%' identified by 'Wangzhigang123!';
Query OK, 0 rows affected, 1 warning (0.00 sec)

mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)

mysql> select user,host from mysql.user where user='root';
+------+-----------+
| user | host      |
+------+-----------+
| root | %         |
| root | localhost |
+------+-----------+
2 rows in set (0.00 sec)

mysql> exit;
Bye
[root@master ~]#

二、配置Hive元件

1、設定Hive環境變數

# 解壓hive檔案
[root@master ~]# tar -zxf /opt/software/apache-hive-2.0.0-bin.tar.gz 
[root@master ~]# mv apache-hive-2.0.0-bin/ /usr/local/src/hive

# 修改使用者和使用者組為 hadoop
[root@master ~]# chown -R hadoop:hadoop /usr/local/src/hive

# 新增配置資訊
[root@master ~]# vi /etc/profile
# set hive environment
export HIVE_HOME=/usr/local/src/hive
export PATH=$PATH:$HIVE_HOME/bin

[root@master ~]# source /etc/profile

2、修改Hive元件配置檔案

[root@master ~]# su - hadoop

[hadoop@master ~]$ cp /usr/local/src/hive/conf/hive-default.xml.template /usr/local/src/hive/conf/hive-site.xml

[hadoop@master ~]$ vi /usr/local/src/hive/conf/hive-site.xml

# 設定 MySQL 資料庫連線
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://master:3306/hive?createDatabaseIfNotExist=true&amp;useSSL=false</value>
    <description>JDBC connect string for a JDBC metastore</description>

# 配置 MySQL 資料庫 root 的密碼
    <name>javax.jdo.option.ConnectionPassword</name>
    <value>Wangzhigang123!</value>
    <description>password to use against metastore database</description>

# 驗證元資料儲存版本一致性,預設 false

    <name>hive.metastore.schema.verification</name>
    <value>false</value>

# 配置資料庫驅動
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>com.mysql.jdbc.Driver</value>
    <description>Driver class name for a JDBC metastore</description>
    
# 配置資料庫使用者名稱
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>root</value>
    <description>Username to use against metastore database</description>
    
# 配置目錄資訊（共4處）
    <name>hive.querylog.location</name>
    <value>/usr/local/src/hive/tmp</value>
    <description>Location of Hive run time structured log file</description>
    
    <name>hive.exec.local.scratchdir</name>
    <value>/usr/local/src/hive/tmp</value>
    
    <name>hive.downloaded.resources.dir</name>
    <value>/usr/local/src/hive/tmp/resources</value>
    
    <name>hive.server2.logging.operation.log.location</name>
    <value>/usr/local/src/hive/tmp/operation_logs</value>


[hadoop@master ~]$ mkdir -p /usr/local/src/hive/tmp{resources,operation_logs}

3、重啟all.sh

[hadoop@master ~]$ cp /opt/software/mysql-connector-java-5.1.46.jar /usr/local/src/hive/lib/

[hadoop@master ~]$ stop-all.sh
This script is Deprecated. Instead use stop-dfs.sh and stop-yarn.sh
Stopping namenodes on [master]
master: stopping namenode
192.168.100.20: stopping datanode
192.168.100.30: stopping datanode
Stopping secondary namenodes [0.0.0.0]
0.0.0.0: stopping secondarynamenode
stopping yarn daemons
stopping resourcemanager
192.168.100.20: stopping nodemanager
192.168.100.30: stopping nodemanager
no proxyserver to stop

[hadoop@master ~]$ start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
Starting namenodes on [master]
master: starting namenode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-namenode-master.example.com.out
192.168.100.20: starting datanode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-datanode-slave1.example.com.out
192.168.100.30: starting datanode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-datanode-slave2.example.com.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-secondarynamenode-master.example.com.out
starting yarn daemons
starting resourcemanager, logging to /usr/local/src/hadoop/logs/yarn-hadoop-resourcemanager-master.example.com.out
192.168.100.30: starting nodemanager, logging to /usr/local/src/hadoop/logs/yarn-hadoop-nodemanager-slave2.example.com.out
192.168.100.20: starting nodemanager, logging to /usr/local/src/hadoop/logs/yarn-hadoop-nodemanager-slave1.example.com.out

4、初始化資料庫

[hadoop@master ~]$ schematool -initSchema -dbType mysql
which: no hbase in (/usr/local/src/jdk/bin:/usr/local/src/hadoop/bin:/usr/local/src/hadoop/sbin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/local/src/hive/bin:/home/hadoop/.local/bin:/home/hadoop/bin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/src/hive/lib/hive-jdbc-2.0.0-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/src/hive/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/src/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL:        jdbc:mysql://master:3306/hive?createDatabaseIfNotExist=true&useSSL=false
Metastore Connection Driver :    com.mysql.jdbc.Driver
Metastore connection User:       root
Starting metastore schema initialization to 2.0.0
Initialization script hive-schema-2.0.0.mysql.sql
Initialization script completed
schemaTool completed

5、啟動hive

[hadoop@master ~]$ hive
which: no hbase in (/usr/local/src/jdk/bin:/usr/local/src/hadoop/bin:/usr/local/src/hadoop/sbin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/local/src/hive/bin:/home/hadoop/.local/bin:/home/hadoop/bin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/src/hive/lib/hive-jdbc-2.0.0-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/src/hive/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/src/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

Logging initialized using configuration in jar:file:/usr/local/src/hive/lib/hive-common-2.0.0.jar!/hive-log4j2.properties
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
hive> show databases;
OK
default
Time taken: 0.241 seconds, Fetched: 1 row(s)
hive> exit;
[hadoop@master ~]$

6、驗證

[hadoop@master ~]$ mysql -uroot -pWangzhigang123!
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 24
Server version: 5.7.18 MySQL Community Server (GPL)

Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| hive               |
| mysql              |
| performance_schema |
| sys                |
+--------------------+
5 rows in set (0.00 sec)

mysql> exit;
Bye
[hadoop@master ~]$ jps
47057 NameNode
47411 ResourceManager
47254 SecondaryNameNode
47967 Jps
[hadoop@master ~]$

slave1節點

[root@slave1 ~]# su - hadoop

[hadoop@slave1 ~]$ jps
43621 Jps
43351 DataNode
43464 NodeManager

slave2節點

[root@slave2 ~]# su - hadoop

[hadoop@slave2 ~]$ jps
41607 Jps
41337 DataNode
41450 NodeManager

宣告：未經許可，不得轉載

相關推薦

Hadoop-Hive元件部署

目錄 一、基礎環境 二、安裝MySQL 1、傳輸需要的包到/opt目錄下 2、解除安裝自帶的MariaDB

hadoop hive元件介紹及常用cli命令

hadoop hive元件介紹及常用cli命令 Hive架構圖 Hive產生原因 1 關係型資料庫以產生多年sql成熟

Docker部署Hadoop+Hive

Docker部署Hadoop+Hive 由於hadoop與hive等存在版本相容問題，安裝前可以先通過官網確認版本相容情況：

【Kubernetes系列】第7篇 CI/CD之元件部署

前言 應對敏捷開發的需求，對CI(持續整合))/CD（持續交付）的提出了更高的標準，今天來討論下，如何基於開源元件（gitlab/jenkins/harbor/kubernetes）使用CI/CD，賦能團隊的開發、運維。

CentOS7 下 Hadoop 單節點部署

準備工作 hadoop下載：(hadoop2.7.5) http://archive.apache.org/dist/hadoop/core/ 關閉防火牆：

大資料運維（37） Hadoop+Hive+HBase+Kylin 偽分散式安裝

問題導讀 1.Centos7如何安裝配置？ 2.linux網路配置如何進行？ 3.linux環境下java如何安裝？

HIVE報錯：Error: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask (state=08S01,code=2)

執行insert into table video_orc select * from video_ori;時報錯 檢視hive日誌發現具體報錯資訊如下：

hadoop yarn元件介紹

hadoop yarn元件介紹 Yarn的產生 mapReduc1.0 1單點故障 2擴充套件效率低 3資源利用率高

OpenStack之keystone元件部署

前言：Keystone部署在控制節點上，用以為OpenStack提供身份認證服務，之後OpenStack的所有核心服務元件都要依靠keystone來識別認證租戶的許可權。

hive啟動一直失敗 org.apache.hadoop.hive.metastore.HiveMetaException: Failed to load driver

技術標籤：大資料環境搭建 好不容把大資料叢集裝好了，但是hive始終啟動不了。

org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move source錯誤解決辦法

解決辦法參考： HDP3.1中spark2.3無法讀取Hive3.0資料 問題描述：ambari部署的spark和hive，在sparksql中執行insert into table xxx partition(dt=\'xxx\') select xxx from xxx where dt=\'xxx\'，報錯如下錯誤

hadoop實用元件啟停指令碼

技術標籤：hadoop運維hadoopshell 1）在hadoop102的/home/mars/bin目錄下建立指令碼 [[email protected] bin]$ vim zk.sh

Hadoop單節點部署

技術標籤：大資料hadoophdfs大資料單節點 前言 ​ 去年六月份已經搭過叢集模式hadoop生態HA叢集，為何如今又要編寫這篇單節點的文章？因為環境總是會變化的，叢集搭起來也是較為耗費時間，且作為一個生態是環環相

Hive | 安裝部署（三種訪問Hive的方式）

Hive 安裝（第一種） Hive 官網地址 文件檢視地址 下載地址 安裝部署 上傳、解壓、改名 到opt/module/hive

hive安裝部署

hive安裝部署 hive官網地址：http://hive.apache.org/ 下載地址：http://archive.apache.org/dist/hive/

hive 安裝部署（一）

1. hive 安裝 hive 下載地址 1、下載解壓： // 這裡選擇的是 1.2.1 的版本 [hadoop@hadoop1 apps]$ tar -zxvf apache-hive-1.2.1-bin.tar.gz

hadoop完全分散式部署

軟體版本 虛擬機器管理軟體：VMware WorkStation Pro 12.0 作業系統：CentOS 6.7 Hadoop：Hadoop 2.7.4

大資料Hadoop-Spark叢集部署知識總結（一）

大資料Hadoop-Spark叢集部署知識總結 一、啟動/關閉 hadoop myhadoop.sh start/stop 分步啟動：

Hadoop+Hive

前言： 在按照自己之前的文件搭建Hadoop和hive時遇到不少的問題，特此記錄。 參考部落格：

Hive 元件安裝配置

下載和解壓安裝檔案 基礎環境和安裝準備 Hive元件的部署規劃和軟體包路徑如下：