hdfs、hive、hbase的搭建總結
阿新 • • 發佈：2020-07-19
jdk的安裝
-
上傳jdk的安裝包到linux中
-
解壓、更名
[root@qianfeng01 software]# tar -zxvf jdk-8u221-linux-x64.tar.gz -C /usr/local/ [root@qianfeng01 software]# cd /usr/local [root@qianfeng01 local]# mv jdk1.8.0_221/ jdk -
環境變數的配置
[root@qianfeng01 local]# vi /etc/profile ......省略......... # java environment JAVA_HOME=/usr/local/jdk PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH export JAVA_HOME PATH -
重新載入配置檔案
[root@qianfeng01 local]# source /etc/profile -
驗證是否配置成功
[root@qianfeng01 local]# java -version [root@qianfeng01 local]# javachdfs完全分散式的搭建
-
上傳並解壓hadoop
[root@qianfeng01 ~]# tar -zxvf hadoop-2.7.6.tar.gz -C /usr/local/ -
更名
[root@qianfeng01 ~]# cd /usr/local [root@qianfeng01 local]# mv hadoop-2.7.6/ hadoop -
環境變數的配置
root@qianfeng01 local]# vi /etc/profile .........省略.......... #hadoop environment export HADOOP_HOME=/usr/local/hadoop export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH -
重新載入配置檔案
[root@qianfeng01 local]# source /etc/profile -
驗證是否裝好
[root@qianfeng01 local]# hadoop version
佈局
qianfeng01: namenode datanode resourcemanager nodemanager qianfeng02: secondarynamenode datanode nodemanager qianfeng03:
-
配置core-site.xml
<!-- 完全分散式檔案系統的名稱 ：schema ip port --> <property> <name>fs.defaultFS</name> <value>hdfs://qianfeng01/</value> </property> <!-- 分散式檔案系統的其他路徑的所依賴的一個基礎路徑，完全分散式不能使用預設值，因為臨路徑不安全，linux系統在重啟時，可能會刪除此目錄下的內容--> <property> <name>hadoop.tmp.dir</name> <value>/usr/local/hadoop/tmp</value> </property> -
配置hdfs-site.xml
<!-- namenode守護程序所管理檔案的儲存路徑 --> <property> <name>dfs.namenode.name.dir</name> <value>file://${hadoop.tmp.dir}/dfs/name</value> </property> <!-- datanode守護程序所管理檔案的儲存路徑 --> <property> <name>dfs.datanode.data.dir</name> <value>file://${hadoop.tmp.dir}/dfs/data</value> </property> <!-- hdfs的塊的副本數 --> <property> <name>dfs.replication</name> <value>3</value> </property> <!-- hdfs的塊大小，預設是128M --> <property> <name>dfs.blocksize</name> <value>134217728</value> </property> <!-- secondarynamenode的http服務的ip和port --> <property> <name>dfs.namenode.secondary.http-address</name> <value>qianfeng02:50090</value> </property> -
配置mapred-site.xml
<!-- mapreduce程式執行時所使用的框架的名稱--> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> <!-- mapreduce程式執行的歷史伺服器的ip和port--> <property> <name>mapreduce.jobhistory.address</name> <value>qianfeng01:10020</value> </property> <!-- mapreduce程式執行的webui的ip和port--> <property> <name>mapreduce.jobhistory.webapp.address</name> <value>qianfeng01:19888</value> </property> -
配置yarn-site.xml
<!-- 配置yarn框架使用其核心技術：shuffle--> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <!-- 配置resourcemanager所在的主機的名稱--> <property> <description>The hostname of the RM.</description> <name>yarn.resourcemanager.hostname</name> <value>qianfeng01</value> </property> -
配置hadoop-env.sh
[root@qianfeng01 hadoop]# vi hadoop-env.sh ......... # The java implementation to use. export JAVA_HOME=/usr/local/jdk -
配置yarn-env.sh
[root@qianfeng01 hadoop]# vi yarn-env.sh ......... # some Java parameters export JAVA_HOME=/usr/local/jdk -
配置slaves檔案（重點）
datanode守護程序所在主機的主機名稱 [root@qianfeng01 hadoop]# vi slaves qianfeng01 qianfeng02 qianfeng03
免密登陸
- 生產金鑰
ssh-keygen -t rsa 一路回車即可
-
分發金鑰，分發給自己就行了，克隆後再ssh
語法格式：ssh-copy-id -i 公鑰檔案 遠端使用者名稱@遠端機器IP 作用：將本機當前使用者的公鑰檔案，複製到遠端機器的相關使用者的主目錄的隱藏目錄.ssh下，同時自動更名為authorised_keys. 注意：.ssh目錄的許可權是700 authorised_keys的許可權是600
firewalld和NetworkManager以及selinux的關閉
檢視服務狀態: systemctl status firewalld
臨時關閉: systemctl stop firewalld
臨時啟動: systemctl start firewalld
設定開機不啟動: systemctl disable firewalld #下次開機生效
設定開機啟動: systemctl enable firewalld #下次開機生效
systemctl status NetworkManager
systemctl start NetworkManager
systemctl stop NetworkManager
systemctl disable NetworkManager
systemctl enable NetworkManager
[root@nz202 ~]# vi /etc/selinux/config
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=enforcing <---- 需要將enforcing改為disabled
# SELINUXTYPE= can take one of three values:
# targeted - Targeted processes are protected,
# minimum - Modification of targeted policy. Only selected processes are protected.
# mls - Multi Level Security protection.
SELINUXTYPE=targeted
克隆虛擬機器
-
修改主機名
]# hostnamectl set-hostname qianfeng02 -
修改IP
[root@qianfeng02 ~]# vi /etc/sysconfig/network-scripts/ifcfg-ens33 TYPE=Ethernet BOOTPROTO=static NAME=ens33 DEVICE=ens33 ONBOOT=yes IPADDR=192.168.10.102 <--- 修改ip地址即可，別的都不用動 NETMASK=255.255.255.0 GATEWAY=192.168.10.2 DNS1=192.168.10.2 DNS2=8.8.8.8 DNS3=114.114.114.114 -
重啟網路並檢查ip
systemctl restart network ip addr ping外網 ping主機 主機ping虛擬機器mysql的安裝
mysql的安裝
-
上傳安裝包，解壓
[root@qianfeng01 ~]# tar -xvf mysql-5.7.28-1.el7.x86_64.rpm-bundle.tar -C /usr/local/mysql mysql-community-embedded-5.7.28-1.el7.x86_64.rpm mysql-community-libs-compat-5.7.28-1.el7.x86_64.rpm mysql-community-devel-5.7.28-1.el7.x86_64.rpm mysql-community-embedded-compat-5.7.28-1.el7.x86_64.rpm mysql-community-libs-5.7.28-1.el7.x86_64.rpm mysql-community-test-5.7.28-1.el7.x86_64.rpm mysql-community-common-5.7.28-1.el7.x86_64.rpm mysql-community-embedded-devel-5.7.28-1.el7.x86_64.rpm mysql-community-client-5.7.28-1.el7.x86_64.rpm mysql-community-server-5.7.28-1.el7.x86_64.rpm -
安裝mysql所依賴的環境perl，移除mysql的衝突軟體mariadb
[root@qianfeng01 ~]# yum -y install perl [root@qianfeng01 ~]# yum -y install net-tools [root@qianfeng01 ~]# rpm -qa | grep mariadb mariadb-libs-5.5.64-1.el7.x86_64 [root@qianfeng01 ~]# rpm -e mariadb-libs-5.5.64-1.el7.x86_64 --nodeps -
按照mysql的依賴順序來安裝mysql的rpm包
[root@qianfeng01 ~]# rpm -ivh mysql-community-common-5.7.28-1.el7.x86_64.rpm [root@qianfeng01 ~]# rpm -ivh mysql-community-libs-5.7.28-1.el7.x86_64.rpm [root@qianfeng01 ~]# rpm -ivh mysql-community-client-5.7.28-1.el7.x86_64.rpm [root@qianfeng01 ~]# rpm -ivh mysql-community-server-5.7.28-1.el7.x86_64.rpm -
啟動mysql的服務項，並檢查狀態
[root@qianfeng01 ~]# systemctl start mysqld [root@qianfeng01 ~]# systemctl status mysqld ● mysqld.service - MySQL Server Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled) Active: active (running) since 五 2020-05-29 11:25:57 CST; 9s ago Docs: man:mysqld(8) http://dev.mysql.com/doc/refman/en/using-systemd.html Process: 2406 ExecStart=/usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid $MYSQLD_OPTS (code=exited, status=0/SUCCESS) Process: 2355 ExecStartPre=/usr/bin/mysqld_pre_systemd (code=exited, status=0/SUCCESS) Main PID: 2409 (mysqld) CGroup: /system.slice/mysqld.service └─2409 /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid 5月 29 11:25:52 qianfeng01 systemd[1]: Starting MySQL Server... 5月 29 11:25:57 qianfeng01 systemd[1]: Started MySQL Server. -
查詢mysql的初始密碼(密碼儲存在/var/log/mysqld.log檔案中，此檔案是服務項啟動後生成的)
[root@qianfeng01 ~]# cat /var/log/mysqld.log | grep password -
使用初始密碼進行登入
[root@qianfeng01 ~]# mysql -uroot -p'密碼' -
登入成功後，要降低密碼策略機制，改為low，也可以將密碼長度6.
set global validate_password_policy=low; set global validate_password_length=6; 檢視密碼策略，是否修改成功 show variables like '%validate_password%'; -
修改密碼
alter user root@localhost identified by '新密碼' -
如果想要遠端連線mysql，需要進行遠端授權操作(注意，一定要關閉虛擬機器防火牆)
*.*：所有庫下的所有表 root@“%”：root下的所有ip grant all privileges on *.* to root@"%" identified by '111111' with grant option;
hive的安裝
-
上傳解壓，更名
[root@qianfeng01 local]# tar -zxvf apache-hive-2.1.1-bin.tar.gz -C /usr/local [root@qianfeng01 local]# mv apache-hive-2.1.1-bin/ hive -
環境變數的配置
[root@qianfeng01 local]# vi /etc/profile # 新增如下內容: export HIVE_HOME=/usr/local/hive export PATH=$HIVE_HOME/bin:$PATH #讓profile生效 [root@qianfeng01 local ]# source /etc/profile -
hive-env.sh
export HIVE_CONF_DIR=/usr/local/hive/conf export JAVA_HOME=/usr/local/jdk export HADOOP_HOME=/urs/local/hadoop export HIVE_AUX_JARS_PATH=/usr/local/hive/lib -
hive-site.xml
<!--hive倉庫在hdfs的位置--> <property> <name>hive.metastore.warehouse.dir</name> <value>/user/hive/warehouse</value> <description>location of default database for the warehouse</description> </property> <!-- 該引數主要指定Hive的臨時檔案儲存目錄 --> <property> <name>hive.exec.scratchdir</name> <value>/tmp/hive</value> </property> <!--連線mysql的url地址--> <property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:mysql://qianfeng03:3306/hive?createDatabaseIfNotExist=true&characterEncoding=latin1</value> </property> <!--mysql的驅動類--> <property> <name>javax.jdo.option.ConnectionDriverName</name> <value>com.mysql.jdbc.Driver</value> </property> <!--mysql的使用者名稱--> <property> <name>javax.jdo.option.ConnectionUserName</name> <value>root</value> </property> <!--mysql遠端登陸的密碼--> <property> <name>javax.jdo.option.ConnectionPassword</name> <value>111111</value> </property> <!--hive工作的本地臨時儲存空間--> <property> <name>hive.exec.local.scratchdir</name> <value>/usr/local/hive/iotmp/root</value> </property> <!--如果啟用了日誌功能，則儲存操作日誌的頂級目錄--> <property> <name>hive.server2.logging.operation.log.location</name> <value>/usr/local/hive/iotmp/root/operation_logs</value> </property> <!--Hive執行時結構化日誌檔案的位置--> <property> <name>hive.querylog.location</name> <value>/usr/local/hive/iotmp/root</value> </property> <!--用於在遠端檔案系統中新增資源的臨時本地目錄--> <property> <name>hive.downloaded.resources.dir</name> <value>/usr/local/hive/iotmp/${hive.session.id}_resources</value> </property> 說明：使用遠端模式，需要在hadoop的core-site.xml檔案中新增一下屬性 <property> <name>hadoop.proxyuser.root.hosts</name> <value>*</value> </property> <property> <name>hadoop.proxyuser.root.groups</name> <value>*</value> </property>
hbase完全分散式
安裝zookeeper
-
上傳並解壓更名
[root@qianfeng01 ~]# tar -zxvf zookeeper-3.4.10.tar.gz -C /usr/local/ [root@qianfeng01 ~]# cd /usr/local/ [root@qianfeng01 local]# mv zookeeper-3.4.10 zookeeper -
配置環境變數
[root@qianfeng01 local]# vi /etc/profile .........省略...... export ZOOKEEPER_HOME=/usr/local/zookeeper export PATH=$ZOOKEEPER_HOME/bin:$PATH -
生效
[root@qianfeng01 local]# source /etc/profile -
驗證配置是否成功：使用tab鍵看看是否可以提示zookeeper相關指令碼
-
進入conf目錄下，複製一個zoo.cfg檔案
cp zoo_sample.cfg zoo.cfg -
修改zoo.cfg檔案
dataDir=/usr/local/zookeeper/zkData clientPort=2181 server.1=qianfeng01:2888:3888 server.2=qianfeng02:2888:3888 server.3=qianfeng03:2888:3888 -
如果dataDir屬性指定的目錄不存在，那麼要創建出來
mkdir /usr/local/zookeeper/zkData -
在zkData目錄下建立myid寫入相應數字
-
將/etc/profile檔案和zookeeper目錄scp到其他機器上
-
啟動zookeeper叢集，每臺機器上都要執行一下命令
zkServer.sh start zkServer.sh status
安裝hbase
-
上傳解壓更名
[root@qphone01 software]# tar -zxvf hbase-1.2.1-bin.tar.gz -C /opt/apps/ -
配置hbase-env.sh
[root@qphone01 conf]# vi hbase-env.sh # The java implementation to use. Java 1.7+ required. export JAVA_HOME=/opt/apps/jdk1.8.0_45 # Tell HBase whether it should manage it's own instance of Zookeeper or not. export HBASE_MANAGES_ZK=true -
hbase-site.xml
<configuration> <property> <name>hbase.cluster.distributed</name> <value>true</value> </property> <property> <name>hbase.rootdir</name> <value>hdfs://qphone01:9000/hbase</value> </property> <property> <name>hbase.zookeeper.quorum</name> <value>qphone01,qphone02,qphone03</value> </property> </configuration> -
regionservers
qphone01 qphone02 qphone03 -
將/etc/profile檔案和hbase目錄scp到其他機器上
-
在qphone01的hbase的conf目錄下建立backup-masters，寫入qphone02做備用機器
qphone02 -
測試
[root@qphone01 apps]# start-hbase.sh http://192.168.49.200:16010/master-status -
jps檢視程序
- HMaster//必須的，表示hbase正常，因為配置了兩個master一個是active狀態一個是備用狀態，所以在qianfeng01和qiangfeng02上各有一個hmaster
- QuorumPeerMain//必須單獨配置的Zookeeper叢集，如果是內建的則為HQuorumPeer，表示zookeeper正常
-
進入hbase的shell
hbase shell