技術標籤：hadoop

前言

Hadoop單節點偽分佈部署參考這篇，本篇來進行真正多節點部署。

角色分配

cdh01，cdh02，cdh03

解壓安裝及配置

• cdh01：

tar -zxvf hadoop-2.6.0-cdh5.15.1.tar.gz -C /opt/

！！！修改配置檔案

1.修改HDFS配置檔案 /opt/hadoop-2.6.0-cdh5.15.1/etc/hadoop路徑下面

hadoop-env.sh
	export
 
 JAVA_HOME=/usr/java/jdk1.8.0_261

core-site.xml
	<property>
	    <name>fs.defaultFS</name>
	    <value>hdfs://cdh01：8020</value>   # 寫namenode的ip和埠，8020和9000都可以，只要其他元件對應
	</property>
	<property>
	    <name>hadoop.tmp.dir</name>
	    <value>/opt/hadoop-2.6.0-cdh5.15.1/tmp<
 
/value>  #建立一個tmp資料夾用來儲存臨時檔案
	</property>

hdfs-site.xml
	<property>
	    <name>dfs.replication</name>
	    <value>3</value>    # 副本數量最大為datanode數量，3副本就夠了
	</property> 
	<property>
        <name>dfs.namenode.secondary.http-address</name>
        <
 
value>cdh02:50090</value>
	</property>


slaves      #slaves檔案，hdfs啟動時，通過這個檔案來啟動datanode，只要啟動節點有這個檔案就可以
	cdh01
	cdh02
	cdh03
			
修改YARN配置檔案 /opt/hadoop-2.6.0-cdh5.15.1/etc/hadoop路徑下面
yarn-env.sh
	export JAVA_HOME=/usr/java/jdk1.8.0_261

yarn-site.xml
	<property>
       	<name>yarn.nodemanager.aux-services</name>
       	<value>mapreduce_shuffle</value>
   	</property>

   	<!-- 指定YARN的ResourceManager的地址 -->
   	<property>
       	<name>yarn.resourcemanager.hostname</name>
       	<value>cdh01</value>
   	</property>

• cdh02-03

# 把hadoop檔案傳輸到其他兩臺機器上
xsync /opt/hadoop-2.6.0-cdh5.15.1

• 三臺機器新增配置檔案

vi /etc/profile
# 新增
export HADOOP_HOME=/opt/hadoop-2.6.0-cdh5.15.1
export PATH=$PATH:$HADOOP_HOME/bin

source /etc/profile

• 啟動
  在NameNode機器上

# 第一次啟動前，在cdh01上執行初始化
hdfs namenode -format

# 啟動
[[email protected] sbin] ./start-all.sh 
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
Starting namenodes on [cdh01]
cdh01: starting namenode, logging to /opt/hadoop-2.6.0-cdh5.15.1/logs/hadoop-root-namenode-cdh01.out
cdh03: starting datanode, logging to /opt/hadoop-2.6.0-cdh5.15.1/logs/hadoop-root-datanode-cdh03.out
cdh02: starting datanode, logging to /opt/hadoop-2.6.0-cdh5.15.1/logs/hadoop-root-datanode-cdh02.out
cdh01: starting datanode, logging to /opt/hadoop-2.6.0-cdh5.15.1/logs/hadoop-root-datanode-cdh01.out
Starting secondary namenodes [cdh02]
cdh02: starting secondarynamenode, logging to /opt/hadoop-2.6.0-cdh5.15.1/logs/hadoop-root-secondarynamenode-cdh02.out
starting yarn daemons
starting resourcemanager, logging to /opt/hadoop-2.6.0-cdh5.15.1/logs/yarn-root-resourcemanager-cdh01.out
cdh03: starting nodemanager, logging to /opt/hadoop-2.6.0-cdh5.15.1/logs/yarn-root-nodemanager-cdh03.out
cdh02: starting nodemanager, logging to /opt/hadoop-2.6.0-cdh5.15.1/logs/yarn-root-nodemanager-cdh02.out
cdh01: starting nodemanager, logging to /opt/hadoop-2.6.0-cdh5.15.1/logs/yarn-root-nodemanager-cdh01.out

# jps看一下各節點的程序
[[email protected] sbin]# jps
8055 Jps
7454 NameNode
7806 ResourceManager
7902 NodeManager
7551 DataNode

[[email protected] ~]# jps
3318 SecondaryNameNode
3223 DataNode
3384 NodeManager
3498 Jps

[[email protected] ~]# jps
2480 NodeManager
2599 Jps
2379 DataNode
# 各個節點的程序都啟動成功了

看一下webUI：cdh01:50070

# 可以通過命令看一下
hdfs dfsadmin -report

Configured Capacity: 119101992960 (110.92 GB)
Present Capacity: 88014856192 (81.97 GB)
DFS Remaining: 88014819328 (81.97 GB)
DFS Used: 36864 (36 KB)
DFS Used%: 0.00%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0

-------------------------------------------------
Live datanodes (3):

Name: xxx.xxx.xxx.xxx:50010 (cdh01)
Hostname: cdh01
Decommission Status : Normal
Configured Capacity: 39700664320 (36.97 GB)
DFS Used: 12288 (12 KB)
Non DFS Used: 20862103552 (19.43 GB)
DFS Remaining: 18838548480 (17.54 GB)
DFS Used%: 0.00%
DFS Remaining%: 47.45%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Sun Dec 06 22:08:04 CST 2020


Name: xxx.xxx.xxx.xxx:50010 (cdh02)
Hostname: cdh02
Decommission Status : Normal
Configured Capacity: 39700664320 (36.97 GB)
DFS Used: 12288 (12 KB)
Non DFS Used: 5113217024 (4.76 GB)
DFS Remaining: 34587435008 (32.21 GB)
DFS Used%: 0.00%
DFS Remaining%: 87.12%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Sun Dec 06 22:08:05 CST 2020


Name: xxx.xxx.xxx.xxx:50010 (cdh03)
Hostname: cdh03
Decommission Status : Normal
Configured Capacity: 39700664320 (36.97 GB)
DFS Used: 12288 (12 KB)
Non DFS Used: 5111816192 (4.76 GB)
DFS Remaining: 34588835840 (32.21 GB)
DFS Used%: 0.00%
DFS Remaining%: 87.12%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Sun Dec 06 22:08:05 CST 2020

# 關閉hadoop
[[email protected] sbin]# ./stop-all.sh 
This script is Deprecated. Instead use stop-dfs.sh and stop-yarn.sh
Stopping namenodes on [cdh01]
cdh01: stopping namenode
cdh01: stopping datanode
cdh03: stopping datanode
cdh02: stopping datanode
Stopping secondary namenodes [cdh02]
cdh02: stopping secondarynamenode
stopping yarn daemons
stopping resourcemanager
cdh03: stopping nodemanager
cdh02: stopping nodemanager
cdh01: stopping nodemanager
no proxyserver to stop