0 環境

• Ubuntu18.04
• openssh-server
• Hadoop3.0.2
• JDK1.8.0_191

1 ssh配置

1.0 安裝openssh-server

sudo apt-get install openssh-server

1.2 配置ssh登入

# 進入ssh目錄
cd ~/.ssh
# 使用rsa演算法生成祕鑰和公鑰對
ssh-keygen -t rsa
# 授權
cat ./id_rsa.pub >> ./authroized_keys

2 安裝Hadoop

2.1 下載Hadoop

Hadoop:

2.2 解壓至指定目錄

tar -zxvf hadoop-3.0.2.tar.gz -C /xindaqi/hadoop
 

2.3 配置hadoop檔案

2.3.1 core-site.xml

<configuration>
	<property>
		<name>hadoop.tmp</name>
		<value>file:/home/xdq/xinPrj/hadoop/hadoop-3.0.2/tmp</value>
		<description>Abase for other temporary directories.</description>
	</property>
</configuration
 
>

2.3.2 hadoop-env.sh

export JAVA_HOME=/usr/java/jdk1.8.0_191

2.3.3 hdfs-site.xml

<configuration>
	<property>
		<name>dfs.replication</name>
		<value>1</value>
	</property>
	<property>
		<name>dfs.permission</name>
		<value>false</value>
	</property>
	<property>
		<name>dfs.namenode.name.dir</name>
		<value>file:/home/xdq/xinPrj/hadoop/hadoop-3.0.2/tmp/dfs/name</value>
	</property>

	<property>
		<name>dfs.datanode.data.dir</name>
		<value>file:/home/xdq/xinPrj/hadoop/hadoop-3.0.2/tmp/dfs/data</value>
	</property>

	<property>
		<name>dfs.namenode.rpc-address</name>
		<value>localhost:8080</value>
	</property>

	<property>
		<name>dfs.namenode.secondary.http-address</name>
		<value>localhost:9001</value>
	</property>

	<property>
		<name>dfs.webhdfs.enabled</name>
		<value>true</value>
	</property>
</configuration>

2.3.4 log4j.properties

log4j.logger.org.apache.hadoop.util.NativeCodeLoader=ERROR

2.3.5 yarn-site.xml

<configuration>
	<property>
		<name>mapreduce.framework.name</name>
		<value>yarn</value>
	</property>
</configuration>

2.3.6 新建資料夾

hadoop-3.0.2建立儲存資料夾,目錄結構:

-- hadoop-3.0.2
   |      
   `-- tmp
       `-- dfs
           |-- data
           `-- name

3 測試

hadoop各部分的啟停檔案在sbin路徑下,sbin目錄結構如下:

-- hadoop-3.0.2
    |-- sbin
    |   |-- FederationStateStore
    |   |   |-- MySQL
    |   |   `-- SQLServer
    |   |-- distribute-exclude.sh
    |   |-- hadoop-daemon.sh
    |   |-- hadoop-daemons.sh
    |   |-- httpfs.sh
    |   |-- kms.sh
    |   |-- mr-jobhistory-daemon.sh
    |   |-- refresh-namenodes.sh
    |   |-- start-all.cmd
    |   |-- start-all.sh
    |   |-- start-balancer.sh
    |   |-- start-dfs.cmd
    |   |-- start-dfs.sh
    |   |-- start-secure-dns.sh
    |   |-- start-yarn.cmd
    |   |-- start-yarn.sh
    |   |-- stop-all.cmd
    |   |-- stop-all.sh
    |   |-- stop-balancer.sh
    |   |-- stop-dfs.cmd
    |   |-- stop-dfs.sh
    |   |-- stop-secure-dns.sh
    |   |-- stop-yarn.cmd
    |   |-- stop-yarn.sh
    |   |-- workers.sh
    |   |-- yarn-daemon.sh
    |   `-- yarn-daemons.sh

3.1 啟停hdfs

3.1.0 啟動

start-dfs.sh

3.1.2 停止

stop-dfs.sh

3.1.3 web測試

localhost:9870

Hadoop檔案系統

3.2 啟停yarn

3.2.1 啟動

start-yarn.sh

3.2.2 停止

stop-yarn.sh

3.2.3 web測試

localhost:8088

Hadoop資源管理系統

4 總結

• Hadoop3.x以後的版本,dfs檔案系統的埠為9870;
• Hadoop2.x版本,dfs檔案系統介面為50070;
• Hadoop2x和3.x的yarn介面均為8088;
• 相對於Mac部署Hadoop,Ubuntu部署沒走彎路,除了埠9870;
• (網上資源很重要呀,找對了少走彎路!哈哈哈!)

Mac部署Hadoop參考:Mac部署Hadoop環境

[參考文獻]
[1]https://blog.csdn.net/weixin_42001089/article/details/81865101
[2]https://blog.csdn.net/Xin_101/article/details/85225604