說明

本博文較長，但是有效，如若計劃安裝多節點的hadoop，請一步一步堅持下去，有問題請留言，我們可以討論來解決問題。
本人將該4個節點的hadoop安裝在了vmware上了，同時支援安裝在物理機或者vmware ESXi上。

節點	說明	ip地址
hadoop01	主節點	192.168.10.61
hadoop02	從節點	192.168.10.62
hadoop03	從節點	192.168.10.63
hadoop04	從節點	192.168.10.64

請注意以下說明：

以root身份執行的命令為紅色字型
hadoop使用者執行的為黑色字型

環境

其中hadoop2.7.0和Jdk7.9.1兩個軟體我已經做好iso映象，大家可以來百度網盤下載：點選下載

準備模板

由於搭建hadoop過程中有許多地方的配置是重複的，故我們需要做一個模板避免過度重複勞動。

安裝RedhatServer

作業系統的安裝大家可直接從網上尋找，在此不再囉嗦。

關閉防火牆

根據自己的需要執行如下命令

service iptables status –檢視當前防火牆狀態

service iptables stop –關閉防火牆

chkconfig iptables off –永久關閉防火牆

關閉SElinux

執行如下命令

vim /etc/sysconfig/selinux

將SELINUX設定為disabled

開啟rsync

chkconfig rsync on

配置hosts

vim /etc/hosts

新增如下四行資料

192.168.10.61 hadoop01
192.168.10.62 hadoop02
192.168.10.63 hadoop03
192.168.10.64 hadoop04

建立hadoop使用者

執行如下命令建立hadoop使用者並設定其密碼為hadoop123

useradd hadoop
echo “hadoop123” | passwd –stdin hadoop

解壓hadoop、java檔案

本文中我是將所有的軟體包用軟碟通打包到hadoop2.7.1dvd.iso中去，然後掛載到虛擬機器的虛擬光碟機中，大家也可以用FileZilla等工具將軟體上傳到該系統中去，在media中會顯示一個20151220_122127的資料夾。
我的iso檔案如下

執行如下命令建立相應目錄並將安裝包複製到相應目錄

mkdir /opt/moudles
cd /media/20151220_122127/Hadoop2.7.1/
cp hadoop-2.7.1.tar.gz /opt/moudles/
cd /media/20151220_122127/JDK1.7.91/
cp jdk-7u91-linux-x64.tar.gz /opt/moudles/

執行如下命令將所屬使用者修改為hadoop使用者

chown hadoop /opt/moudles
chown hadoop /opt/moudles/hadoop-2.7.1.tar.gz
chown hadoop /opt/moudles/jdk-7u91-linux-x64.tar.gz

切換到hadoop使用者，對壓縮包進行解壓

su hadoop
cd /opt/moudles/
tar zxvf hadoop-2.7.1.tar.gz
tar zxvf jdk-7u91-linux-x64.tar.gz

配置環境變數

依舊以hadoop身份執行如下命令

cat>>~hadoop/.bashrc <<<EOF
JAVA_HOME=/opt/moudles/jdk1.7.0_91
export JAVA_HOME
HADOOP_HOME=/opt/moudles/hadoop-2.7.1
export HADOOP_HOME
PATH=\$HADOOP_HOME/bin:\$HADOOP_HOME/sbin:\$JAVA_HOME/bin:$PATH
EOF
exit

建立臨時目錄

以root使用者身份建立臨時目錄，並賦予hadoop使用者

mkdir -p /hadoopdata/hadoop/temp
chown -R hadoop /hadoopdata

關機，克隆4個節點

克隆4次，分別克隆出hadoop01~hadoop04，其中hadoop01為主節點，hadoop02-04為從節點。

開始安裝hadoop 叢集

對hadoop01~hadoop04修改

對hadoop01~hadoop04分別配置ip地址

節點	說明	ip地址
hadoop01	主節點	192.168.10.61
hadoop02	從節點	192.168.10.62
hadoop03	從節點	192.168.10.63
hadoop04	從節點	192.168.10.64

執行如下命令分別對各個節點修改主機名

vim /etc/sysconfig/network

將HOSTNAME設定為對應的名稱，如主節點將HOSTNAME設定為hadoop01，從節點分別設定為對應的hadoop02.hadoop03，hadoop04。
修改成功後執行reboot命令重新啟動

reboot

配置ssh

修改配置檔案

對主節點hadoop01進行修改

對環境檔案進行修改

涉及到的檔案主要有：

• /opt/moudles/hadoop-2.7.1/etc/hadoop/hadoop-env.sh
• /opt/moudles/hadoop-2.7.1/etc/hadoop/yarn-env.sh
• /opt/moudles/hadoop-2.7.1/etc/hadoop/mapred-env.sh
• /opt/moudles/hadoop-2.7.1/etc/hadoop/slaves

以hadoop使用者執行如下命令對hadoop-env.sh檔案進行修改

cd /opt/moudles/hadoop-2.7.1
vim etc/hadoop/hadoop-env.sh

修改：

export JAVA_HOME=/opt/moudles/jdk1.7.0_91

繼續執行如下命令對yarn-env.sh檔案進行修改

vim etc/hadoop/yarn-env.sh

修改：

export JAVA_HOME=/opt/moudles/jdk1.7.0_91

繼續執行如下命令對mapred-env.sh檔案修改

vim etc/hadoop/mapred-env.sh

修改：

export JAVA_HOME=/opt/moudles/jdk1.7.0_91

繼續執行如下命令對slaves檔案進行修改

cd /opt/moudles/hadoop-2.7.1/etc/hadoop/
cat>slaves<<EOF
hadoop02
hadoop03
hadoop04
EOF

對配置檔案進行修改

主要涉及的檔案有：

• /opt/moudles/hadoop-2.7.1/etc/hadoop/core-site.xml
• /opt/moudles/hadoop-2.7.1/etc/hadoop/hdfs-site.xml
• /opt/moudles/hadoop-2.7.1/ etc/hadoop/mapred-site.xml
• /opt/moudles/hadoop-2.7.1/ etc/hadoop/yarn-site.xml

執行如下命令，對core-site.xml進行修改

cd /opt/moudles/hadoop-2.7.1/etc/hadoop/
vim core-site.xml

修改為如下

<configuration>

<property>
<name>hadoop.tmp.dir</name>
<value>/hadoopdata/hadoop/temp</value>
<description>A base for other temporary directories.</description>
</property>

<property>
<name>fs.default.name</name>
<value>hdfs://hadoop01:9000</value>
<description>The name of the default file system.</description>
</property>

<property>
<name>io.file.buffer.size</name>
<value>131072</value>
<description>io file buffer size</description>
</property>

</configuration>

執行如下命令，對hdfs-site.xml進行修改

cd /opt/moudles/hadoop-2.7.1/etc/hadoop/
vim hdfs-site.xml

修改為如下

<configuration>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/hadoopdata/hadoop/hdfs/namenode</value>
</property>

<property>
<name>dfs.datanode.data.dir</name>
<value> file:/hadoopdata/hadoop/hdfs/datanode</value>
</property>

<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
</configuration>

執行如下命令，對mapred-site.xml進行修改

cd /opt/moudles/hadoop-2.7.1/etc/hadoop/
cp mapred-site.xml.template mapred-site.xml
vi mapred-site.xml

修改為如下

<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>

<property>
<name>mapreduce.jobhistory.address</name>
<value>hadoop01:10020</value>
</property>

<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value> hadoop01:19888</value>
</property>
</configuration>

執行如下命令，對yarn-site.xml進行修改

cd /opt/moudles/hadoop-2.7.1/etc/hadoop/
vim yarn-site.xml

修改為如下

<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>hadoop01:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>hadoop01:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>hadoop01:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>hadoop01:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>hadoop01:8088</value>
</property>
</configuration>

把配置檔案複製到其他Hadoop叢集節點

執行如下命令將配置檔案打包，然後傳到slave節點上

cd /opt/moudles/hadoop-2.7.1/etc/hadoop/
rm -rf hadoopconf
mkdir hadoopconf
cp hadoop-env.sh hadoopconf
cp core-site.xml hadoopconf
cp mapred-site.xml hadoopconf
cp slaves hadoopconf
cp hdfs-site.xml hadoopconf
cp yarn-site.xml hadoopconf
cp yarn-env.sh hadoopconf
cd hadoopconf
tar cvf hadoopconf.tar *
scp hadoopconf.tar hadoop@hadoop02:/opt/moudles/hadoop-2.7.1/etc/hadoop
scp hadoopconf.tar hadoop@hadoop03:/opt/moudles/hadoop-2.7.1/etc/hadoop
scp hadoopconf.tar hadoop@hadoop04:/opt/moudles/hadoop-2.7.1/etc/hadoop

分別在hadoop02，hadoop03，hadoop04上執行如下命令對其解包，完成Hadoop叢集配置檔案的同步

cd  /opt/moudles/hadoop-2.7.1/etc/hadoop/
tar xvf hadoopconf.tar

第一次啟動hadoop

格式化namenode

在主節點上執行

cd  /opt/moudles/hadoop-2.7.1/ 
./bin/hdfs namenode -format

啟動hdfs

在主節點上執行

cd  /opt/moudles/hadoop-2.7.1/
./sbin/start-dfs.sh

啟動yarn

在主節點上執行

cd  /opt/moudles/hadoop-2.7.1/
./sbin/start-yarn.sh
```shell
![yarn](https://img-blog.csdn.net/20160613155223649)




<div class="se-preview-section-delimiter"></div>

## 驗證安裝成功




<div class="se-preview-section-delimiter"></div>

###瀏覽器檢視
通過瀏覽器訪問http://hadoop01:50070
![瀏覽器](https://img-blog.csdn.net/20160613155409636)
通過瀏覽器訪問http://hadoop01:8088
![瀏覽器](https://img-blog.csdn.net/20160613155503215)




<div class="se-preview-section-delimiter"></div>

###程式驗證
執行如下程式碼執行帶有12個map和100個樣本的pi例項




<div class="se-preview-section-delimiter"></div>

```shell
cd /opt/moudles/hadoop-2.7.1/share/hadoop/mapreduce
yarn jar ./hadoop-mapreduce-examples-2.7.1.jar pi  12 100

執行結果如下所示：

Number of Maps = 12
Samples per Map = 100
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Wrote input for Map #4
Wrote input for Map #5
Wrote input for Map #6
Wrote input for Map #7
Wrote input for Map #8
Wrote input for Map #9
Wrote input for Map #10
Wrote input for Map #11
Starting Job
16/06/11 17:07:12 INFO client.RMProxy: Connecting to ResourceManager at hadoop01/192.168.10.61:8032
16/06/11 17:07:12 INFO input.FileInputFormat: Total input paths to process : 12
16/06/11 17:07:12 INFO mapreduce.JobSubmitter: number of splits:12
16/06/11 17:07:12 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1465618407612_0006
16/06/11 17:07:13 INFO impl.YarnClientImpl: Submitted application application_1465618407612_0006
16/06/11 17:07:13 INFO mapreduce.Job: The url to track the job: http://hadoop01:8088/proxy/application_1465618407612_0006/
16/06/11 17:07:13 INFO mapreduce.Job: Running job: job_1465618407612_0006
16/06/11 17:07:17 INFO mapreduce.Job: Job job_1465618407612_0006 running in uber mode : false
16/06/11 17:07:17 INFO mapreduce.Job: map 0% reduce 0%
16/06/11 17:07:29 INFO mapreduce.Job: map 8% reduce 0%
16/06/11 17:07:30 INFO mapreduce.Job: map 67% reduce 0%
16/06/11 17:07:36 INFO mapreduce.Job: map 75% reduce 0%
16/06/11 17:07:37 INFO mapreduce.Job: map 100% reduce 100%
16/06/11 17:07:37 INFO mapreduce.Job: Job job_1465618407612_0006 completed successfully
16/06/11 17:07:37 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=270
FILE: Number of bytes written=1505992
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=3170
HDFS: Number of bytes written=215
HDFS: Number of read operations=51
HDFS: Number of large read operations=0
HDFS: Number of write operations=3
Job Counters
Launched map tasks=12
Launched reduce tasks=1
Data-local map tasks=12
Total time spent by all maps in occupied slots (ms)=157380
Total time spent by all reduces in occupied slots (ms)=5029
Total time spent by all map tasks (ms)=157380
Total time spent by all reduce tasks (ms)=5029
Total vcore-seconds taken by all map tasks=157380
Total vcore-seconds taken by all reduce tasks=5029
Total megabyte-seconds taken by all map tasks=161157120
Total megabyte-seconds taken by all reduce tasks=5149696
Map-Reduce Framework
Map input records=12
Map output records=24
Map output bytes=216
Map output materialized bytes=336
Input split bytes=1754
Combine input records=0
Combine output records=0
Reduce input groups=2
Reduce shuffle bytes=336
Reduce input records=24
Reduce output records=0
Spilled Records=48
Shuffled Maps =12
Failed Shuffles=0
Merged Map outputs=12
GC time elapsed (ms)=3029
CPU time spent (ms)=35170
Physical memory (bytes) snapshot=3409559552
Virtual memory (bytes) snapshot=11427811328
Total committed heap usage (bytes)=2604138496
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=1416
File Output Format Counters
Bytes Written=97
Job Finished in 25.365 seconds
Estimated value of Pi is 3.14666666666666666667

關閉hadoop

停止yarn

在主節點上執行如下命令

cd  /opt/moudles/hadoop-2.7.1/
./sbin/stop-yarn.sh

停止hdfs

在主節點執行如下命令

cd  /opt/moudles/hadoop-2.7.1/
./sbin/stop-dfs.sh