Ubuntu(16.04)下安裝配置hadoop(2.7.3)

系統採用vm下ubuntu16.04

一、Java環境搭建(參考我的其它部落格)

二、安裝ssh-server並實現免密碼登入

1、下載安裝ssh-server

sudo apt-get install openssh-server

2、啟動ssh

sudo /etc/init.d/ssh start

3、設定免密碼登入

cd ~/.ssh
ssh-keygen -t rsa  一直回車，直到生成rsa

4、匯入authorized_keys

cat ./id_rsa.pub >> ./authorized_keys

 

5、測試免密碼登入

ssh localhost

6、關閉防火牆

ufw disable

二、安裝執行Hadoop

1、官網下載hadoop安裝檔案，這裡用的是如下版本

hadoop-2.7.3.tar.gz
拖到虛擬機器桌面

2、將其解壓到/usr/local/hadoop目錄下

cd /usr/local
sudo mkdir hadoop
cd ~/桌面
sudo mv hadoop-2.7.3.tar.gz /usr/local/hadoop
cd /usr/local/hadoop
tar -zxvf hadoop-2.7.3.tar.gz

3、配置.bashrc檔案

sudo gedit ~/.bashrc

//末尾加上
#HADOOP VARIABLES START  
export JAVA_HOME=/usr/local/java/jvm/jdk1.8.0_162  
export HADOOP_INSTALL=/usr/local/hadoop/hadoop-2.7.3
export PATH=$PATH:$HADOOP_INSTALL/bin  
export PATH=$PATH:$HADOOP_INSTALL/sbin  
export HADOOP_MAPRED_HOME=$HADOOP_INSTALL  
export HADOOP_COMMON_HOME=$HADOOP_INSTALL  
export HADOOP_HDFS_HOME=$HADOOP_INSTALL  
export YARN_HOME=$HADOOP_INSTALL  
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native  
export HADOOP_OPTS="-Djava.library.path=$HADOOP_INSTALL/lib"  
#HADOOP VARIABLES END  

//修改完成之後儲存關閉，並輸入以下命令使環境變數立即生效
source ~/.bashrc
 

4、配置Hadoop

(1)修改 core-site.xml 配置檔案

cd /usr/local/hadoop/hadoop-2.7.3/etc/hadoop
sudo gedit core-site.xml

//修改內容如下：
<configuration>
        <property>
             <name>hadoop.tmp.dir</name>
             <value>file:/usr/local/hadoop/hadoop-2.7.3/tmp</value>
             <description>Abase for other temporary directories.</description>
        </property>
        <property>
             <name>fs.default.name </name>
             <value>hdfs://localhost:9000</value>
        </property>
</configuration>

(2)修改hdfs-site.xml配置檔案

sudo gedit hdfs-site.xml
//修改內容如下：
<configuration>
        <property>
             <name>dfs.replication</name>
             <value>1</value>
        </property>
        <property>
             <name>dfs.namenode.name.dir</name>
             <value>file:/usr/local/hadoop/hadoop-2.7.3/tmp/dfs/name</value>
        </property>
        <property>
             <name>dfs.datanode.data.dir</name>
             <value>file:/usr/local/hadoop/hadoop-2.7.3/tmp/dfs/data</value>
        </property>
</configuration>

(3)配置JAVA_HOME

sudo gedit  hadoop-env.sh

//修改內容如下：
# The java implementation to use.  
export JAVA_HOME=/usr/local/java/jvm/jdk1.8.0_162 
export HADOOP=/usr/local/hadoop/hadoop-2.7.3  
export PATH=$PATH:/usr/local/hadoop/hadoop-2.7.3/bin

5、執行測試

(1)首次啟動對NameNode格式化

hdfs namenode –format

(2)啟動hadoop

start-all.sh

(3)測試

jps
//這裡輸出了6個節點，說明安裝成功

注意如果jps缺少namenode需要刪除tmp，然後新建資料夾，再格式化，再重啟