Centos 7 配置單機Hadoop
阿新 • • 發佈:2018-12-14
系統映象:CentuOS-7-x86_64-Everything-1708
Java環境:JDK-8u181-linux-x64.tar.gz
Hadoop版本:hadoop-2.7.6.tar.gz
1、安裝配置JDK,配置環境變數(路徑很重要,我的路徑是:/usr/local/java/jdk1.8.0_181)
java環境的具體安裝過程不做記錄,下載好相應的壓縮包,解壓至你想要的路徑就好
環境變數如下::
export JAVA_HOME=/usr/local/java/jdk1.8.0_181 export JRE_HOME=${JAVA_HOME}/jre export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib export PATH=${JAVA_HOME}/bin:$PATH
2、下載hadoop
3、解壓到/opt目錄下
tar -zxvf hadoop-2.7.6.tar.gz -C /opt/
4、配置hadoop環境變數:
vim /etc/profile
export HADOOP_HOME=/opt/hadoop-2.7.6
export PATH=$PATH:$HADOOP_HOME/bin
source /etc/profile
5、配置hadoop
5.1 配置hadoop-env.sh
vim /opt/hadoop-2.7.6/etc/hadoop/hadoop-env.sh export JAVA_HOME=/opt/jdk1.8.0_45
5.2 配置core-site.xml
vim /opt/hadoop-2.7.6/etc/hadoop/core-site.xml <configuration> <property> <name>hadoop.tmp.dir</name> <value>file:///opt/hadoop-2.7.6</value> <description>Abase for other temporary directories.</description> </property> <property> <name>fs.defaultFS</name> <value>hdfs://192.168.139.129:8888</value> </property> </configuration>
5.3 配置 hdfs-site.xml
vim /opt/hadoop-2.7.6/etc/hadoop/hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:///opt/hadoop-2.7.6/tmp/dfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:///opt/hadoop-2.7.6/tmp/dfs/data</value>
</property>
</configuration>
6、配置ssh免密碼登入
ssh-keygen -t rsa
cd ~/.ssh
cat id_rsa.pub>>authorized_keys
#如果不能綿密登入,執行以下命令
chmod 710 authorized_keys
7、啟動
7.1 格式化HDFS
hdfs namenode -format
7.2 啟動
./sbin/start-dfs.sh
7.3 測試 http://192.168.139.129:50070
如果訪問不成功,請關閉linux系統防火牆,或者開放埠
8、配置yarn
8.1 配置mapred-site.xml
cd /opt/hadoop-2.7.6/etc/hadoop/
cp mapred-site.xml.template mapred-site.xml
vim mapred-site.xml
<configuration>
<!-- 通知框架MR使用YARN -->
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
8.2 配置yarn-site.xml
vim yarn-site.xml
<configuration>
<!-- reducer取資料的方式是mapreduce_shuffle -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
8.3啟動
cd /opt/hadoop-2.7.6
./sbin/start-yarn.sh
8.4 測試 http://192.168.139.129:8088