Hadoop2.6完全分散式多節點叢集安裝配置
<name>fs.defaultFS</name>
<value>hdfs://Master.Hadoop:9000</value>
</property>
<property>
<name>io.file.buffer.size</name>
<value>131072</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/hadoop/tmp</value>
<description>Abasefor other temporary directories.</description>
</property>
<property>
</configuration>
配置hdfs-site.xml//增加hdfs配置資訊(namenode、datanode埠和目錄位置)
<configuration>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>Master.Hadoop:9001</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/usr/hadoop/dfs/data</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/usr/hadoop/dfs/data</value>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
</configuration>
配置 mapred-site.xml 檔案//增加mapreduce配置(使用yarn框架、jobhistory使用地址以及web地址)
注意/usr/local/hadoop/etc/hadoop/資料夾下有mapred.xml.template檔案,需要複製並重命名
cp mapred-site.xml.template mapred-site.xml
sudo gedit mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>Master.Hadoop:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>Master.Hadoop:19888</value>
</property>
</configuration>
配置yarn-site.xml//增加yarn功能
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>Master.Hadoop:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>Master.Hadoop:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>Master.Hadoop:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>Master.Hadoop:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>Master.Hadoop:8088</value>
</property>
</configuration>
以上Master的hadoop配置完畢,配置salver的hadoop
把master下的hadoo資料夾傳到salver即可
普通使用者和root使用者均可注意sudo 和su的區別
命令:scp -r /usr/hadoop [email protected]:/usr/
給ysu使用者分配hadoop資料夾讀許可權 chown -R ysu:ysu hadoop 修改 /etc/profile檔案 新增hadoop路徑 cd profile sudo gedit /etc/profile 檔案中 # set hadoop pathexport HADOOP_HOME=/usr/hadoop
export PATH=PATH:PATH:HADOOP_HOME/bin 這樣slave的機器hadoop也配置好了 3.8 啟動Hadoop
格式化HDFS hdfs namenode -format 啟動hadoop /usr/hadoop/sbin/start-dfs.sh /usr/hadoop/sbin/start-yarn.sh 3.9檢視叢集和程序
jps//jps命令 /usr/hadoop/bin/hdfs dfsadmin -report //檢視叢集 (結果沒出來,有待修復) 火狐瀏覽器開啟 Master.Hadoop:50070 Master.Hadoop:8088(結果沒出來,有待修復) Slaver1.Hadoop:8042