環境 rhel6.5
hadoop 安裝與測試
[[email protected] ~]# useradd -u 800 hadoop
##id隨意，需要注意的是所有節點id必須一致
[[email protected] ~]# id hadoop
uid=800(hadoop) gid=800(hadoop) 組=800(hadoop
[[email protected] ~]# su - hadoop
[[email protected] ~]$ ls
hadoop-2.7.3.tar.gz jdk-7u79-linux-x64.tar.gz
[[email protected]

The java implementation to use.

export JAVA_HOME=/home/hadoop/jdk
##注意java變數路徑，否則後面系統起不來。
[[email protected] hadoop]$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount input output
偽分散式操作(需要ssh免密)
[[email protected] hadoop]$ vim etc/hadoop/core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://172.25.35.6:9000</value>
</property>
</configuration>
[[email protected] hadoop]$ vim etc/hadoop/hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
[[email protected] hadoop]$ vim etc/hadoop/slaves
172.25.35.6 ##將預設localhost改為master本主機ip地址
ssh 免密
[[email protected] hadoop]$ exit
logout
[[email protected] ~]# passwd hadoop
[[email protected] ~]# su - hadoop
[[email protected] ~]$ ssh-keygen
[[email protected] ~]$ ssh-copy-id 172.25.35.6
[[email protected] ~]$ ssh 172.25.35.6 ##測試登陸，不需要輸密碼就ok
也可這樣：
[[email protected] ~]$ ssh-keygen
得到公鑰後：
[[email protected] ~]$ cd .ssh/
[[email protected] .ssh]$ ls
[[email protected] .ssh]$ mv id_rsa.pub authorized_keys
[[email protected] hadoop]$ bin/hdfs namenode -format ##進行格式化
[[email protected] hadoop]$ sbin/start-dfs.sh ##啟動hadoop
[[email protected] hadoop]$ jps ##用jps檢驗各後臺程序是否成功啟動，看到以下四個程序，就成功了
6376 DataNode
6274 NameNode
6544 SecondaryNameNode
6687 Jps
瀏覽器輸入：172.25.35.5：70050

偽分佈的操作：
[[email protected] hadoop]$ bin/hdfs dfs -mkdir /user
[[email protected] hadoop]$ bin/hdfs dfs -mkdir /user/hadoop
[[email protected] hadoop]$ bin/hdfs dfs -put input test ##上傳本地的 input 並改名為 test
那怎麼檢視呢？用下面的命令
[[email protected] hadoop]$ bin/hdfs dfs -cat output/*
...
within 1
without 1
work 1
writing, 8
you 9
[[email protected] hadoop]$ bin/hdfs dfs -get output . ##將output下載到本地
[[email protected] hadoop]$ ls
bin include libexec logs output sbin
etc lib LICENSE.txt NOTICE.txt README.txt share
[[email protected] hadoop]$ bin/hdfs dfs -rm -r output ##刪除
17/10/24 21:11:24 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
Deleted output

hadoop 完全分佈模式搭建
用nfs網路檔案系統，就不用每個節點安裝一遍了，需要rpcbind和nfs開啟
[email protected] hadoop]$ sbin/stop-dfs.sh
[[email protected] hadoop]$ logout
[[email protected] ~]# yum install -y rpcbind
[email protected] ~]# yum install -y nfs-utils
[[email protected] ~]# vim /etc/exports
/home/hadoop (rw,anonuid=800,anongid=800)
[[email protected] ~]# /etc/init.d/rpcbind start
[[email protected] ~]# /etc/init.d/rpcbind status
[[email protected] ~]# /etc/init.d/nfs restart
啟動 NFS 服務： [確定]
關掉 NFS 配額： [確定]
啟動 NFS mountd： [確定]
啟動 NFS 守護程序：
[[email protected] ~]# showmount -e
Export list for server6:
/home/hadoop
[[email protected] ~]# exportfs -v
/home/hadoop <world>(rw,wdelay,root_squash,no_subtree_check,anonuid=800,anongid=800)
[[email protected] hadoop]$ vim etc/hadoop/core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://masters</value>
</property>
<property>
<name>ha.zookeeper.quorum</name>
<value>172.25.35.7:2181,172.25.35.8:2181</value>
</property>
</configuration>