目錄

啟動YARN，並執行MapReduce程式

配置歷史伺服器

配置日誌聚集

啟動YARN，並執行MapReduce程式

[[email protected] hadoop]$ pwd
/opt/module/hadoop-2.7.2/etc/hadoop


#1，配置yarn-env.sh 
[[email protected] hadoop]$ vim yarn-env.sh 
# some Java parameters
export JAVA_HOME=/opt/module/jdk1.8.0_144

#2，配置
[[email protected]
 
 hadoop]$ vim yarn-site.xml 

<configuration>

<!-- Site specific YARN configuration properties -->

<!-- Reducer獲取資料的方式 -->
    <property>
          <name>yarn.nodemanager.aux-services</name>
           <value>mapreduce_shuffle</value>
    </property>

<!-- 指定YARN的ResourceManager的地址 -->
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>hadoop104</value>
    </property>
</configuration>

#3，配置mapred-env.sh 的java_home
[
 
[email protected] hadoop]$ vim mapred-env.sh 
export JAVA_HOME=/opt/module/jdk1.8.0_144

#4，將 mapred-site.xml.template更名為mapred-site.xml然後在配置該檔案
[[email protected] hadoop]$ mv mapred-site.xml.template mapred-site.xml
[[email protected] hadoop]$ vim mapred-site.xml 
<configuration>

<!-- 指定MR執行在YARN上 -->
<property>
                <name>mapreduce.framework.name</name>
                <value>yarn</value>
</property>
</configuration>

#5，啟動namenode和datanode，並且檢查是否啟動成功
[
 
[email protected] hadoop]$ hadoop-daemon.sh start namenode
starting namenode, logging to /opt/module/hadoop-2.7.2/logs/hadoop-isea-namenode-hadoop104.out
[[email protected] hadoop]$ hadoop-daemon.sh start datanode
starting datanode, logging to /opt/module/hadoop-2.7.2/logs/hadoop-isea-datanode-hadoop104.out
[[email protected] hadoop]$ jps
2817 Jps
2634 NameNode
2732 DataNode

#6，啟動yarn的resourcemanager 和 nodemanager並檢查是否成功，（命令列）（瀏覽器）
[[email protected] hadoop]$ yarn-daemon.sh start resourcemanager
starting resourcemanager, logging to /opt/module/hadoop-2.7.2/logs/yarn-isea-resourcemanager-hadoop104.out
[[email protected] hadoop]$ yarn-daemon.sh start nodemanager
starting nodemanager, logging to /opt/module/hadoop-2.7.2/logs/yarn-isea-nodemanager-hadoop104.out
[[email protected] hadoop]$ jps
2634 NameNode
2860 ResourceManager
2732 DataNode
3150 Jps
3102 NodeManager

在瀏覽器端驗證yarn是否啟動成功：

http://hadoop104:8088/cluster

#7，執行MapReduce程式，並檢視結果
[[email protected] hadoop-2.7.2]$ pwd
/opt/module/hadoop-2.7.2
[[email protected] hadoop-2.7.2]$ hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar wordcount /user/isea/input /user/isea/output
18/11/15 10:18:53 INFO client.RMProxy: Connecting to ResourceManager at hadoop104/192.168.1.104:8032
18/11/15 10:18:56 INFO input.FileInputFormat: Total input paths to process : 1
18/11/15 10:18:56 INFO mapreduce.JobSubmitter: number of splits:1
18/11/15 10:18:56 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1542247719338_0001
18/11/15 10:18:57 INFO impl.YarnClientImpl: Submitted application application_1542247719338_0001
18/11/15 10:18:57 INFO mapreduce.Job: The url to track the job: http://hadoop104:8088/proxy/application_1542247719338_0001/
18/11/15 10:18:57 INFO mapreduce.Job: Running job: job_1542247719338_0001
18/11/15 10:19:10 INFO mapreduce.Job: Job job_1542247719338_0001 running in uber mode : false
18/11/15 10:19:10 INFO mapreduce.Job:  map 0% reduce 0%
18/11/15 10:19:31 INFO mapreduce.Job:  map 100% reduce 0%
18/11/15 10:19:44 INFO mapreduce.Job:  map 100% reduce 100%
18/11/15 10:19:45 INFO mapreduce.Job: Job job_1542247719338_0001 completed successfully
*
*
*
[[email protected] hadoop-2.7.2]$ hdfs dfs -cat /user/isea/output/*
i	    2
isea	2
know	1
sea	    3
that	1
相應的在瀏覽器上也能獲取到相關的資訊

配置歷史伺服器

[[email protected] hadoop]$ pwd
/opt/module/hadoop-2.7.2/etc/hadoop

#1，配置mapred-site.xml，新增歷史伺服器的相關內容 
[[email protected] hadoop]$ vim mapred-site.xml 


<!-- 歷史伺服器端地址 -->
<property>
<name>mapreduce.jobhistory.address</name>
<value>hadoop104:10020</value>
</property>
<!-- 歷史伺服器web端地址 -->
<property>
    <name>mapreduce.jobhistory.webapp.address</name>
    <value>hadoop104:19888</value>
</property>

#2，啟動歷史伺服器，並檢視是否啟動成功
[[email protected] hadoop-2.7.2]$ pwd
/opt/module/hadoop-2.7.2
[[email protected] hadoop-2.7.2]$ mr-jobhistory-daemon.sh start historyserver
starting historyserver, logging to /opt/module/hadoop-2.7.2/logs/mapred-isea-historyserver-hadoop104.out
[[email protected] hadoop-2.7.2]$ jps
2634 NameNode
3884 Jps
2860 ResourceManager
2732 DataNode
3837 JobHistoryServer
3102 NodeManager

瀏覽器端訪問：

http://hadoop104:19888/jobhistory

配置日誌聚集

#1，配置yarn-site.xml 增加如下的內容
[[email protected] hadoop]$ pwd
/opt/module/hadoop-2.7.2/etc/hadoop
[[email protected] hadoop]$ vim yarn-site.xml 

<!-- 日誌聚集功能使能 -->
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>

<!-- 日誌保留時間設定7天 -->
<property>
<name>yarn.log-aggregation.retain-seconds</name>
<value>604800</value>
</property>

#2，關閉NodeManager 、ResourceManager和HistoryManager
[[email protected] hadoop]$ yarn-daemon.sh stop nodemanager
stopping nodemanager
nodemanager did not stop gracefully after 5 seconds: killing with kill -9
[[email protected] hadoop]$ yarn-daemon.sh stop resourcemanager
stopping resourcemanager
[[email protected] hadoop]$ mr-jobhistory-daemon.sh stop historyserver
stopping historyserver
[[email protected] hadoop]$ jps
2634 NameNode
4332 Jps
2732 DataNode

#3，啟動NodeManager 、ResourceManager和HistoryManager
[[email protected] hadoop]$ yarn-daemon.sh start resourcemanager
starting resourcemanager, logging to /opt/module/hadoop-2.7.2/logs/yarn-isea-resourcemanager-hadoop104.out
[[email protected] hadoop]$ yarn-daemon.sh start nodemanager
starting nodemanager, logging to /opt/module/hadoop-2.7.2/logs/yarn-isea-nodemanager-hadoop104.out
[[email protected] hadoop]$ mr-jobhistory-daemon.sh start historyserver
starting historyserver, logging to /opt/module/hadoop-2.7.2/logs/mapred-isea-historyserver-hadoop104.out
[[email protected] hadoop]$ jps
4804 JobHistoryServer
4933 Jps
4630 NodeManager
4377 ResourceManager
2634 NameNode
2732 DataNode

#4，刪除之前的輸出檔案，並執行MapReduce程式
[[email protected] hadoop-2.7.2]$ pwd
/opt/module/hadoop-2.7.2
[[email protected] hadoop-2.7.2]$ hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar wordcount /user/isea/input /user/isea/output


#5，檢視日誌檔案在瀏覽器端

http://hadoop104:19888/jobhistory