1. 程式人生 > >Yarn命令使用及wordcount解析

Yarn命令使用及wordcount解析

Hadoop Yarn

前言:

前面幾篇博客主要介紹了MapReduce與Yarn的架構設計及簡單工作流程,本篇文章將以wordcount程序為例,簡單介紹下Yarn的使用。

1.wordcount示例運行

[root@hadoop000 ~]# su - hadoop
[hadoop@hadoop000 ~]$ jps
9201 SecondaryNameNode
9425 ResourceManager
13875 Jps
9540 NodeManager
8852 NameNode
8973 DataNode
# 創建wordcount目錄
[hadoop@hadoop000 ~]$ hdfs dfs -mkdir -p /wordcount/input
[hadoop@hadoop000 ~]$ vi test.log
jepson ruoze
hero yimi xjp
123
a b a
[hadoop@hadoop000 ~]$ hdfs dfs -put test.log /wordcount/input
[hadoop@hadoop000 ~]$ hdfs dfs -ls /wordcount/input           
Found 1 items
-rw-r--r--   1 hadoop supergroup         37 2018-05-29 20:38 /wordcount/input/test.log
# 執行wordcount示例jar包
[hadoop@hadoop000 ~]$ yarn jar > /opt/software/hadoop-2.8.1/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.1.jar > wordcount > /wordcount/input > /wordcount/output
18/05/29 20:40:59 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
18/05/29 20:40:59 INFO input.FileInputFormat: Total input files to process : 1
18/05/29 20:41:00 INFO mapreduce.JobSubmitter: number of splits:1
18/05/29 20:41:00 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1526991305992_0001
18/05/29 20:41:01 INFO impl.YarnClientImpl: Submitted application application_1526991305992_0001
18/05/29 20:41:01 INFO mapreduce.Job: The url to track the job: http://hadoop000:8088/proxy/application_1526991305992_0001/
18/05/29 20:41:01 INFO mapreduce.Job: Running job: job_1526991305992_0001
18/05/29 20:41:14 INFO mapreduce.Job: Job job_1526991305992_0001 running in uber mode : false
18/05/29 20:41:14 INFO mapreduce.Job:  map 0% reduce 0%
18/05/29 20:41:23 INFO mapreduce.Job:  map 100% reduce 0%
18/05/29 20:41:29 INFO mapreduce.Job:  map 100% reduce 100%
18/05/29 20:41:30 INFO mapreduce.Job: Job job_1526991305992_0001 completed successfully
18/05/29 20:41:30 INFO mapreduce.Job: Counters: 49
# 查看結果
[hadoop@hadoop000 ~]$ hdfs dfs -ls /wordcount/output
Found 2 items
-rw-r--r--   1 hadoop supergroup          0 2018-05-29 20:41 /wordcount/output/_SUCCESS
-rw-r--r--   1 hadoop supergroup         51 2018-05-29 20:41 /wordcount/output/part-r-00000
[hadoop@hadoop000 ~]$ hdfs dfs -cat /wordcount/output/part-r-00000
123     1
a       2
b       1
hero    1
jepson  1
ruoze   1
xjp     1
yimi    1

登錄網頁查看相關信息:http://192.168.6.217:8088/cluster
技術分享圖片

2.Yarn常用命令總結

yarn jar <jar>              --run a jar file
yarn application -list      --列出在跑的job
yarn application -kill application_1526991305992_0001(job的id) --殺掉在跑的job

3.wordcount流程詳解

技術分享圖片
參考:https://blog.csdn.net/yczws1/article/details/21794873

Yarn命令使用及wordcount解析