檢視YARN任務日誌的幾種方式
阿新 • • 發佈:2018-12-11
1、通過history server
通過history server,直接在web ui上檢視(如果任務異常退出,可能會看不到)
All Applications
application
application log
2、通過yarn命令(使用者要和提交任務的使用者一致)
1)yarn application -list -appStates ALL(這個不顯示時間資訊)
[[email protected] opt]$ yarn application -list -appStates ALL 17/07/26 10:51:33 INFO impl.TimelineClientImpl: Timeline service address: http://node2.hde.h3c.com:8188/ws/v1/timeline/ 17/07/26 10:51:33 INFO client.RMProxy: Connecting to ResourceManager at node2.hde.h3c.com/192.38.0.232:8050 Total number of applications (application-types: [] and states: [NEW, NEW_SAVING, SUBMITTED, ACCEPTED, RUNNING, FINISHED, FAILED, KILLED]):2 Application-Id Application-Name Application-Type User Queue State Final-State Progress Tracking-URL application_1500702983136_0002 word count MAPREDUCE ambari-qa default FINISHED SUCCEEDED 100% http://node2.hde.h3c.com:19888/jobhistory/job/job_1500702983136_0002 application_1500702983136_0001 DistributedShell YARN ambari-qa default FINISHED SUCCEEDED 100% N/A
2)yarn logs -applicationId application_1493700892407_0007
[[email protected] opt]$ yarn logs -applicationId application_1500702983136_0002 17/07/26 10:56:23 INFO impl.TimelineClientImpl: Timeline service address: http://node2.hde.h3c.com:8188/ws/v1/timeline/ 17/07/26 10:56:23 INFO client.RMProxy: Connecting to ResourceManager at node2.hde.h3c.com/192.38.0.232:8050 17/07/26 10:56:25 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library 17/07/26 10:56:25 INFO compress.CodecPool: Got brand-new decompressor [.deflate] Container: container_1500702983136_0002_01_000002 on node1.hde.h3c.com_45454 ============================================================================== LogType:stderr Log Upload Time:Sat Jul 22 01:59:41 -0400 2017 LogLength:0 Log Contents: End of LogType:stderr LogType:stdout Log Upload Time:Sat Jul 22 01:59:41 -0400 2017 LogLength:0 Log Contents: End of LogType:stdout LogType:syslog Log Upload Time:Sat Jul 22 01:59:41 -0400 2017 LogLength:3674 Log Contents: 2017-07-22 01:59:16,446 WARN [main] org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-maptask.properties,hadoop-metrics2.properties
3、直接檢視hdfs路徑的log
1)檢視yarn-site.xml,確定log配置目錄
<property> <name>yarn.nodemanager.remote-app-log-dir</name> <value>/app-logs</value> </property> ``` 2)檢視日誌檔案資訊(注意日期和時間) [[email protected] root]$ hdfs dfs -ls /app-logs/hdfs/logs Found 1 items drwxrwx--- - hdfs hadoop 0 2017-05-02 04:18 /app-logs/hdfs/logs/application_1493700892407_0007 3)檢視日誌詳情(注意檢視節點重啟前的幾個敏感app) yarn logs -applicationId application_1493700892407_0007(同2)