oozie捕獲標準輸出&異常capture-output
阿新 • • 發佈：2018-12-10
對於普通的java-action或者shell-action 都是支援的只要標準輸出是"k1=v1"這中格式的就行：
現用test.py進行測試：
1 ##test.py 2 #! /opt/anaconda3/bin/python 3 4 import re 5 import os 6 import sys 7 import traceback 8 9 if __name__ == '__main__': 10 try: 11 print("k1=v1") 12 13 print(aaa) ##這裡是個故意設定的錯誤14 except Exception as e: 15 16 print(traceback.format_exc()) 17 exit(0) ##這個地方要特別注意，當異常退出時capture-output將會失效，所以要想獲取異常資訊，一定要正常退出，然後在decison節點處理錯誤退出
#workflow.xml <workflow-app xmlns="uri:oozie:workflow:0.4" name="adaf4df46a6597914b9ff6cd80eff542c6a"> <start to="python-node"/> <action name="python-node"> <shell xmlns="uri:oozie:shell-action:0.2"> <job-tracker>108412.server.bigdata.com.cn:8032</job-tracker> <name-node>hdfs://108474.server.bigdata.com.cn:8020</name-node> <configuration> <property> <name>oozie.launcher.mapred.job.queue.name</name> <value>ada.oozielauncher</value> </property> </configuration> <exec>model.py</exec> <file>model.py</file> <capture-output/> </shell> <ok to="python-node1"/> <error to="fail"/> </action> <action name="python-node1"> <shell xmlns="uri:oozie:shell-action:0.2"> <job-tracker>108412.server.bigdata.com.cn:8032</job-tracker> <name-node>hdfs://108474.server.bigdata.com.cn:8020</name-node> <configuration> <property> <name>oozie.launcher.mapred.job.queue.name</name> <value>ada.oozielauncher</value> </property> </configuration> <exec>echo</exec> <argument>k1=${wf:actionData("python-node")["k1"]}</argument> <capture-output/> </shell> <ok to="check-output"/> <error to="fail"/> </action> <decision name="check-output"> <switch> <case to="end"> ${wf:actionData('python-node1')['k1'] eq 'Hello Oozie'} </case> <default to="fail"/> </switch> </decision> <kill name="fail"> <message>Python action failed, error message[${wf:actionData('python-node')['k1']}]</message> <!--message>Python action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message--> </kill> <end name="end"/> </workflow-app>
1 #job.properties 2 oozie.use.system.libpath=True 3 security_enabled=False 4 dryrun=False 5 jobTracker=108412.server.bigdata.com.cn:8032 6 nameNode=hdfs://108474.server.bigdata.com.cn:8020 7 user.name=root 8 queueName=test 9 10 #sharelib配置不能配置hive,會報錯 11 #oozie.action.sharelib.for.spark=spark,hive spark-action 12 #oozie.action.sharelib.for.sqoop=sqoop,hbase 13 oozie.wf.application.path=${nameNode}/user/lyy/oozie/test
將以上test.py和workflow.xml放到hdfs的/user/lyy/oozie/test目錄下，使用一下命令提交：
oozie job -oozie http://10.8.4.46:11000/oozie -config job.properties -run
另外如果程式碼中有標準輸出，但是格式不是"k=v"型別的，則用el函式wf:actionData無法獲取，然而capture-output依舊把標準輸出的資訊捕獲了，儲存在oozie元資料表oozie.WF_ACTIONS中data欄位存放，這個欄位是mediumblob型別不能直接檢視，可以通過下面restfulAPI獲取json格式的資料，如下：
http://108446.server.bigdata.com.cn:11000/oozie/v1/job/0000106-181129152008300-oozie-oozi-W
{ "appPath":"hdfs://108474.server.bigdata.com.cn:8020/user/lyy/oozie/3a0c7d3a2ed5468087d93c69db651f3f", "acl":null, "status":"KILLED", "createdTime":"Mon, 10 Dec 2018 03:50:13 GMT", "conf":"<configuration> <property> <name>user.name</name> <value>root</value> </property> <property> <name>oozie.use.system.libpath</name> <value>True</value> </property> <property> <name>mapreduce.job.user.name</name> <value>root</value> </property> <property> <name>security_enabled</name> <value>False</value> </property> <property> <name>queueName</name> <value>ada.spark</value> </property> <property> <name>nameNode</name> <value>hdfs://108474.server.bigdata.com.cn:8020</value> </property> <property> <name>dryrun</name> <value>False</value> </property> <property> <name>jobTracker</name> <value>108412.server.bigdata.com.cn:8032</value> </property> <property> <name>oozie.wf.application.path</name> <value>hdfs://108474.server.bigdata.com.cn:8020/user/lyy/oozie/3a0c7d3a2ed5468087d93c69db651f3f</value> </property> </configuration>", "lastModTime":"Mon, 10 Dec 2018 03:51:17 GMT", "run":0, "endTime":"Mon, 10 Dec 2018 03:51:17 GMT", "externalId":null, "appName":"adaf4df46a6597914b9ff6cd80eff542c6a", "id":"0000106-181129152008300-oozie-oozi-W", "startTime":"Mon, 10 Dec 2018 03:50:13 GMT", "parentId":null, "toString":"Workflow id[0000106-181129152008300-oozie-oozi-W] status[KILLED]", "group":null, "consoleUrl":"http://108446.server.bigdata.com.cn:11000/oozie?job=0000106-181129152008300-oozie-oozi-W", "user":"root", "actions":[ { "errorMessage":null, "status":"OK", "stats":null, "data":null, "transition":"python-node", "externalStatus":"OK", "cred":"null", "conf":"", "type":":START:", "endTime":"Mon, 10 Dec 2018 03:50:14 GMT", "externalId":"-", "id":"[email protected]:start:", "startTime":"Mon, 10 Dec 2018 03:50:13 GMT", "userRetryCount":0, "externalChildIDs":null, "name":":start:", "errorCode":null, "trackerUri":"-", "retries":0, "userRetryInterval":10, "toString":"Action name[:start:] status[OK]", "consoleUrl":"-", "userRetryMax":0 }, { "errorMessage":null, "status":"OK", "stats":null, "data":"# #Mon Dec 10 11:50:24 CST 2018 File="./model.py", line 12, in <module> Traceback=(most recent call last)\: print(aaa)= NameError=name 'aaa' is not defined ####這個就是出錯的棧資訊 k1=v1 ##這個是標準輸出的資訊 ", "transition":"python-node1", "externalStatus":"SUCCEEDED", "cred":"null", "conf":"<shell xmlns="uri:oozie:shell-action:0.2"> <job-tracker>108412.server.bigdata.com.cn:8032</job-tracker> <name-node>hdfs://108474.server.bigdata.com.cn:8020</name-node> <configuration> <property xmlns=""> <name>oozie.launcher.mapred.job.queue.name</name> <value>ada.oozielauncher</value> <source>programatically</source> </property> </configuration> <exec>model.py</exec> <file>model.py</file> <capture-output /> </shell>", "type":"shell", "endTime":"Mon, 10 Dec 2018 03:50:24 GMT", "externalId":"job_1542533868365_0510", "id":"[email protected]", "startTime":"Mon, 10 Dec 2018 03:50:14 GMT", "userRetryCount":0, "externalChildIDs":null, "name":"python-node", "errorCode":null, "trackerUri":"108412.server.bigdata.com.cn:8032", "retries":0, "userRetryInterval":10, "toString":"Action name[python-node] status[OK]", "consoleUrl":"http://108412.server.bigdata.com.cn:8088/proxy/application_1542533868365_0510/", "userRetryMax":0 }, { "errorMessage":null, "status":"OK", "stats":null, "data":"# #Mon Dec 10 11:51:16 CST 2018 k1=v1 ", "transition":"check-output", "externalStatus":"SUCCEEDED", "cred":"null", "conf":"<shell xmlns="uri:oozie:shell-action:0.2"> <job-tracker>108412.server.bigdata.com.cn:8032</job-tracker> <name-node>hdfs://108474.server.bigdata.com.cn:8020</name-node> <configuration> <property xmlns=""> <name>oozie.launcher.mapred.job.queue.name</name> <value>ada.oozielauncher</value> <source>programatically</source> </property> </configuration> <exec>echo</exec> <argument>k1=v1</argument> ##這個就是正常的k1=v1標準輸出傳遞到了python-node1節點了 <capture-output /> </shell>", "type":"shell", "endTime":"Mon, 10 Dec 2018 03:51:16 GMT", "externalId":"job_1542533868365_0511", "id":"[email protected]", "startTime":"Mon, 10 Dec 2018 03:50:24 GMT", "userRetryCount":0, "externalChildIDs":null, "name":"python-node1", "errorCode":null, "trackerUri":"108412.server.bigdata.com.cn:8032", "retries":0, "userRetryInterval":10, "toString":"Action name[python-node1] status[OK]", "consoleUrl":"http://108412.server.bigdata.com.cn:8088/proxy/application_1542533868365_0511/", "userRetryMax":0 }, { "errorMessage":null, "status":"OK", "stats":null, "data":null, "transition":"fail", "externalStatus":"fail", "cred":"null", "conf":"<switch xmlns="uri:oozie:workflow:0.4"> <case to="end">false</case> <default to="fail" /> </switch>", "type":"switch", "endTime":"Mon, 10 Dec 2018 03:51:17 GMT", "externalId":"-", "id":"[email protected]", "startTime":"Mon, 10 Dec 2018 03:51:16 GMT", "userRetryCount":0, "externalChildIDs":null, "name":"check-output", "errorCode":null, "trackerUri":"-", "retries":0, "userRetryInterval":10, "toString":"Action name[check-output] status[OK]", "consoleUrl":"-", "userRetryMax":0 }, { "errorMessage":"Python action failed, error message[v1]", "status":"OK", "stats":null, "data":null, "transition":null, "externalStatus":"OK", "cred":"null", "conf":"Python action failed, error message[${wf:actionData('python-node')['k1']}]", "type":":KILL:", "endTime":"Mon, 10 Dec 2018 03:51:17 GMT", "externalId":"-", "id":"[email protected]", "startTime":"Mon, 10 Dec 2018 03:51:17 GMT", "userRetryCount":0, "externalChildIDs":null, "name":"fail", "errorCode":"E0729", "trackerUri":"-", "retries":0, "userRetryInterval":10, "toString":"Action name[fail] status[OK]", "consoleUrl":"-", "userRetryMax":0 } ] }