hadoop執行python指令碼出錯:subprocess failed with code 127
阿新 • • 發佈:2018-12-12
一開始在ubuntu上,用vim寫了兩個.py檔案:mapper.py 和 reducer.py ,並通過
# hadoop jar /usr/lib/hadoop-0.20-mapreduce/contrib/streaming/hadoop-streaming-2.6.0-mr1-cdh5.8.0.jar \ > -input /user/cloudera/In/test.txt \ > -output /user/cloudera/test \ > -mapper ./mapper.py \ > -reducer ./reducer.py \ > -file /home/cloudera/Documents/map.py
執行成功。
後來在windows中,用pycharm寫了同樣內容的兩個檔案,mapper.py 和 reducer.py。再次進行上面的操作時,出現瞭如下錯誤:
INFO mapreduce.Job: Task Id : attempt_1490617885665_0008_m_000001_0, Status : FAILED Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 127 at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:325) at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:538) at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61) at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
然而經過仔細對照前後兩組mapper.py 和 reducer.py,實在發現不出差異,完全一樣。
最後在這裡找到了答案。
具體做法就是:在windows中的pycharm裡寫.py指令碼的時候,要把右下角的CRLF切換成LF
然後再把兩個.py檔案放到hadoop中執行,就OK了。