idea工具裡執行MapReduce報許可權問題的錯誤
阿新 • • 發佈:2018-12-08
問題描述:程式在eclipse正常執行,在idea裡不能執行
報錯資訊
java.lang.NoSuchFieldError: workaroundNonThreadSafePasswdCalls at org.apache.hadoop.io.nativeio.NativeIO.initNative(Native Method) at org.apache.hadoop.io.nativeio.NativeIO.<clinit>(NativeIO.java:89) at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:655) at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:514) at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:349) at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:193) at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:126) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:942) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190) at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936) at org.apache.hadoop.mapreduce.Job.submit(Job.java:550) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:580) at MapReducePhone.PhoneNum.main(PhoneNum.java:32) Exception in thread "main" java.io.IOException: Failed to set permissions of path: \tmp\hadoop-hp\mapred\staging\hp1455504419\.staging to 0700 at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:691) at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:664) at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:514) at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:349) at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:193) at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:126) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:942) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190) at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936) at org.apache.hadoop.mapreduce.Job.submit(Job.java:550) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:580) at MapReducePhone.PhoneNum.main(PhoneNum.java:32)
在嘗試Hadoop的各個版本後依然無法解決,最後看原始碼才看到問題的原因
在原始碼FileUtil.class下的490行有下列程式碼
private static void checkReturnValue(boolean rv, File p, FsPermission permission) throws IOException { if (!rv) { throw new IOException("Failed to set permissions of path: " + p + " to " + String.format("%04o", permission.toShort())); } }
這是一個許可權的檢查功能,我們要把他註釋掉,但原始碼是不能改動的,所以這裡教大家一個取巧的方法
我們任意建一個包名為org.apache.hadoop.fs,類名為FileUtil的類,將原始碼的程式碼複製過來刪除490行的許可權檢查。
然後去原始碼FileUtil.class,在本機所在路徑下一般在maven下的org\apache\hadoop\hadoop-core\1.2.1下面有一個名為hadoop-core-1.2.1.jar的jar包,將這個jar包拿到一個新建資料夾名hadoop-core-1.2.1為下解壓,找到FileUtil.class檔案並刪除,將我們建的FileUtil的類複製進去修改後綴名為.class。
再將資料夾hadoop-core-1.2.1壓縮,修改壓縮後文件的字尾名為.jar,再將這個檔案替代我們org\apache\hadoop\hadoop-core\1.2.1資料夾下的jar包。