1. 程式人生 > >idea工具裡執行MapReduce報許可權問題的錯誤

idea工具裡執行MapReduce報許可權問題的錯誤

問題描述:程式在eclipse正常執行,在idea裡不能執行

報錯資訊

java.lang.NoSuchFieldError: workaroundNonThreadSafePasswdCalls
	at org.apache.hadoop.io.nativeio.NativeIO.initNative(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO.<clinit>(NativeIO.java:89)
	at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:655)
	at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:514)
	at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:349)
	at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:193)
	at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:126)
	at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:942)
	at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
	at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)
	at org.apache.hadoop.mapreduce.Job.submit(Job.java:550)
	at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:580)
	at MapReducePhone.PhoneNum.main(PhoneNum.java:32)
Exception in thread "main" java.io.IOException: Failed to set permissions of path: \tmp\hadoop-hp\mapred\staging\hp1455504419\.staging to 0700
	at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:691)
	at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:664)
	at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:514)
	at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:349)
	at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:193)
	at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:126)
	at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:942)
	at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
	at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)
	at org.apache.hadoop.mapreduce.Job.submit(Job.java:550)
	at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:580)
	at MapReducePhone.PhoneNum.main(PhoneNum.java:32)

在嘗試Hadoop的各個版本後依然無法解決,最後看原始碼才看到問題的原因

在原始碼FileUtil.class下的490行有下列程式碼

  private static void checkReturnValue(boolean rv, File p, FsPermission permission) throws IOException {
        if (!rv) {
            throw new IOException("Failed to set permissions of path: " + p + " to " + String.format("%04o", permission.toShort()));
        }
    }
這是一個許可權的檢查功能,我們要把他註釋掉,但原始碼是不能改動的,所以這裡教大家一個取巧的方法

我們任意建一個包名為org.apache.hadoop.fs,類名為FileUtil的類,將原始碼的程式碼複製過來刪除490行的許可權檢查。
然後去原始碼FileUtil.class,在本機所在路徑下一般在maven下的org\apache\hadoop\hadoop-core\1.2.1下面有一個名為hadoop-core-1.2.1.jar的jar包,將這個jar包拿到一個新建資料夾名hadoop-core-1.2.1為下解壓,找到FileUtil.class檔案並刪除,將我們建的FileUtil的類複製進去修改後綴名為.class。
再將資料夾hadoop-core-1.2.1壓縮,修改壓縮後文件的字尾名為.jar,再將這個檔案替代我們org\apache\hadoop\hadoop-core\1.2.1資料夾下的jar包。

最後去 idea執行MapReduce,測試成功。

下面這個是我自己修改完成的jar包

https://download.csdn.net/download/weixin_43345864/10836668