java 解決Could not locate executable null\bin\winutils.exe in the Hadoop bin
阿新 • • 發佈:2018-12-03
問題描述
在windows環境下執行連線hadoop服務的程式報告以下錯誤資訊:
og4j:ERROR Could not find value for key log4j.appender.logRollingFile
log4j:ERROR Could not instantiate appender named "logRollingFile".
[ERROR] 2018-09-17 17:40:02,151 method:org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:374)
Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:356)
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:371)
at org.apache.hadoop.util.Shell.<clinit>(Shell.java :364)
at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80)
at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2807)
at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2802)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java :2668)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
at org.apache.hadoop.fs.FileSystem$1.run(FileSystem.java:160)
at org.apache.hadoop.fs.FileSystem$1.run(FileSystem.java:157)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:157)
at com.yue.modular.hdfsClient.main(hdfsClient.java:31)
[WARN ] 2018-09-17 17:40:03,432 method:org.apache.hadoop.util.NativeCodeLoader.<clinit>(NativeCodeLoader.java:62)
Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
原因分析
根據錯誤資訊進入hadoop原始碼發現:
static {
WINDOWS = osType == Shell.OSType.OS_TYPE_WIN;
SOLARIS = osType == Shell.OSType.OS_TYPE_SOLARIS;
MAC = osType == Shell.OSType.OS_TYPE_MAC;
FREEBSD = osType == Shell.OSType.OS_TYPE_FREEBSD;
LINUX = osType == Shell.OSType.OS_TYPE_LINUX;
OTHER = osType == Shell.OSType.OS_TYPE_OTHER;
PPC_64 = System.getProperties().getProperty("os.arch").contains("ppc64");
//注意這行
HADOOP_HOME_DIR = checkHadoopHome();
WINUTILS = getWinUtilsPath();
isSetsidAvailable = isSetsidSupported();
TOKEN_SEPARATOR_REGEX = WINDOWS ? "[|\n\r]" : "[ \t\n\r\f]";
}
private static String checkHadoopHome() {
//高能高能~~~~~~~~~
String home = System.getProperty("hadoop.home.dir");
if (home == null) {
home = System.getenv("HADOOP_HOME");
}
try {
if (home == null) {
throw new IOException("HADOOP_HOME or hadoop.home.dir are not set.");
}
if (home.startsWith("\"") && home.endsWith("\"")) {
home = home.substring(1, home.length() - 1);
}
File homedir = new File(home);
if (!homedir.isAbsolute() || !homedir.exists() || !homedir.isDirectory()) {
throw new IOException("Hadoop home directory " + homedir + " does not exist, is not a directory, or is not an absolute path.");
}
home = homedir.getCanonicalPath();
} catch (IOException var2) {
if (LOG.isDebugEnabled()) {
LOG.debug("Failed to detect a valid hadoop home directory", var2);
}
home = null;
}
return home;
}
public static final String getQualifiedBinPath(String executable) throws IOException {
String fullExeName = HADOOP_HOME_DIR + File.separator + "bin" + File.separator + executable;
File exeFile = new File(fullExeName);
if (!exeFile.exists()) {
throw new IOException("Could not locate executable " + fullExeName + " in the Hadoop binaries.");
} else {
return exeFile.getCanonicalPath();
}
}
問題解決
分析發現是hadoop.home.dir 為null 和獲取HADOOP_HOME 變數沒找到目錄沒找到,
解決辦法2種:
第一種方法:新增環境變數HADOOP_HOME
第二種方法:直接java設定hadoop.home.dir 指向 winutil.exe 的目錄
主要說一下設定hadoop.home.dir 目錄指向:
System.setProperty("hadoop.home.dir", "C:\\Program Files\\Hadoop\\hadoop2.7.2-bin");
//構造一個訪問指定hdfs系統的客戶端物件
FileSystem fs = FileSystem.get(URI.create("hdfs://192.168.119.129:9000"), new Configuration(), "root");
winutils.exe下載地址:
winutils.exe下載地址