loadFileSystems error & ExceptionUtils錯誤原因分析
C/C++程式通過hdfs.h訪問HDFS,執行時遇到如下錯誤,會是什麼原因了?(注:hadoop安裝在/data/hadoop/hadoop-2.4.0,而/data/hadoop/current是指向它的軟連結):
loadFileSystems error:
(unable to get stack trace for java.lang.NoClassDefFoundError exception: ExceptionUtils::getStackTrace error.)
hdfsBuilderConnect(forceNewInstance=0, nn=172.25.40.171, port=9001, kerbTicketCachePath=(NULL), userName=(NULL)) error:
(unable to get stack trace for java.lang.NoClassDefFoundError exception: ExceptionUtils::getStackTrace error.)
E0507 19:02:57.251287 17859 hdfs_persistence.cpp:31] connect hdfs://172.25.40.171:9001 error: 未知的錯誤 255
上述資訊中的關鍵項是“NoClassDefFoundError”和“ExceptionUtils”,也就是找不到ExceptionUtils,一般可推斷是因為找不到相應的jar
進一步用Google去搜索“apache-commons-lang.jar”,找到下載網址:http://commons.apache.org/proper/commons-lang/download_lang.cgi,上面可以下載commons-lang3-3.3.2-bin.tar.gz,解壓後就可以看到commons-lang3-3.3.2.jar。
hadoop的二進位制安裝包,應當自帶了這個檔案,通過努力,在hadoop安裝目錄下的
使用命令“hadoop classpath”可以,可以檢視hadoop的classpath:
./hadoop classpath
/data/hadoop/hadoop-2.4.0/etc/hadoop:/data/hadoop/hadoop-2.4.0/share/hadoop/common/lib/*:/data/hadoop/hadoop-2.4.0/share/hadoop/common/*:/data/hadoop/hadoop-2.4.0/share/hadoop/hdfs:/data/hadoop/hadoop-2.4.0/share/hadoop/hdfs/lib/*:/data/hadoop/hadoop-2.4.0/share/hadoop/hdfs/*:/data/hadoop/hadoop-2.4.0/share/hadoop/yarn/lib/*:/data/hadoop/hadoop-2.4.0/share/hadoop/yarn/*:/data/hadoop/hadoop-2.4.0/share/hadoop/mapreduce/lib/*:/data/hadoop/hadoop-2.4.0/share/hadoop/mapreduce/*:/data/hadoop/current/contrib/capacity-scheduler/*.jar
遺憾的是,tools並沒有出現,這應當就是問題所在,於是手工將它加進去(hadoop被安裝在/data/hadoop/current):
export CLASSPATH=`/data/hadoop/current/bin/hadoop classpath`:/data/hadoop/current/share/hadoop/tools/lib
重執行,發現還是不行:
loadFileSystems error:
(unable to get stack trace for java.lang.NoClassDefFoundError exception: ExceptionUtils::getStackTrace error.)
hdfsBuilderConnect(forceNewInstance=0, nn=172.25.40.171, port=9001, kerbTicketCachePath=(NULL), userName=(NULL)) error:
(unable to get stack trace for java.lang.NoClassDefFoundError exception: ExceptionUtils::getStackTrace error.)
E0507 19:52:48.197748 27787 hdfs_persistence.cpp:31] connect hdfs://172.25.40.171:9001 error: 未知的錯誤 255
好吧,來把硬的,直接指定commons-lang-2.6.jar:
export CLASSPATH=`/data/hadoop/current/bin/hadoop classpath`:/data/hadoop/current/share/hadoop/tools/lib/commons-lang-2.6.jar
重新執行程式,ExceptionUtils錯誤消失了,但遇到新錯誤:
loadFileSystems error:
java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FileSystem
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FileSystem
at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
hdfsBuilderConnect(forceNewInstance=0, nn=172.25.40.171, port=9001, kerbTicketCachePath=(NULL), userName=(NULL)) error:
java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration
at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
E0507 19:12:56.522274 19834 hdfs_persistence.cpp:31] connect hdfs://172.25.40.171:9001 error: 未知的錯誤 255
仍然是NoClassDefFoundError錯誤,原因應當是一樣的:classpath中漏了哪個目錄,這就要看FileSystem和Configuration在哪個jar中了。
FileSystem是在hadoop-common-2.4.0.jar中,而Configuration在commons-configuration-1.6.jar中,它們兩個都已經在classpath上,為何還報錯了?
對java不熟,嘗試將hadoop-common-2.4.0.jar和commons-configuration-1.6.jar直接加入到classpath:
export CLASSPATH=`/data/hadoop/current/bin/hadoop classpath`:/data/hadoop/current/share/hadoop/tools/lib/commons-lang-2.6.jar:/data/hadoop/current/share/hadoop/common/hadoop-common-2.4.0.jar:/data/hadoop/current/share/hadoop/common/lib/commons-configuration-1.6.jar
發現FileSystem和Configuration錯誤消失了,說明有效:
loadFileSystems error:
java.lang.NoClassDefFoundError: org/apache/commons/logging/LogFactory
at org.apache.hadoop.fs.FileSystem.<clinit>(FileSystem.java:95)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.logging.LogFactory
at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 1 more
hdfsBuilderConnect(forceNewInstance=0, nn=172.25.40.171, port=9001, kerbTicketCachePath=(NULL), userName=(NULL)) error:
java.lang.NoClassDefFoundError: org/apache/commons/logging/LogFactory
at org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:169)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.logging.LogFactory
at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 1 more
通過尋找,LogFactory在tools目錄下的commons-logging-1.1.3.jar中,把它也加入到classpath中:
export CLASSPATH=`/data/hadoop/current/bin/hadoop classpath`:/data/hadoop/current/share/hadoop/tools/lib/commons-lang-2.6.jar:/data/hadoop/current/share/hadoop/common/hadoop-common-2.4.0.jar:/data/hadoop/current/share/hadoop/common/lib/commons-configuration-1.6.jar:/data/hadoop/current/share/hadoop/common/lib/commons-logging-1.1.3.jar
再次執行,還是報錯:
java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem: Provider org.apache.hadoop.fs.viewfs.ViewFileSystem could not be instantiated: java.lang.NoClassDefFoundError: com/google/common/collect/Maps
at java.util.ServiceLoader.fail(ServiceLoader.java:232)
at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2364)
還是類似的錯誤,這樣下會去搞死人。通過上述的一些操作,估計需要將所有的jar檔案一個個的將入到classpath中。由於對java不熟悉,也只有先這樣做一做了:
find /data/hadoop/current/ -name *.jar|awk '{ printf("export CLASSPATH=%s:$CLASSPATH\n", $0); }'
終於可以了^_^,好折騰啊。