hadoop+mahout——錯誤集錦
阿新 • • 發佈:2018-12-02
收藏使用 hadoop jar mahout-example-xxxx-job org.apache.....遇到的錯誤
INFO kmeans.Job: Running with default arguments 2018-12-02 11:38:29,046 INFO kmeans.Job: Preparing Input 2018-12-02 11:38:29,169 INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:8032 2018-12-02 11:38:29,713 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 2018-12-02 11:38:30,282 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/root/.staging/job_1543721488371_0002 2018-12-02 11:38:32,950 INFO input.FileInputFormat: Total input files to process : 1 2018-12-02 11:38:34,301 INFO mapreduce.JobSubmitter: number of splits:1 2018-12-02 11:38:34,918 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled 2018-12-02 11:38:35,923 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1543721488371_0002 2018-12-02 11:38:35,927 INFO mapreduce.JobSubmitter: Executing with tokens: [] 2018-12-02 11:38:36,194 INFO conf.Configuration: resource-types.xml not found 2018-12-02 11:38:36,194 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'. 2018-12-02 11:38:36,265 INFO impl.YarnClientImpl: Submitted application application_1543721488371_0002 2018-12-02 11:38:36,307 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1543721488371_0002/ 2018-12-02 11:38:36,307 INFO mapreduce.Job: Running job: job_1543721488371_0002 2018-12-02 11:38:44,388 INFO mapreduce.Job: Job job_1543721488371_0002 running in uber mode : false 2018-12-02 11:38:44,390 INFO mapreduce.Job: map 0% reduce 0% 2018-12-02 11:38:44,419 INFO mapreduce.Job: Job job_1543721488371_0002 failed with state FAILED due to: Application application_1543721488371_0002 failed 2 times due to AM Container for appattempt_1543721488371_0002_000002 exited with exitCode: 1 Failing this attempt.Diagnostics: [2018-12-02 11:38:43.376]Exception from container-launch. Container id: container_1543721488371_0002_02_000001 Exit code: 1 [2018-12-02 11:38:43.379]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/local/hadoop-3.1.1/tmp/nm-local-dir/usercache/root/appcache/application_1543721488371_0002/filecache/11/job.jar/job.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] [2018-12-02 11:38:43.380]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/local/hadoop-3.1.1/tmp/nm-local-dir/usercache/root/appcache/application_1543721488371_0002/filecache/11/job.jar/job.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] For more detailed output, check the application tracking page: http://localhost:8088/cluster/app/application_1543721488371_0002 Then click on links to logs of each attempt. . Failing the application. 2018-12-02 11:38:44,449 INFO mapreduce.Job: Counters: 0 Exception in thread "main" java.lang.IllegalStateException: Job failed! at org.apache.mahout.clustering.conversion.InputDriver.runJob(InputDriver.java:110) at org.apache.mahout.clustering.syntheticcontrol.kmeans.Job.run(Job.java:130) at org.apache.mahout.clustering.syntheticcontrol.kmeans.Job.main(Job.java:60) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:318) at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
當出現以上程式碼提示時可能是jar包路徑問題,我們可以在yarn-site.xml顯示新增以下程式碼指明。
<property> <name>yarn.application.classpath</name> <value>/usr/local/hadoop-3.1.1/etc/hadoop:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/*:/usr/local/hadoop-3.1.1/share/hadoop/common/*:/usr/local/hadoop-3.1.1/share/hadoop/hdfs:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/*:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/*:/usr/local/hadoop-3.1.1/share/hadoop/mapreduce/lib/*:/usr/local/hadoop-3.1.1/share/hadoop/mapreduce/*:/usr/local/hadoop-3.1.1/share/hadoop/yarn:/usr/local/hadoop-3.1.1/share/hadoop/yarn/lib/*:/usr/local/hadoop-3.1.1/share/hadoop/yarn/* </value> </property>
其中value值通過命令列執行hadoop classpath得到