hadoop+mahout——错误集锦

75 阅读 0 评论 50 点赞

我是靠谱客的博主唠叨人生，最近开发中收集的这篇文章主要介绍hadoop+mahout——错误集锦，觉得挺不错的，现在分享给大家，希望可以做个参考。

概述

收藏使用 hadoop jar mahout-example-xxxx-job org.apache.....遇到的错误

INFO kmeans.Job: Running with default arguments
2018-12-02 11:38:29,046 INFO kmeans.Job: Preparing Input
2018-12-02 11:38:29,169 INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:8032
2018-12-02 11:38:29,713 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2018-12-02 11:38:30,282 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/root/.staging/job_1543721488371_0002
2018-12-02 11:38:32,950 INFO input.FileInputFormat: Total input files to process : 1
2018-12-02 11:38:34,301 INFO mapreduce.JobSubmitter: number of splits:1
2018-12-02 11:38:34,918 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled
2018-12-02 11:38:35,923 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1543721488371_0002
2018-12-02 11:38:35,927 INFO mapreduce.JobSubmitter: Executing with tokens: []
2018-12-02 11:38:36,194 INFO conf.Configuration: resource-types.xml not found
2018-12-02 11:38:36,194 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2018-12-02 11:38:36,265 INFO impl.YarnClientImpl: Submitted application application_1543721488371_0002
2018-12-02 11:38:36,307 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1543721488371_0002/
2018-12-02 11:38:36,307 INFO mapreduce.Job: Running job: job_1543721488371_0002
2018-12-02 11:38:44,388 INFO mapreduce.Job: Job job_1543721488371_0002 running in uber mode : false
2018-12-02 11:38:44,390 INFO mapreduce.Job:
map 0% reduce 0%
2018-12-02 11:38:44,419 INFO mapreduce.Job: Job job_1543721488371_0002 failed with state FAILED due to: Application application_1543721488371_0002 failed 2 times due to AM Container for appattempt_1543721488371_0002_000002 exited with
exitCode: 1
Failing this attempt.Diagnostics: [2018-12-02 11:38:43.376]Exception from container-launch.
Container id: container_1543721488371_0002_02_000001
Exit code: 1
[2018-12-02 11:38:43.379]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/hadoop-3.1.1/tmp/nm-local-dir/usercache/root/appcache/application_1543721488371_0002/filecache/11/job.jar/job.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
[2018-12-02 11:38:43.380]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/hadoop-3.1.1/tmp/nm-local-dir/usercache/root/appcache/application_1543721488371_0002/filecache/11/job.jar/job.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
For more detailed output, check the application tracking page: http://localhost:8088/cluster/app/application_1543721488371_0002 Then click on links to logs of each attempt.
. Failing the application.
2018-12-02 11:38:44,449 INFO mapreduce.Job: Counters: 0
Exception in thread "main" java.lang.IllegalStateException: Job failed!
at org.apache.mahout.clustering.conversion.InputDriver.runJob(InputDriver.java:110)
at org.apache.mahout.clustering.syntheticcontrol.kmeans.Job.run(Job.java:130)
at org.apache.mahout.clustering.syntheticcontrol.kmeans.Job.main(Job.java:60)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
at org.apache.hadoop.util.RunJar.main(RunJar.java:232)

当出现以上代码提示时可能是jar包路径问题，我们可以在yarn-site.xml显示添加以下代码指明。

<property>
<name>yarn.application.classpath</name>
<value>/usr/local/hadoop-3.1.1/etc/hadoop:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/*:/usr/local/hadoop-3.1.1/share/hadoop/common/*:/usr/local/hadoop-3.1.1/share/hadoop/hdfs:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/*:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/*:/usr/local/hadoop-3.1.1/share/hadoop/mapreduce/lib/*:/usr/local/hadoop-3.1.1/share/hadoop/mapreduce/*:/usr/local/hadoop-3.1.1/share/hadoop/yarn:/usr/local/hadoop-3.1.1/share/hadoop/yarn/lib/*:/usr/local/hadoop-3.1.1/share/hadoop/yarn/*
</value>
</property>

其中value值通过命令行执行hadoop classpath得到