spark 处理大量数据时报错： ERROR scheduler.LiveListenerBus: Listener EventLoggingListener threw an exception j

59 阅读 0 评论 39 点赞

我是靠谱客的博主柔弱皮带，最近开发中收集的这篇文章主要介绍spark 处理大量数据时报错： ERROR scheduler.LiveListenerBus: Listener EventLoggingListener threw an exception j，觉得挺不错的，现在分享给大家，希望可以做个参考。

概述

1、数据处理环境：

spark版本： 2.0.2

2、数据处理的特征有几千维度，具体报错信息为：

18/03/09 23:28:49 ERROR scheduler.LiveListenerBus: Listener EventLoggingListener threw an exception
java.io.IOException: All datanodes DatanodeInfoWithStorage[192.168.200.172:50010,DS-02bbf554-7d77-4f3e-bac6-1c69a2776cbf,DISK] are bad. Aborting...
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1109)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:871)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:401)
18/03/09 23:28:49 INFO scheduler.DAGScheduler: Job 28 finished: csv at FeatherAnalyseOnline.java:49, took 12636.035413 s
18/03/09 23:28:49 ERROR scheduler.LiveListenerBus: Listener EventLoggingListener threw an exception
java.io.IOException: All datanodes DatanodeInfoWithStorage[192.168.200.172:50010,DS-02bbf554-7d77-4f3e-bac6-1c69a2776cbf,DISK] are bad. Aborting...
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1109)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:871)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:401)
18/03/09 23:28:52 INFO datasources.DefaultWriterContainer: Job job_201803091958_0000 committed.
18/03/09 23:28:52 ERROR scheduler.LiveListenerBus: Listener EventLoggingListener threw an exception
java.io.IOException: All datanodes DatanodeInfoWithStorage[192.168.200.172:50010,DS-02bbf554-7d77-4f3e-bac6-1c69a2776cbf,DISK] are bad. Aborting...
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1109)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:871)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:401)
18/03/09 23:28:53 INFO spark.SparkContext: Invoking stop() from shutdown hook
18/03/09 23:28:53 ERROR scheduler.LiveListenerBus: Listener EventLoggingListener threw an exception
java.io.IOException: All datanodes DatanodeInfoWithStorage[192.168.200.172:50010,DS-02bbf554-7d77-4f3e-bac6-1c69a2776cbf,DISK] are bad. Aborting...
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1109)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:871)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:401)

3、解决办法；

修改 /etc/security/limits.conf

文件个数设置为：955350
，具体如下：

[root@dmp171 security]# vim limits.conf 
# /etc/security/limits.conf
#
#Each line describes a limit for a user in the form:
#
#<domain>        <type>  <item>  <value>
#
#Where:
#<domain> can be:
#        - a user name
#        - a group name, with @group syntax
#        - the wildcard *, for default entry
#        - the wildcard %, can be also used with %group syntax,
#                 for maxlogin limit
#
#<type> can have the two values:
#        - "soft" for enforcing the soft limits
#        - "hard" for enforcing hard limits
#
#<item> can be one of the following:
#        - core - limits the core file size (KB)
#        - data - max data size (KB)
#        - fsize - maximum filesize (KB)
#        - memlock - max locked-in-memory address space (KB)
#        - nofile - max number of open file descriptors
#        - rss - max resident set size (KB)
#        - stack - max stack size (KB)
#        - cpu - max CPU time (MIN)
#        - nproc - max number of processes
#        - as - address space limit (KB)
#        - maxlogins - max number of logins for this user
#        - maxsyslogins - max number of logins on the system
#        - priority - the priority to run user process with
#        - locks - max number of file locks the user can hold
#        - sigpending - max number of pending signals
#        - msgqueue - max memory used by POSIX message queues (bytes)
#        - nice - max nice priority allowed to raise to values: [-20, 19]
#        - rtprio - max realtime priority
#
#<domain>      <type>  <item>         <value>
#

#*               soft    core            0
#*               hard    rss             10000
#@student        hard    nproc           20
#@faculty        soft    nproc           20
#@faculty        hard    nproc           50
#ftp             hard    nproc           0
#@student        -       maxlogins       4

# End of file

root soft nofile 955350
root hard nofile 955350
root soft nproc unlimited
root hard nproc unlimited

4、测试结果：OK