troubleshooting-执行Oozie调度Hive导数脚本抛java.io.IOException: output.properties data exceeds its limit [204...原因分析解决办法异常日志

62 阅读 0 评论 41 点赞

我是靠谱客的博主忧虑诺言，最近开发中收集的这篇文章主要介绍troubleshooting-执行Oozie调度Hive导数脚本抛java.io.IOException: output.properties data exceeds its limit [204...原因分析解决办法异常日志，觉得挺不错的，现在分享给大家，希望可以做个参考。

概述

执行Oozie调度Hive导数脚本抛java.io.IOException: output.properties data exceeds its limit [2048]

原因分析

shell脚本中一次提交的hql-mr作业量太大，其中包含的信息超过oozie launcher一次容许的最大值2K（2K是默认值）

解决办法

1）修改oozie-site.xml：
<property>
<name>oozie.action.max.output.data</name>
<value>204800</value>
</property>
2）CDH集群中修改 oozie-site.xml 的 Oozie Server 高级配置代码段（安全阀），如图：

参考：https://blog.csdn.net/abysscarry/article/details/82120855

异常日志

18/09/15 16:21:47 INFO CuratorFrameworkSingleton: Closing ZooKeeper client.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.0.0-1.cdh6.0.0.p0.537114/jars/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.0.0-1.cdh6.0.0.p0.537114/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
18/09/15 16:21:58 INFO client.RMProxy: Connecting to ResourceManager at master.prodcdh.com/172.18.205.119:8032
18/09/15 16:21:58 INFO client.RMProxy: Connecting to ResourceManager at master.prodcdh.com/172.18.205.119:8032
Launcher AM execution failed
java.io.IOException: output.properties data exceeds its limit [2048]
at org.apache.oozie.action.hadoop.LocalFsOperations.getLocalFileContentAsString(LocalFsOperations.java:86)
at org.apache.oozie.action.hadoop.LauncherAM.processActionData(LauncherAM.java:521)
at org.apache.oozie.action.hadoop.LauncherAM.handleActionData(LauncherAM.java:501)
at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:229)
at org.apache.oozie.action.hadoop.LauncherAM$1.run(LauncherAM.java:153)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685)
at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:141)
Exception in thread "main" java.io.IOException: output.properties data exceeds its limit [2048]
at org.apache.oozie.action.hadoop.LocalFsOperations.getLocalFileContentAsString(LocalFsOperations.java:86)
at org.apache.oozie.action.hadoop.LauncherAM.processActionData(LauncherAM.java:521)
at org.apache.oozie.action.hadoop.LauncherAM.handleActionData(LauncherAM.java:501)
at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:229)
at org.apache.oozie.action.hadoop.LauncherAM$1.run(LauncherAM.java:153)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685)
at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:141)