mysql hive的导数_troubleshooting-执行Oozie调度Hive导数脚本抛java.io.IOException: output.properties data exceeds ...

执行Oozie调度Hive导数脚本抛java.io.IOException: output.properties data exceeds its limit [2048]

原因分析

shell脚本中一次提交的hql-mr作业量太大，其中包含的信息超过oozie launcher一次容许的最大值2K(2K是默认值)

解决办法

1)修改oozie-site.xml：

oozie.action.max.output.data

204800

2)CDH集群中修改 oozie-site.xml 的 Oozie Server 高级配置代码段(安全阀)，如图：

异常日志

18/09/15 16:21:47 INFO CuratorFrameworkSingleton: Closing ZooKeeper client.

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.0.0-1.cdh6.0.0.p0.537114/jars/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.0.0-1.cdh6.0.0.p0.537114/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

18/09/15 16:21:58 INFO client.RMProxy: Connecting to ResourceManager at master.prodcdh.com/172.18.205.119:8032

18/09/15 16:21:58 INFO client.RMProxy: Connecting to ResourceManager at master.prodcdh.com/172.18.205.119:8032

Launcher AM execution failed

java.io.IOException: output.properties data exceeds its limit [2048]

at org.apache.oozie.action.hadoop.LocalFsOperations.getLocalFileContentAsString(LocalFsOperations.java:86)

at org.apache.oozie.action.hadoop.LauncherAM.processActionData(LauncherAM.java:521)

at org.apache.oozie.action.hadoop.LauncherAM.handleActionData(LauncherAM.java:501)

at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:229)

at org.apache.oozie.action.hadoop.LauncherAM$1.run(LauncherAM.java:153)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:422)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685)

at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:141)

Exception in thread "main" java.io.IOException: output.properties data exceeds its limit [2048]

at org.apache.oozie.action.hadoop.LocalFsOperations.getLocalFileContentAsString(LocalFsOperations.java:86)

at org.apache.oozie.action.hadoop.LauncherAM.processActionData(LauncherAM.java:521)

at org.apache.oozie.action.hadoop.LauncherAM.handleActionData(LauncherAM.java:501)

at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:229)

at org.apache.oozie.action.hadoop.LauncherAM$1.run(LauncherAM.java:153)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:422)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685)

at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:141)

随机推荐

【OpenGL】VAO与VBO

1.我们先了解什么是OpenGL对象(OpenGL Object) 根据OpenGL Wiki的定义: An OpenGL Object is an OpenGL construct that con ...

利用循环播放dataurl的视频来防止锁屏：NoSleep.js

mark下. 地址:http://www.open-open.com/lib/view/open1430796889882.html

ASP.NET地址栏form提交安全验证

以下类可以在web.config中直接配置,可以防范地址栏.表单提交的恶意数据. 安全模块作用: a.针对URL参数验证的功能,防止sql注入 b.针对form表单XSS漏洞的防护功能 c.针对上传文 ...

_doPostBack用法总结

转载在以下两篇博客: http://www.cnblogs.com/yongtaiyu/archive/2011/05/13/2045746.html http://www.cnblogs.com/F ...

cas sso单点登录系列7_ 单点登录cas常见问题系列汇总

转:http://blog.csdn.net/matthewei6/article/details/50709252 单点登录cas常见问题(一) - 子系统是否还要做session处理?单点登录ca ...

组队赛第二场：字符串哈希+DP

长春现场赛 HDU 4821  思路:这题周赛的时候没做出来,有点可惜了.要是当时记起来unsigned long long自己主动取模,然后提醒一下大帝的话,后续大帝就能过了. 唉,导致让他取了好多 ...

世纪互联、微软Azure与无穷小微积分

今年9月25日,世纪互联正式开通微软Azure商用服务,有感.          我是世纪互联创业历程的见证人(之中的一个),现在看到世纪互联推出微软Azure公有云的商用服务,心 ...

How to get the file in a resource folder

In a Maven project, we may often struggle to get a certain file (e.g. json file or sql file). Here i ...

笔记：Spring Cloud Ribbon 客户端配置详解

自动化配置 由于 Ribbon 中定义的每一个接口都有多种不同的策略实现,同时这些接口之间又有一定的依赖关系,Spring Cloud Ribbon 中的自动化配置能够很方便的自动化构建接口的具体实现 ...

css.1

background-repeat背景平铺background-x水平平铺 background-y垂直平铺 background:url()no-repeat;不平铺 background-size ...