HIVE2:ERROR [main]: ql.Driver (:()) - FAILED: Execution Error, return code 1 from org.apache.hadoop.

51 阅读 0 评论 34 点赞

我是靠谱客的博主哭泣奇迹，最近开发中收集的这篇文章主要介绍HIVE2:ERROR [main]: ql.Driver (:()) - FAILED: Execution Error, return code 1 from org.apache.hadoop.，觉得挺不错的，现在分享给大家，希望可以做个参考。

概述

在Hive2.1 on Tez环境中运行select count(*) from students;时，遇到

ERROR [main]: ql.Driver (:()) - FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask

查看hive日志，具体问题是：

2016-12-20T17:38:50,298 INFO [Tez session start thread]: client.TezClient (:()) - Could not connect to AM, killing session via YARN, sessionName=HIVE-c6220a27-a8a5-4d55-a36d-a6d87bd8931c, applicationId=application_1482283108292_0001
2016-12-20T17:38:50,313 INFO [Tez session start thread]: impl.YarnClientImpl (:()) - Killed application application_1482283108292_0001
2016-12-20T17:38:50,313 ERROR [Tez session start thread]: tez.TezSessionState (:()) - Failed to start Tez session
org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. Application application_1482283108292_0001 failed 2 times due to AM Container for appattempt_1482283108292_0001_000002 exited with exitCode: -103

For more detailed output, check application tracking page:http://master:8088/cluster/app/application_1482283108292_0001Then, click on links to logs of each attempt.
Diagnostics: Container [pid=3048,containerID=container_1482283108292_0001_02_000001] is running beyond virtual memory limits. Current usage: 139.1 MB of 1 GB physical memory used;2.6 GB of 2.1 GB virtual memory used. Killing container.

解决方法：

调整hadoop配置文件yarn-site.xml中值：

<property>
    <name>yarn.scheduler.minimum-allocation-mb</name>
    <value>2048</value>
    <description>default value is 1024</description>
</property>
<property>
    <name>yarn.nodemanager.vmem-pmem-ratio</name>
    <value>2.1</value>
    <description>default value is 2.1</description>
</property>

增加yarn.scheduler.minimum-allocation-mb 数量，从缺省1024改为2048；上述运行问题即刻得到解决；

单独调整yarn.nodemanager.vmem-pmem-ratio从缺省值2.1调整到3.0，从计算上Vm=3.0*1=3.0>2.6 ，但是上述问题依然存在（原因？）

原因：

该错误是YARN的虚拟内存计算方式导致，上例中用户程序申请的内存为1Gb，YARN根据此值乘以一个比例（默认为2.1）得出申请的虚拟内存的值，当YARN计算的用户程序所需虚拟内存值大于计算出来的值时，就会报出以上错误。调节比例值可以解决该问题。具体参数为：yarn-site.xml中的yarn.nodemanager.vmem-pmem-ratio