hive 动态分区(Dynamic Partition)异常处理

绝地灬酷狼 2022-05-28 01:19 292阅读 0赞

** hive 动态分区(Dynamic Partition)异常处理**

# Changing Hive Dynamic Partition Limits #

Symptoms:

Hive enforces limits on the number of dynamic partitions that it creates.  The default is 100 dynamic partitions per node, with a total (default) limit of 1000 dynamic partitions across all nodes. However, this can be tuned.  If your job tries to create too many dynamic partitions, you may see a stack trace similar to the following:

Caused by: org.apache.hadoop.hive.ql.metadata.HiveFatalException: [Error 20004]: Fatal error occurred when node tried to create too many dynamic partitions. The maximum number of dynamic partitions is controlled by hive.exec.max.dynamic.partitions and hive.exec.max.dynamic.partitions.pernode. Maximum was set to 100 partitions per node, number of dynamic partitions on this node: 101
    
    	at org.apache.hadoop.hive.ql.exec.FileSinkOperator.getDynOutPaths(FileSinkOperator.java:933)
    	at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:704)
    	at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879)
    	at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
    	at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879)
    	at org.apache.hadoop.hive.ql.exec.FilterOperator.process(FilterOperator.java:126)
    	at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879)
    	at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
    	at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:149)
    	at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:489)
    	... 18 more
    ]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0, Vertex vertex_1508385188096_13199_1_00 [Map 1] killed/failed due to:OWN_TASK_FAILURE]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0

Cause:

Your Hive job tried to create too many dynamic partitions on a node, exceeding the current limits. The number of dynamic partitions is controlled by the settings for`hive.exec.max.dynamic.partitions` and `hive.exec.max.dynamic.partitions.pernode`.  The default is 100 dynamic partitions on a single node, with a limit of 1000 overall.

Solution:

Set the values for `hive.exec.max.dynamic.partitions` and `hive.exec.max.dynamic.partitions.pernode` to higher values.  For example:

SET hive.exec.dynamic.partition=true;
    SET hive.exec.max.dynamic.partitions=2048;
    SET hive.exec.max.dynamic.partitions.pernode=256;

参考网址：https://blog.csdn.net/odailidong/article/details/49884571