Hive UDTF 报错

今天药忘吃喽~ 2022-01-28 00:57 791阅读 0赞

报错：

Status: Failed  
Vertex failed, vertexName=Map 2, vertexId=vertex\_1556099599099\_3665\_1\_00, diagnostics=\[Task failed, taskId=task\_1556099599099\_3665\_1\_00\_000000, diagnostics=\[TaskAttempt 0 failed, info=\[Container container\_e08\_1556099599099\_3665\_01\_000002 finished with diagnostics set to \[Container completed. \]\], TaskAttempt 1 failed, info=\[Container container\_e08\_1556099599099\_3665\_01\_000024 finished with diagnostics set to \[Container completed. \]\], TaskAttempt 2 failed, info=\[Container container\_e08\_1556099599099\_3665\_01\_000044 finished with diagnostics set to \[Container completed. \]\], TaskAttempt 3 failed, info=\[Container container\_e08\_1556099599099\_3665\_01\_000060 finished with diagnostics set to \[Container completed. \]\]\], Vertex did not succeed due to OWN\_TASK\_FAILURE, failedTasks:1 killedTasks:0, Vertex vertex\_1556099599099\_3665\_1\_00 \[Map 2\] killed/failed due to:OWN\_TASK\_FAILURE\]  
Vertex killed, vertexName=Map 1, vertexId=vertex\_1556099599099\_3665\_1\_01, diagnostics=\[Vertex received Kill while in RUNNING state., Vertex did not succeed due to OTHER\_VERTEX\_FAILURE, failedTasks:0 killedTasks:21, Vertex vertex\_1556099599099\_3665\_1\_01 \[Map 1\] killed/failed due to:OTHER\_VERTEX\_FAILURE\]  
DAG did not succeed due to VERTEX\_FAILURE. failedVertices:1 killedVertices:1  
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 2, vertexId=vertex\_1556099599099\_3665\_1\_00, diagnostics=\[Task failed, taskId=task\_1556099599099\_3665\_1\_00\_000000, diagnostics=\[TaskAttempt 0 failed, info=\[Container container\_e08\_1556099599099\_3665\_01\_000002 finished with diagnostics set to \[Container completed. \]\], TaskAttempt 1 failed, info=\[Container container\_e08\_1556099599099\_3665\_01\_000024 finished with diagnostics set to \[Container completed. \]\], TaskAttempt 2 failed, info=\[Container container\_e08\_1556099599099\_3665\_01\_000044 finished with diagnostics set to \[Container completed. \]\], TaskAttempt 3 failed, info=\[Container container\_e08\_1556099599099\_3665\_01\_000060 finished with diagnostics

原因：

UDTF打包将Hive的依赖打到了一起，和Tez的DAG产生了冲突。

解决：

1.maven 打包，取original的包，依赖的jar如fastjson也以 add jar 形式加载到缓存中。额外一句，文件也可以add file 进去。

2.更换Hive底层引擎，使用MapReduce；

排查过程：

1.UDTF里面直接system.out；

2.查看Hive日志，在配置文件中可以找到；

3.根据applilcationId 去看一下Yarn的日志。