org.apache.hive.com.esotericsoftware.kryo.kryoexception: encountered unregistered class id 错误解决办法

骑猪看日落 2022-08-08 16:58 7阅读 0赞

执行hive 任务的时候 有些任务会报下列错误 hive 0.14 + 版本才会有这个问题，

任务重做之后可能又会成功。

1.错误信息：

hdfs://nameservice1/tmp/hive/dbs/9c29873a-664f-45a4-87f5-aab6601b9661/hive\_2015-09-22\_14-51-49\_927\_648109380558598376-11/-mr-10066/129d4884-d21a-44f2-970c-454148932292/map.xml:   
org.apache.hive.com.esotericsoftware.kryo.kryoexception: encountered unregistered class id: 109  
serialization trace:  
parentoperators (org.apache.hadoop.hive.ql.exec.tablescanoperator)  
aliastowork (org.apache.hadoop.hive.ql.plan.mapwork)  
at org.apache.hadoop.hive.ql.exec.utilities.getbasework(utilities.java:450)  
at org.apache.hadoop.hive.ql.exec.utilities.getmapwork(utilities.java:296)  
at org.apache.hadoop.hive.ql.io.hiveinputformat.init(hiveinputformat.java:268)  
at org.apache.hadoop.hive.ql.io.hiveinputformat.pushprojectionsandfilters(hiveinputformat.java:484)  
at org.apache.hadoop.hive.ql.io.hiveinputformat.pushprojectionsandfilters(hiveinputformat.java:477)  
at org.apache.hadoop.hive.ql.io.combinehiveinputformat.getrecordreader(combinehiveinputformat.java:715)  
at org.apache.hadoop.mapred.maptask$trackedrecordreader.<init>(maptask.java:169)  
at org.apache.hadoop.mapred.maptask.runoldmapper(maptask.java:432)  
at org.apache.hadoop.mapred.maptask.run(maptask.java:343)  
at org.apache.hadoop.mapred.yarnchild$2.run(yarnchild.java:163)  
at java.security.accesscontroller.doprivileged(native method)  
at javax.security.auth.subject.doas(subject.java:415)  
at org.apache.hadoop.security.usergroupinformation.doas(usergroupinformation.java:1671)  
at org.apache.hadoop.mapred.yarnchild.main(yarnchild.java:158)  
caused by: org.apache.hive.com.esotericsoftware.kryo.kryoexception: encountered unregistered class id: 109  
serialization trace:  
parentoperators (org.apache.hadoop.hive.ql.exec.tablescanoperator)  
aliastowork (org.apache.hadoop.hive.ql.plan.mapwork)  
at org.apache.hive.com.esotericsoftware.kryo.util.defaultclassresolver.readclass(defaultclassresolver.java:119)  
at org.apache.hive.com.esotericsoftware.kryo.kryo.readclass(kryo.java:656)  
at org.apache.hive.com.esotericsoftware.kryo.serializers.objectfield.read(objectfield.java:99)  
at org.apache.hive.com.esotericsoftware.kryo.serializers.fieldserializer.read(fieldserializer.java:507)  
at org.apache.hive.com.esotericsoftware.kryo.kryo.readclassandobject(kryo.java:776)  
at org.apache.hive.com.esotericsoftware.kryo.serializers.mapserializer.read(mapserializer.java:139)  
at org.apache.hive.com.esotericsoftware.kryo.serializers.mapserializer.read(mapserializer.java:17)  
at org.apache.hive.com.esotericsoftware.kryo.kryo.readobject(kryo.java:694)  
at org.apache.hive.com.esotericsoftware.kryo.serializers.objectfield.read(objectfield.java:106)  
at org.apache.hive.com.esotericsoftware.kryo.serializers.fieldserializer.read(fieldserializer.java:507)  
at org.apache.hive.com.esotericsoftware.kryo.kryo.readobject(kryo.java:672)  
at org.apache.hadoop.hive.ql.exec.utilities.deserializeobjectbykryo(utilities.java:1069)  
at org.apache.hadoop.hive.ql.exec.utilities.deserializeplan(utilities.java:960)  
at org.apache.hadoop.hive.ql.exec.utilities.deserializeplan(utilities.java:974)  
at org.apache.hadoop.hive.ql.exec.utilities.getbasework(utilities.java:416)  
... 13 more  
  
ended job = job\_1442370393660\_27415 with errors  
error during job, obtaining debugging information...  
examining task id: task\_1442370393660\_27415\_m\_000004 (and more) from job job\_1442370393660\_27415  
examining task id: task\_1442370393660\_27415\_m\_000002 (and more) from job job\_1442370393660\_2741  
examining task id: task\_1442370393660\_27415\_m\_000001 (and more) from job job\_1442370393660\_27415

2.参数分析：

<name>hive.plan.serialization.format</name>

Query plan format serialization between client and task nodes.

Two supported values are : kryo and javaXML. Kryo is default.

</description>

</property>

执行计划相关的参数  默认 kryo

在客户端设置为 javaXML，如果是用CM安装的则在客户端高级选项里面添加为 javaXML 则可。

3.临时解决办法：

在经常报错的hive  前面把参数设置为javaXML  要是失败的任务比较少 则再相应的任务里面用session级别设置下就可以了。

set  hive.plan.serialization.format=javaXML;