Flume使用Hive作为Sink总结

拼搏现实的明天。 2021-06-24 15:57 718阅读 0赞

项目中打算使用Flume把数据直接传到[Hive][]表而不是HDFS上。使用[hive][Hive]作为Sink，Flume版本为1.7.0。

开始时候遇到Failed connecting to EndPoint错误，Caused by: org.apache.hive.hcatalog.streaming.StreamingException: Cannot stream to table that has not been bucketed

原来使用Hive做为Sink，Hive表必须cluster by bucket。

修改表之后重启Flume，又遇到

org.apache.flume.EventDeliveryException: [Java][].lang.ClassCastException: org.apache.[Hadoop][].hive.ql.io.HiveIgnoreKeyTextOutputFormat cannot be cast to org.apache.[hadoop][Hadoop].hive.ql.io.AcidOutputFormat 错误。

看了下源码实现AcidOutputFormat的类只有OrcOutputFormat，那么答案显而易见，Hive表需要stored as orc。

重启之后又遇到错误，

Failed connecting to EndPoint

Caused by: org.apache.hive.hcatalog.streaming.InvalidColumn: Column 'URL' not found in table for input field 81

原来在Flume配置的Hive 列名必须都为小写字母。

再重启之后成功运行。

总结如下，Hive表必须设置bucket并且 stored as orc。

Flume Hive Sink：

tier1.sinks.sink\_sitevisit.channel = channel\_sitevisit  
tier1.sinks.sink\_sitevisit.type = hive  
tier1.sinks.sink\_sitevisit.hive.metastore = thrift://localhost:9083  
tier1.sinks.sink\_sitevisit.hive.database = sitevisit  
tier1.sinks.sink\_sitevisit.hive.table = visitlog  
tier1.sinks.sink\_sitevisit.hive.partition = %Y-%m-%d  
tier1.sinks.sink\_sitevisit.hive.txnsPerBatchAsk = 10  
tier1.sinks.sink\_sitevisit.hive.batchSize = 1500  
tier1.sinks.sink\_sitevisit.serializer = DELIMITED  
tier1.sinks.sink\_sitevisit.serializer.fieldnames = visitor\_id,user\_id,session\_id,url,keyword  
tier1.sinks.sink\_sitevisit.roundUnit = hour  
tier1.sinks.sink\_sitevisit.roundValue = 1

Hive table：

create table weblogs ( id int , msg string )  
    partitioned by (continent string, country string, time string)  
    clustered by (id) into 2 buckets  
    stored as orc;

参考：http://henning.kropponline.de/2015/05/19/hivesink-for-flume/

[Hive]: http://lib.csdn.net/base/hive
[Java]: http://lib.csdn.net/base/java
[Hadoop]: http://lib.csdn.net/base/hadoop