flume+Hbase搭建问题详解

ゝ一纸荒年。 2022-08-13 12:41 283阅读 0赞

近期在做日志系统，所以选配了一些方案，在搭建时候遇到了问题，所以记录下

flume-ng去连接hbase

# Describe  the sink
    a1.sinks.k1.type  = org.apache.flume.sink.hbase.AsyncHBaseSink
    a1.sinks.k1.table = Router #设置hbase的表名
    a1.sinks.k1.columnFamily = log #设置hbase中的columnFamily
    a1.sinks.k1.serializer.payloadColumn=serviceTime,browerOS,clientTime,screenHeight,screenWidth,url,userAgent,mobileDevice,gwId,mac # 设置hbase的column
    a1.sinks.k1.serializer = org.apache.flume.sink.hbase.BaimiAsyncHbaseEventSerializer # 设置serializer的处理类

重点说明几个属性 a1.sinks.k1.serializer.payloadColumn 中列出了所有的列名。a1.sinks.k1.serializer设置了flume serializer的处理类。BaimiAsyncHbaseEventSerializer类中会获取payloadColumn的内容，将它以逗号分隔，从而得出所有的列名。

因为flume-ng一般只支持简单的存入hbase

由于依赖的版本问题。此处需要将flume的lib [文件夹][Link 1] 下的protobuf用Hadoop-2.2.0中的2.5.0版本替换，还需要用hadoop-2.2.0中的guava替换flume的lib文件夹下的guava，删除原来相应的jar文件。启动即可生效。（PS：这是网上的说法，其实在用到最新的版本根本不需要这么搞）

在搭建hbase时候发成了这样的问题

java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/HBaseConfiguration
            at org.apache.flume.sink.hbase.HBaseSink.<init>(HBaseSink.java:116)
            at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
            at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
            at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
            at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
            at java.lang.Class.newInstance(Class.java:374)
            at org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:43)
            at org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:415)
            at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:103)
            at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
            at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
            at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
            at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            at java.lang.Thread.run(Thread.java:745)
    Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.HBaseConfiguration
            at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
            at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
            at java.security.AccessController.doPrivileged(Native Method)
            at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
            at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
            at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
            at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
            ... 17 more

真无语 腰包hbase hbase-common-0.98.7-hadoop2.jar 考到flume里面

follows.

java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/client/HTable

at org.apache.flume.sink.hbase.HBaseSink$1.run(HBaseSink.java:148)

at org.apache.flume.sink.hbase.HBaseSink$1.run(HBaseSink.java:145)

at org.apache.flume.sink.hbase.HBaseSink.runPrivileged(HBaseSink.java:427)

at org.apache.flume.sink.hbase.HBaseSink.start(HBaseSink.java:145)

at org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)

at org.apache.flume.SinkRunner.start(SinkRunner.java:79)

at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745)

Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.client.HTable

at java.net.URLClassLoader$1.run(URLClassLoader.java:366)

at java.net.URLClassLoader$1.run(URLClassLoader.java:355)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(URLClassLoader.java:354)

at java.lang.ClassLoader.loadClass(ClassLoader.java:425)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)

at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

... 14 more

又TM出现了这个，同解，考包吧hbase-client-0.98.7-hadoop2.jar这个

java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/protobuf/generated/MasterProtos$MasterService$BlockingInterface

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:190)

at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:399)

at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:388)

at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:269)

at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:198)

at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:160)

at org.apache.flume.sink.hbase.HBaseSink$1.run(HBaseSink.java:148)

at org.apache.flume.sink.hbase.HBaseSink$1.run(HBaseSink.java:145)

at org.apache.flume.sink.hbase.HBaseSink.runPrivileged(HBaseSink.java:427)

at org.apache.flume.sink.hbase.HBaseSink.start(HBaseSink.java:145)

at org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)

at org.apache.flume.SinkRunner.start(SinkRunner.java:79)

at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745)

Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingInterface

at java.net.URLClassLoader$1.run(URLClassLoader.java:366)

at java.net.URLClassLoader$1.run(URLClassLoader.java:355)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(URLClassLoader.java:354)

at java.lang.ClassLoader.loadClass(ClassLoader.java:425)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)

at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

... 21 more

依然缺少jar  hbase-protocol-0.98.7-hadoop2.jar

org.apache.flume.FlumeException: Could not load table, test from HBase

at org.apache.flume.sink.hbase.HBaseSink.start(HBaseSink.java:159)

at org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)

at org.apache.flume.SinkRunner.start(SinkRunner.java:79)

at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745)

Caused by: java.io.IOException: java.lang.reflect.InvocationTargetException

at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:411)