日志系统系列flume-ng+kafka+storm+HDFS搭建实验

╰半橙微兮° 2022-08-10 12:51 47阅读 0赞

最近公司要求搞个日志系统，以前接触不多，最近看了好多资料什么flume flume-ng kafka storm hdfs es 等等，感觉其实要做个小的直接kafka或者flume-ng就可以(现在已经代替了flume)，看了N多大神的文章比如这个[《实时系统搭建》][Link 1]就很不错，但是对于我这种新手来说不一定合适，所以自己就准备边搞边记录了！

首先我的技术选型就是flume-ng kafka  storm hdfs 其中每个技术我都会详细写出步骤，省的到时候自己忘记了，原型就是根据[Jonathan-Wei][]大神的文章来的，上边已经说了就不再重复！

好了下边就来搞搞flume-ng，大家应该都知道与flume的不同之处吧！不知道的小伙伴也不要紧请看转载的文章flume的革命好了，下面我们就来看下flume的搭建

flume-ng的东西在整理下

===================================================================================================================================

hadoop hbase 单机版搭建

后面需要安装集群，整合到hadoop，所以注意选择与hadoop对应的版本：

**Table 2.1. Hadoop version support matrix**

<table>       
 <thead> 
  <tr> 
   <th align="left" style="line-height:1.2">&nbsp;</th> 
   <th align="center" style="line-height:1.2">HBase-0.92.x</th> 
   <th align="center" style="line-height:1.2">HBase-0.94.x</th> 
   <th align="center" style="line-height:1.2">HBase-0.96</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td align="left" style="line-height:1.2">Hadoop-0.20.205</td> 
   <td align="center" style="line-height:1.2">S</td> 
   <td align="center" style="line-height:1.2">X</td> 
   <td align="center" style="line-height:1.2">X</td> 
  </tr> 
  <tr> 
   <td align="left" style="line-height:1.2">Hadoop-0.22.x</td> 
   <td align="center" style="line-height:1.2">S</td> 
   <td align="center" style="line-height:1.2">X</td> 
   <td align="center" style="line-height:1.2">X</td> 
  </tr> 
  <tr> 
   <td align="left" style="line-height:1.2">Hadoop-1.0.x</td> 
   <td align="center" style="line-height:1.2">S</td> 
   <td align="center" style="line-height:1.2">S</td> 
   <td align="center" style="line-height:1.2">S</td> 
  </tr> 
  <tr> 
   <td align="left" style="line-height:1.2">Hadoop-1.1.x</td> 
   <td align="center" style="line-height:1.2">NT</td> 
   <td align="center" style="line-height:1.2">S</td> 
   <td align="center" style="line-height:1.2">S</td> 
  </tr> 
  <tr> 
   <td align="left" style="line-height:1.2">Hadoop-0.23.x</td> 
   <td align="center" style="line-height:1.2">X</td> 
   <td align="center" style="line-height:1.2">S</td> 
   <td align="center" style="line-height:1.2">NT</td> 
  </tr> 
  <tr> 
   <td align="left" style="line-height:1.2">Hadoop-2.x</td> 
   <td align="center" style="line-height:1.2">X</td> 
   <td align="center" style="line-height:1.2">S</td> 
   <td align="center" style="line-height:1.2">S</td> 
  </tr> 
 </tbody> 
</table>

<table style="color:rgb(51,51,51); font-size:14px; line-height:26px; font-family:Simsun"> 
 <tbody> 
  <tr> 
   <td style="line-height:1.2">S = supported and tested,支持</td> 
  </tr> 
  <tr> 
   <td style="line-height:1.2">X = not supported,不支持</td> 
  </tr> 
  <tr> 
   <td style="line-height:1.2">NT = not tested enough.可以运行但测试不充分</td> 
  </tr> 
 </tbody> 
</table>

由于 HBase 依赖 Hadoop，它配套发布了一个Hadoop jar 文件在它的 lib 下。该套装jar仅用于独立模式。在分布式模式下，Hadoop版本必须和HBase下的版本一致。用你运行的分布式Hadoop版本jar文件替换HBase lib目录下的Hadoop jar文件，以避免版本不匹配问题。确认替换了集群中所有HBase下的jar文件。Hadoop版本不匹配问题有不同表现，但看起来都像挂掉了。

**安装：**

$ tar xfz hbase-0.90.4.tar.gz  
$ cd hbase-0.90.4

我自己使用的是独立的zookeeper 后边连接的是hdfs存储，所以有如下配置

hbase.rootdir 必须和 hadoop的core-sit.xml里面的地址一致

编辑配置文件hbase-site.xml

vi hbase-site.xml
       <configuration> 
         <property> 
           <name>hbase.rootdir</name> 
           <value>hdfs://server01:9000/hbase</value> 
           <description>区域服务器使用存储HBase数据库数据的目录</description> 
         </property> 
         <property> 
           <name>hbase.cluster.distributed</name> 
           <value>true</value> 
           <description>
       指定HBase运行的模式：
       false: 单机模式或者为分布式模式                   
                        true: 全分布模式  
           </description> 
         </property> 
         <property> 
           <name>hbase.zookeeper.quorum</name> 
           <value>server01,server02,server03</value> 
           <description>ZooKeeper集群服务器的位置</description> 
         </property> 
       </configuration>

设置环境变量hbase-env.sh

vi hbase-evn.sh  
       export HBASE_MANAGES_ZK=false

使用独立的ZooKeeper时需要修改HBASE\_MANAGES\_ZK值为false，为不使用默认ZooKeeper实例。

我的环境变量

==============================================================================

\# hadoop  
HADOOP\_HOME=/home/mq/hadoop/hadoop  
\# hbase  
HBASE\_HOME=/home/mq/hbase/hbase-0.98.7  
\# flume  
export FLUME\_HOME=/home/mq/flume/flume  
\# sbt  
SBT=/home/mq/kafka/sbt  
\# kafka  
KAFKA=/home/mq/kafka/kafka  
JAVA\_HOME=/usr/java/jdk1.7.0\_60  
CLASS\_PATH=$JAVA\_HOME/lib:$JAVA\_HOME/jre/bin:$JAVA\_HOME/lib/tools.jar:$CLASS\_PATH:$HBASE\_HOME/bin

=============================================================================================================================

单机启动抛出这样一个错误

[mq@localhost hbase-0.98.7]$ ./bin/start-hbase.sh 
    2014-10-23 10:46:27,658 FATAL [main] conf.Configuration: error parsing conf file: com.sun.org.apache.xerces.internal.impl.io.MalformedByteSequenceException: 1 字节的 UTF-8 序列的字节 1 无效。
    Exception in thread "main" java.lang.RuntimeException: com.sun.org.apache.xerces.internal.impl.io.MalformedByteSequenceException: 1 字节的 UTF-8 序列的字节 1 无效。
            at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1243)
            at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1117)
            at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1053)
            at org.apache.hadoop.conf.Configuration.get(Configuration.java:397)
            at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:594)
            at org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfiguration.java:69)
            at org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfiguration.java:105)
            at org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:116)
            at org.apache.hadoop.hbase.util.HBaseConfTool.main(HBaseConfTool.java:39)
    Caused by: com.sun.org.apache.xerces.internal.impl.io.MalformedByteSequenceException: 1 字节的 UTF-8 序列的字节 1 无效。
            at com.sun.org.apache.xerces.internal.impl.io.UTF8Reader.invalidByte(UTF8Reader.java:687)
            at com.sun.org.apache.xerces.internal.impl.io.UTF8Reader.read(UTF8Reader.java:557)
            at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.load(XMLEntityScanner.java:1753)
            at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.skipChar(XMLEntityScanner.java:1426)
            at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2807)
            at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
            at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:117)
            at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
            at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
            at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
            at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
            at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:243)
            at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:347)
            at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:177)
            at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1156)
            ... 8 more
    starting master, logging to /home/mq/hbase/hbase-0.98.7/bin/../logs/hbase-mq-master-localhost.localdomain.out
    localhost: +======================================================================+
    localhost: |      Error: JAVA_HOME is not set and Java could not be found         |
    localhost: +----------------------------------------------------------------------+
    localhost: | Please download the latest Sun JDK from the Sun Java web site        |
    localhost: |       > http://java.sun.com/javase/downloads/ <                      |
    localhost: |                                                                      |
    localhost: | HBase requires Java 1.6 or later.                                    |
    localhost: | NOTE: This script will find Sun Java whether you install using the   |
    localhost: |       binary or the RPM based installer.                             |
    localhost: +======================================================================+

我这很费解，看样子应该是环境变量问题修改修改conf文件下的hbase-env.sh文件中的java\_home的环境变量

修改完以后错误变化

[mq@localhost hbase-0.98.7]$ ./bin/start-hbase.sh 
    2014-10-23 10:59:39,411 FATAL [main] conf.Configuration: error parsing conf file: com.sun.org.apache.xerces.internal.impl.io.MalformedByteSequenceException: 1 字节的 UTF-8 序列的字节 1 无效。
    Exception in thread "main" java.lang.RuntimeException: com.sun.org.apache.xerces.internal.impl.io.MalformedByteSequenceException: 1 字节的 UTF-8 序列的字节 1 无效。
            at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1243)
            at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1117)
            at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1053)
            at org.apache.hadoop.conf.Configuration.get(Configuration.java:397)
            at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:594)
            at org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfiguration.java:69)
            at org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfiguration.java:105)
            at org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:116)
            at org.apache.hadoop.hbase.util.HBaseConfTool.main(HBaseConfTool.java:39)
    Caused by: com.sun.org.apache.xerces.internal.impl.io.MalformedByteSequenceException: 1 字节的 UTF-8 序列的字节 1 无效。
            at com.sun.org.apache.xerces.internal.impl.io.UTF8Reader.invalidByte(UTF8Reader.java:687)
            at com.sun.org.apache.xerces.internal.impl.io.UTF8Reader.read(UTF8Reader.java:557)
            at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.load(XMLEntityScanner.java:1753)
            at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.skipChar(XMLEntityScanner.java:1426)
            at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2807)
            at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
            at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:117)
            at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
            at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
            at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
            at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
            at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:243)
            at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:347)
            at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:177)
            at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1156)
            ... 8 more
    starting master, logging to /home/mq/hbase/hbase-0.98.7/bin/../logs/hbase-mq-master-localhost.localdomain.out
    localhost: starting regionserver, logging to /home/mq/hbase/hbase-0.98.7/bin/../logs/hbase-mq-regionserver-localhost.localdomain.out
    localhost: Exception in thread "main" java.lang.RuntimeException: com.sun.org.apache.xerces.internal.impl.io.MalformedByteSequenceException: 1 字节的 UTF-8 序列的字节 1 无效。
    localhost:      at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1243)
    localhost:      at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1117)
    localhost:      at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1053)
    localhost:      at org.apache.hadoop.conf.Configuration.get(Configuration.java:397)
    localhost:      at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:594)
    localhost:      at org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfiguration.java:69)
    localhost:      at org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfiguration.java:105)
    localhost:      at org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:116)
    localhost:      at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2484)

</pre><pre name="code" class="html" style="color: rgb(51, 51, 51); font-family: Arial; font-size: 14px; line-height: 26px;">这个是编码问题，代表你有的配置文件不是utf-8的，仔细检查下吧

还有一些问题比如报警告

<h3 class="t" style="margin: 0px 0px 1px; padding: 0px; list-style: none; font-weight: normal;font-size:14px; line-height: 1.54; color: rgb(51, 51, 51); font-family: arial;"><a target=_blank data-click="{'F':'378717EA','F1':'9D73F1E4','F2':'4CA6DE6B','F3':'54E5243F','T':'1414034754','y':'BF6CF7B5' }" href="http://www.baidu.com/link?url=o9-rxdk2Vum7yUDsx_F4fRvvG_VAwahcWTbiEX1l3f-SFwRNFGTSTCbwvFASirhz9bMWR1SQLuPu_IFXNM8hSa" target="_blank"><span style="color: rgb(204, 0, 0);">WARN</span> <span style="color: rgb(204, 0, 0);">util.NativeCodeLoader:</span> <span style="color: rgb(204, 0, 0);">Unable</span> <span style="color: rgb(204, 0, 0);">to</span> <span style="color: rgb(204, 0, 0);">load</span> <span style="color: rgb(204, 0, 0);">native-hadoop</span> ...</a></h3><span style="word-wrap: normal; word-break: normal; line-height: 27px; color: rgb(50, 62, 50); font-size: 13.5pt; font-family: 微软雅黑, sans-serif; background-color: rgb(145, 157, 163);">Apache</span><span style="word-wrap: normal; word-break: normal; line-height: 27px; color: rgb(50, 62, 50); font-size: 13.5pt; font-family: 微软雅黑, sans-serif; background-color: rgb(145, 157, 163);">提供的<span style="word-wrap: normal; word-break: normal;">hadoop</span>本地库是<span style="word-wrap: normal; word-break: normal;">32</span>位的，而在<span style="word-wrap: normal; word-break: normal;">64</span>位的服务器上就会有问题，因此需要自己编译<span style="word-wrap: normal; word-break: normal;">64</span>位的版本</span>

启动服务 bin/stop-hbase.sh

<pre name="code" class="plain">starting master, logging to /home/mq/hbase/hbase-0.98.7/bin/../logs/hbase-mq-master-localhost.localdomain.out
    SLF4J: Class path contains multiple SLF4J bindings.
    SLF4J: Found binding in [jar:file:/home/mq/hbase/hbase-0.98.7/lib/slf4j-log4j12-1.6.4.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: Found binding in [jar:file:/home/mq/hadoop/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

我去掉slf4j-log4j12-1.5.8.jar就解决了问题，没有报错。具体还有待查找

下面我们来验证下是否正确了

发现不对劲，居然没反应报错了，看了下日志

java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster
            at org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:140)
            at org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:202)
            at org.apache.hadoop.hbase.LocalHBaseCluster.<init>(LocalHBaseCluster.java:152)
            at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:179)
            at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:135)
            at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
            at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
            at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2786)
    Caused by: org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot communicate with client version 4
            at org.apache.hadoop.ipc.Client.call(Client.java:1113)
            at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
            at com.sun.proxy.$Proxy8.getProtocolVersion(Unknown Source)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:606)
            at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
            at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
            at com.sun.proxy.$Proxy8.getProtocolVersion(Unknown Source)
            at org.apache.hadoop.ipc.RPC.checkVersion(RPC.java:422)
            at org.apache.hadoop.hdfs.DFSClient.createNamenode(DFSClient.java:183)
            at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:281)
            at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:245)
            at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:100)
            at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1446)
            at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:67)
            at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1464)
            at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:263)
            at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)
            at org.apache.hadoop.hbase.util.FSUtils.getRootDir(FSUtils.java:926)
            at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:460)
            at org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster.<init>(HMasterCommandLine.java:258)
            at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
            at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
            at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
            at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
            at org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:137)
            ... 7 more

无语的问题啊，继续查吧看来下，貌似hadoop和hbase版本不合的问题，突然发型hadoop2.5.1没有core，无语了啊

发现最后还是版本问题，我勒个去啊具体版本可以参考http://www.aboutyun.com/blog-61-62.html

换了个同样的版本的2就可以了，然后zookeeper又不能连接2222端口，本来服务器已经有了个zookeeper想用同一个或者再来一个

编辑hbase-site.xml 加入地址（都在hosts配置好了）和端口2222

<property>  
   <name>hbase.zookeeper.quorum</name>  
   <value>hd204,hd205,hd206</value>  
 </property>

在编辑regionservers加入映射

hd204  
hd205  
hd206

OK 启动hbase

http://192.168.1.116:60010/master-status\#

查看hbase状态，简单的配置就完成了

后期会把hadoop+flume-ng的东西补上留作参考，骑士hbase还是不全，有时间在整理了

[Link 1]: http://blog.csdn.net/weijonathan/article/details/18301321
[Jonathan-Wei]: http://my.csdn.net/WeiJonathan