apache flume collector 安装

我就是我 2022-08-22 03:49 159阅读 0赞

# 2、Flume Collector安装（通过extends AbstractSink implements Configurable，直接写入数据库） #

## 2.1 安装环境 ##

系统:

CentOS release 6.6

软件:

flume-collector.tar.gz

## 2.2 安装步骤 ##

### 2.2.1 部署Flume Collector ###

具体脚本（jyapp用户）：

<table style="border-collapse:collapse; border-spacing:0px; width:1843px; max-width:100%; margin-bottom:20px; color:rgb(51,51,51); font-family:'Helvetica Neue',Helvetica,Arial,sans-serif; font-size:14px; line-height:20px"> 
 <tbody style=""> 
 <tr style=""> 
 <td style="padding:8px; line-height:1.42857143; vertical-align:top; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221)"> 
 <div style=""> 
 cd &nbsp;/home/jyapp 
 tar &nbsp;-zxvf &nbsp;flume-collector.tar.gz 
 <div style="height:1em"></div> 
 cd &nbsp;flume-collector 
 &nbsp; 
 #修改flume环境变量 
 vi &nbsp;conf/flume-env.sh&nbsp;&nbsp;&nbsp; 
 具体配置解析：（注意：红色需修改） 
 ------------------------------------------------------ 
 export JAVA_OPTS="-Xms1000m -Xmx1000m &nbsp;-Dflume.root.logger=INFO,LOGFILE -Dflume.monitoring.type=http &nbsp;-Dflume.monitoring.port=34545 &nbsp;-Dcom.sun.management.jmxremote -Djava.rmi.server.hostname=本机IP&nbsp;&nbsp;-Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.authenticate=false &nbsp;-Dcom.sun.management.jmxremote.ssl=false" 
 ------------------------------------------------------------------------------------- 
 &nbsp; 
 &nbsp; 
 #修改Collector脚本 
 vi &nbsp;conf/collector.conf&nbsp;&nbsp; 
 具体配置解析：（注意：红色需修改） 
 ------------------------------------------------------ 
 collector.sources = r1 
 collector.channels = c1 c2 
 collector.sinks = k1 k2 k3 k4 k5 k6 
 &nbsp; 
 &nbsp; 
 collector.sources.r1.type = avro 
 collector.sources.r1.bind =&nbsp;收集端IP &nbsp;&nbsp;#本机IP 
 collector.sources.r1.port =&nbsp;收集端绑定端口 &nbsp;#本机接口消息端口 
 &nbsp; 
 collector.channels.c1.type = memory 
 collector.channels.c1.capacity = 10000 
 collector.channels.c1.transactionCapacity = 10000 
 collector.channels.c1.byteCapacityBufferPercentage = 30 
 collector.channels.c1.byteCapacity = 1048576000 
 &nbsp; 
 collector.channels.c2.type = memory 
 collector.channels.c2.capacity = 10000 
 collector.channels.c2.transactionCapacity = 10000 
 collector.channels.c2.byteCapacityBufferPercentage = 30 
 collector.channels.c2.byteCapacity = 1048576000 
 &nbsp; 
 collector.sinks.k1.type = com.jy.modules.platform.flume.sink.OracleJdbcSink 
 collector.sinks.k1.url =&nbsp;jdbc:oracle:thin:@IP:PORT:数据库名 
 collector.sinks.k1.user =用户名 
 collector.sinks.k1.password =密码 
 &nbsp; 
 collector.sinks.k2.type = com.jy.modules.platform.flume.sink.OracleJdbcSink 
 collector.sinks.k2.url =&nbsp;jdbc:oracle:thin:@IP:PORT:数据库名 
 collector.sinks.k2.user =用户名 
 collector.sinks.k2.password =密码 
 &nbsp; 
 collector.sinks.k3.type = com.jy.modules.platform.flume.sink.OracleJdbcSink 
 collector.sinks.k3.url =&nbsp;jdbc:oracle:thin:@IP:PORT:数据库名 
 collector.sinks.k3.user =用户名 
 collector.sinks.k3.password =密码 
 &nbsp; 
 collector.sinks.k4.type = com.jy.modules.platform.flume.sink.OracleJdbcSink 
 collector.sinks.k4.url =&nbsp;jdbc:oracle:thin:@IP:PORT:数据库名 
 collector.sinks.k4.user =用户名 
 collector.sinks.k4.password =密码 
 &nbsp; 
 collector.sinks.k5.type = com.jy.modules.platform.flume.sink.OracleJdbcSink 
 collector.sinks.k5.url =&nbsp;jdbc:oracle:thin:@IP:PORT:数据库名 
 collector.sinks.k5.user =用户名 
 collector.sinks.k5.password =密码 
 &nbsp; 
 collector.sinks.k6.type = com.jy.modules.platform.flume.sink.OracleJdbcSink 
 collector.sinks.k6.url =&nbsp;jdbc:oracle:thin:@IP:PORT:数据库名 
 collector.sinks.k6.user =用户名 
 collector.sinks.k6.password =密码 
 &nbsp; 
 collector.sources.r1.channels = c1 c2 
 collector.sinks.k1.channel = c1 
 collector.sinks.k2.channel = c1 
 collector.sinks.k3.channel = c1 
 collector.sinks.k4.channel = c2 
 collector.sinks.k5.channel = c2 
 collector.sinks.k6.channel = c2 
 &nbsp; 
 collector.sources.r1.selector.type = multiplexing 
 collector.sources.r1.selector.header = flume.client.system.flag 
 collector.sources.r1.selector.mapping.S000 = c1 
 collector.sources.r1.selector.mapping.S001 = c2 
 collector.sources.r1.selector.mapping.S002 = c1 
 collector.sources.r1.selector.mapping.S003 = c2 
 collector.sources.r1.selector.mapping.S004 = c1 
 collector.sources.r1.selector.mapping.S005 = c2 
 collector.sources.r1.selector.mapping.S006 = c1 
 collector.sources.r1.selector.mapping.S007 = c2 
 collector.sources.r1.selector.mapping.S008 = c1 
 collector.sources.r1.selector.mapping.S009 = c2 
 collector.sources.r1.selector.default = c1 
 &nbsp; 
 collector.sinkgroups = g1 g2 
 collector.sinkgroups.g1.sinks = k1 k2 k3 
 collector.sinkgroups.g1.processor.type = load_balance 
 collector.sinkgroups.g1.processor.backoff = true 
 collector.sinkgroups.g1.processor.selector = round_robin 
 &nbsp; 
 collector.sinkgroups.g2.sinks = k4 k5 k6 
 collector.sinkgroups.g2.processor.type = load_balance 
 collector.sinkgroups.g2.processor.backoff = true 
 collector.sinkgroups.g2.processor.selector = round_robin 
 ------------------------------------------------------------------------------------- 
 &nbsp; 
 后台启动collector 
 cd &nbsp;/home/jyapp/flume-collector 
 bin/flume-ng &nbsp;agent &nbsp;--conf conf &nbsp;--conf-file &nbsp;conf/collector.conf &nbsp;--name &nbsp;collector &nbsp;&nbsp;&amp; 
 &nbsp; 
 停止agent，使用kill命令 
 &nbsp; 
 </div> </td> 
 </tr> 
 </tbody> 
</table>