轻松入门大数据第13章进阶实战：如何基于Flume实战日志数据收集 13.1 日志收集框架Flume进阶 13.1.10 【重要必掌握】实战之SinkProcessor

Love The Way You Lie 2024-03-16 15:38 17阅读 0赞

### 轻松入门大数据 ###

#### 文章目录 ####

*   *  轻松入门大数据
     *   *  第13章 进阶实战：如何基于Flume实战日志数据收集
         *   *  13.1 日志收集框架Flume进阶
             *   *  13.1.10 【重要必掌握】实战之SinkProcessor

![在这里插入图片描述][80b26e3da1f743b8844f8b81c87d5ae4.png_pic_center]

#### 第13章 进阶实战：如何基于Flume实战日志数据收集 ####

##### 13.1 日志收集框架Flume进阶 #####

> 1.  文件数据收集到HDFS
> 2.  文件夹数据收集到HDFS
> 3.  **TAILDIR方式收集**
> 4.  **avrosink配合avrosource使用**
> 5.  **Channel Selector实战**
> 6.  **Sink Processor实战**

###### 13.1.10 【重要必掌握】实战之SinkProcessor ######

还是先来个场景：

通过netcat 接收数据 → 到channel 后，后面有两个sink，sink1 和 sink2，并且为它们设置不同的优先级

我们来搞三个Agent：

*  agent1：netcat memory avro【而且这个写出必须写到两个不同的端口，因为我们就这一个机器】
 *  agent2：avro memory logger
 *  agent3：avro memory logger

配置文件① ：`failoversink.conf`

agent1.sources = r1
    agent1.sinks = k1 k2
    agent1.channels = c1
    
    agent1.sinkgroups = g1
    agent1.sinkgroups.g1.sinks = k1 k2
    agent1.sinkgroups.g1.processor.type = failover
    agent1.sinkgroups.g1.processor.priority.k1 = 15
    agent1.sinkgroups.g1.processor.priority.k2 = 10
    agent1.sinkgroups.g1.processor.maxpenalty = 10000
    
    agent1.sources.r1.type = netcat
    agent1.sources.r1.bind = 0.0.0.0
    agent1.sources.r1.port = 55555
    
    agent1.channels.c1.type = memory
    
    agent1.sinks.k1.type = avro
    agent1.sinks.k1.hostname = 0.0.0.0
    agent1.sinks.k1.port = 55551
    
    agent1.sinks.k2.type = avro
    agent1.sinks.k2.hostname = 0.0.0.0
    agent1.sinks.k2.port = 55552
    
    agent1.sources.r1.channels = c1
    agent1.sinks.k1.channel = c1
    agent1.sinks.k2.channel = c1

![在这里插入图片描述][01b462ccbd0c43a5b80d24e8c4a2e5c8.png_pic_center]

配置文件② ：`failoversink-1.conf`

collector1.sources = r1
    collector1.sinks = k1 k2
    collector1.channels = c1
    
    collector1.sources.r1.type = avro
    collector1.sources.r1.bind = 0.0.0.0
    collector1.sources.r1.port = 55551
    
    collector1.channels.c1.type = memory
    
    collector1.sinks.k1.type = logger
    
    collector1.sources.r1.channels = c1
    collector1.sinks.k1.channel = c1

![在这里插入图片描述][75b0d84d39d441f2b9f09f20fe359f94.png_pic_center]

配置文件③ ：`failoversink-2.conf`

collector2.sources = r1
    collector2.sinks = k1 k2
    collector2.channels = c1
    
    collector2.sources.r1.type = avro
    collector2.sources.r1.bind = 0.0.0.0
    collector2.sources.r1.port = 55552
    
    collector2.channels.c1.type = memory
    
    collector2.sinks.k1.type = logger
    
    collector2.sources.r1.channels = c1
    collector2.sinks.k1.channel = c1

![在这里插入图片描述][cb35e2110fd14cf0a40008e23621e4b1.png_pic_center]

三个启动脚本：【还是先启下游】

flume-ng agent \
    --name collector1 \
    --conf $FLUME_HOME/conf \
    --conf-file /home/hadoop/config/failoversink-1.conf \
    -Dflume.root.logger=INFO,console
    
    
    flume-ng agent \
    --name collector2 \
    --conf $FLUME_HOME/conf \
    --conf-file /home/hadoop/config/failoversink-2.conf \
    -Dflume.root.logger=INFO,console
    
    
    flume-ng agent \
    --name agent1 \
    --conf $FLUME_HOME/conf \
    --conf-file /home/hadoop/config/failoversink.conf \
    -Dflume.root.logger=INFO,console

![在这里插入图片描述][baad9ac6b0b84932b24bf676e7baac2e.png_pic_center]

![在这里插入图片描述][2396017b999e408cb8da73fdf7ea6fac.png_pic_center]

![在这里插入图片描述][c0771228402743e7a8a5c7be98e7d289.png_pic_center]

OK， 连55555 端口

`telnet localhost 55555`

![在这里插入图片描述][4752f3c47b72487e9635e00da5ed8d6e.png_pic_center]

发送数据【我们配的是 55551 的优先级更高一点】

![在这里插入图片描述][7b92b14fb5f74cc6a87293f5fc056221.png_pic_center]

![在这里插入图片描述][2fab4815c13f43748a4c69c987ea1851.png_pic_center]

没问题。

直接把这个沙雕

![在这里插入图片描述][328bdf7775334f73ab46c191d8adcfe7.png_pic_center]

再发一次

![在这里插入图片描述][9fdd7d0104dd4f0e9f20ea4557827061.png_pic_center]

看看55552 那里

![在这里插入图片描述][ccaad2063c4849bf8e4a81676772b3eb.png_pic_center]

没问题，因为55551 已经挂了，所以走55552

[80b26e3da1f743b8844f8b81c87d5ae4.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/16/e3bbeba997f14814992f94adb8eed87d.png
[01b462ccbd0c43a5b80d24e8c4a2e5c8.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/16/7468ae54c4b14f8586e193f3813584b3.png
[75b0d84d39d441f2b9f09f20fe359f94.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/16/48da4525b94547ec859d23cea59af0ae.png
[cb35e2110fd14cf0a40008e23621e4b1.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/16/13fe581e0b1f46b88d9d4ff881798c47.png
[baad9ac6b0b84932b24bf676e7baac2e.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/16/26e3d46ceb834740b0f0302ba49dc0cc.png
[2396017b999e408cb8da73fdf7ea6fac.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/16/c5a0c0e9d77645ed8d29361b03aa1e96.png
[c0771228402743e7a8a5c7be98e7d289.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/16/ae736af8d44a47fd855a8829c61dd828.png
[4752f3c47b72487e9635e00da5ed8d6e.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/16/faacbbea64e34cc58336d91aca9cc4e3.png
[7b92b14fb5f74cc6a87293f5fc056221.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/16/8b1c2db3a57b4a88a73fceb413f96b67.png
[2fab4815c13f43748a4c69c987ea1851.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/16/e667a01d97404c7997724e869e42aa1a.png
[328bdf7775334f73ab46c191d8adcfe7.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/16/d9ef6dd3fcd6487f94bc293fe9d208fa.png
[9fdd7d0104dd4f0e9f20ea4557827061.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/16/082cfb6aed4d47b09863019ef5d44184.png
[ccaad2063c4849bf8e4a81676772b3eb.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/16/3954acf96b424707bc322e1232e02633.png