轻松入门大数据第12章零基础：极速入门日志收集系统Flume 12.1 日志收集框架Flume 12.1.6 【重要必掌握】Flume核心组件

﹏ヽ暗。殇╰゛Y 2024-03-16 13:04 17阅读 0赞

### 轻松入门大数据 ###

#### 文章目录 ####

*   *  轻松入门大数据
     *   *  第12章 零基础：极速入门日志收集系统Flume
         *   *  12.1 日志收集框架Flume
             *   *  12.1.6 【重要必掌握】Flume核心组件

![在这里插入图片描述][0399f0e7f006462f82c22454f472e6f1.png_pic_center]

#### 第12章 零基础：极速入门日志收集系统Flume ####

##### 12.1 日志收集框架Flume #####

> 1.  产生背景
> 2.  初识Flume
> 3.  竞品分析
> 4.  发展史
> 5.  Agent核心组件
> 6.  部署及使用

###### 12.1.6 【重要必掌握】Flume核心组件 ######

【三大核心组件】

> 当然只是我们先了解三个，其实不止3 个

官网架构图：[https://flume.apache.org/][https_flume.apache.org]

![在这里插入图片描述][85fa7bb6887c4c77acb7c68735118b66.png_pic_center]

![在这里插入图片描述][a31dfae04c7a4f43a5dd6119992eb590.png_pic_center]

**收集：Source**

*  从哪儿收集数据？？？
 *  职责：负责接收数据到Flume
    
    [https://flume.apache.org/releases/content/1.11.0/FlumeUserGuide.html\#flume-sources][https_flume.apache.org_releases_content_1.11.0_FlumeUserGuide.html_flume-sources]
    
    ![在这里插入图片描述][b6e1eccd96a24b77a9339952486dddd1.png_pic_center]
 *  Avro Source
 *  Exec Source
 *  Spooling Directory Source
 *  Taildir Source \*\*\*\*\*
 *  Kafka Source

**聚合：Channel**

*  通道，是一个临时存储的地方
 *  Channel是介于Source和Sink之间的缓冲区
    
    [https://flume.apache.org/releases/content/1.11.0/FlumeUserGuide.html\#flume-sinks][https_flume.apache.org_releases_content_1.11.0_FlumeUserGuide.html_flume-sinks]
    
    ![在这里插入图片描述][1416c4480662433cba4c7f394bc4b60c.png_pic_center]
 *  Memory Channel
 *  Kafka Channel
 *  File Channel
 *  负责数据聚合/暂存，Event在Channel不会呆太长时间的，很快会被Sink消费掉的

**移动：Sink**

*  读取Channel的数据，然后推送到各种不同的目的地去
    
    [https://flume.apache.org/releases/content/1.11.0/FlumeUserGuide.html\#flume-sinks][https_flume.apache.org_releases_content_1.11.0_FlumeUserGuide.html_flume-sinks]
    
    ![在这里插入图片描述][864f3772253d4526a8b2b869a2ca73b3.png_pic_center]
 *  HDFS Sink
 *  Hive Sink
 *  Logger Sink
 *  Avro Sink
 *  Kafka Sink
 *  ES/HBase Sink
 *  负责数据转移存储

最后可以看到，架构图中用了一个框：

![在这里插入图片描述][ad1cfa6341b6454f80351519152155d3.png_pic_center]

Agent：Agent中包含了最主要的三大组件：Source Channel Sink

[0399f0e7f006462f82c22454f472e6f1.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/16/4e3dc761bfe34b44809e7c8f67b54e83.png
[https_flume.apache.org]: https://flume.apache.org/
[85fa7bb6887c4c77acb7c68735118b66.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/16/ed81116ce340465aaeec2b926bd4b1e4.png
[a31dfae04c7a4f43a5dd6119992eb590.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/16/90eee406265743879aa035d78937df9f.png
[https_flume.apache.org_releases_content_1.11.0_FlumeUserGuide.html_flume-sources]: https://flume.apache.org/releases/content/1.11.0/FlumeUserGuide.html#flume-sources
[b6e1eccd96a24b77a9339952486dddd1.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/16/2496d11b548e4ceaa784158ebf88daa6.png
[https_flume.apache.org_releases_content_1.11.0_FlumeUserGuide.html_flume-sinks]: https://flume.apache.org/releases/content/1.11.0/FlumeUserGuide.html#flume-sinks
[1416c4480662433cba4c7f394bc4b60c.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/16/baf056d093df4c6995be030d78085ede.png
[864f3772253d4526a8b2b869a2ca73b3.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/16/06f26d1bc17248ce8a85d991f6db6edf.png
[ad1cfa6341b6454f80351519152155d3.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/16/ffed0173879b4208ac828895c6bd85c1.png