ELK 分布式日志收集分析系统记录

雨点打透心脏的1/2处 2023-03-13 12:23 45阅读 0赞

ELK =  elasticsreach + logstash + kibana + beats

也称为 ： ELK Stack, 是因为加入了  beats.

![20200514093031240.png][]

elasticsreach : 分布式全文检索引擎。

![20200514093055339.png][]

logstash:  日志的收集，过滤， 分析。

Logstash 是服务器端数据处理管道，能够同时从多个来源采集数据，转换数据，然后将数据发送到诸如 Elasticsearch 等“存储库”中

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3Rhbmdfamlhbl9kb25n_size_16_color_FFFFFF_t_70][]

kibana: 可视化界面操作工具。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3Rhbmdfamlhbl9kb25n_size_16_color_FFFFFF_t_70 1][]

beats:  组件包

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3Rhbmdfamlhbl9kb25n_size_16_color_FFFFFF_t_70 2][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3Rhbmdfamlhbl9kb25n_size_16_color_FFFFFF_t_70 3][]

首先需要装好  es + kibana环境

这是我前一篇文章搭建es+kibana.

[https://blog.csdn.net/tang\_jian\_dong/article/details/104446526][https_blog.csdn.net_tang_jian_dong_article_details_104446526]

ELK 原理图：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3Rhbmdfamlhbl9kb25n_size_16_color_FFFFFF_t_70 4][]

下载 logstash： 对应es版本。 或者直接yum安装， 或者docker安装在官网都有说明的。

[https://www.elastic.co/cn/downloads/past-releases/logstash-7-6-0][https_www.elastic.co_cn_downloads_past-releases_logstash-7-6-0]

下载 beats:

nohup  ./filebeat -e -c filebeat.yml -d "publish" > filebeat.log &

[https://www.elastic.co/cn/downloads/beats][https_www.elastic.co_cn_downloads_beats]

上传至目录后解压： tar -zxvf logstash-7.6.0.tar.gz

上传至目录后解压： tar -zxvf  filebeat-7.7.0-linux-x86\_64.tar.gz

![202005141436025.png][]

1: 先配置filebeat， 可以读取单个日志文件， 也可以读取目录

input: 日志文件或目录路径

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3Rhbmdfamlhbl9kb25n_size_16_color_FFFFFF_t_70 5][]

output:  输出到 elasticsearch 或者 logstash,  我是输出到logstash

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3Rhbmdfamlhbl9kb25n_size_16_color_FFFFFF_t_70 6][]

启动： filebeat , 将日志输入到 filebeat.log中

nohup ./filebeat -e -c filebeat.yml -d "publish" > filebeat.log &

2： 配置 logstash

/software/es/logstash-7.6.0/config

新增： first-pipeline.conf， 加入如下内容

input {
        beats {
            #数据来自 filebeat
            port => "5044"
            codec => "json"
            id => "filebeat"
    
        }
    }
     filter {
        #grok过滤器插件，可以将非结构化日志数据解析为结构化和可查询的内容
        grok {
            match => { "message" => "%{COMBINEDAPACHELOG}"}
        }
        #geoip插件查找IP地址，从地址中获取地理位置信息，然后将该位置信息添加到日志中
        geoip {
            source => "clientip"
        }
    }
    output {
         stdout { codec => rubydebug }
        #数据输出到es保存
        elasticsearch {
            hosts => [ "IP:9200" ]
            index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
        }
    }

启动：进入 logstash 的bin目录 ： /software/es/logstash-7.6.0/bin

执行如下命令启动：

nohup ./logstash -f ../config/first-pipeline.conf  > logstash.log &

启动成功： successfully

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3Rhbmdfamlhbl9kb25n_size_16_color_FFFFFF_t_70 7][]

logstash 显示读取的日志：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3Rhbmdfamlhbl9kb25n_size_16_color_FFFFFF_t_70 8][]

kibana 查看logstash传输的数据：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3Rhbmdfamlhbl9kb25n_size_16_color_FFFFFF_t_70 9][]

[20200514093031240.png]: /images/20230312/f6e16b3da15048478605768062ee517b.png
[20200514093055339.png]: /images/20230312/33ddda6aa5b44154a37b3900d6fd2453.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3Rhbmdfamlhbl9kb25n_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20200514093117591.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3Rhbmdfamlhbl9kb25n,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3Rhbmdfamlhbl9kb25n_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20200514093106648.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3Rhbmdfamlhbl9kb25n,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3Rhbmdfamlhbl9kb25n_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/20200514093345132.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3Rhbmdfamlhbl9kb25n,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3Rhbmdfamlhbl9kb25n_size_16_color_FFFFFF_t_70 3]: https://img-blog.csdnimg.cn/20200514093741350.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3Rhbmdfamlhbl9kb25n,size_16,color_FFFFFF,t_70
[https_blog.csdn.net_tang_jian_dong_article_details_104446526]: https://blog.csdn.net/tang_jian_dong/article/details/104446526
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3Rhbmdfamlhbl9kb25n_size_16_color_FFFFFF_t_70 4]: https://img-blog.csdnimg.cn/20200513174052152.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3Rhbmdfamlhbl9kb25n,size_16,color_FFFFFF,t_70
[https_www.elastic.co_cn_downloads_past-releases_logstash-7-6-0]: https://www.elastic.co/cn/downloads/past-releases/logstash-7-6-0
[https_www.elastic.co_cn_downloads_beats]: https://www.elastic.co/cn/downloads/beats
[202005141436025.png]: https://img-blog.csdnimg.cn/202005141436025.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3Rhbmdfamlhbl9kb25n_size_16_color_FFFFFF_t_70 5]: https://img-blog.csdnimg.cn/20200514143743212.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3Rhbmdfamlhbl9kb25n,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3Rhbmdfamlhbl9kb25n_size_16_color_FFFFFF_t_70 6]: https://img-blog.csdnimg.cn/20200514143837514.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3Rhbmdfamlhbl9kb25n,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3Rhbmdfamlhbl9kb25n_size_16_color_FFFFFF_t_70 7]: https://img-blog.csdnimg.cn/20200514152212640.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3Rhbmdfamlhbl9kb25n,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3Rhbmdfamlhbl9kb25n_size_16_color_FFFFFF_t_70 8]: https://img-blog.csdnimg.cn/20200515161844160.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3Rhbmdfamlhbl9kb25n,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3Rhbmdfamlhbl9kb25n_size_16_color_FFFFFF_t_70 9]: https://img-blog.csdnimg.cn/20200515200034868.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3Rhbmdfamlhbl9kb25n,size_16,color_FFFFFF,t_70