Linux云计算架构-搭建基于filebeat和redis高速缓存的ELK架构

桃扇骨 2023-01-08 14:25 128阅读 0赞

### 文章目录 ###

*  Linux云计算架构-搭建基于filebeat和redis高速缓存的ELK架构
 *   *  1. filebeat介绍
     *  2. 基于filebeat收集系统日志
     *  3. 搭建基于redis高速缓存的ELK架构
     *  4. ELK架构总结

# Linux云计算架构-搭建基于filebeat和redis高速缓存的ELK架构 #

## 1. filebeat介绍 ##

**Beats** 是 elastic 公司开源的一款采集系统监控数据的代理agent，是在被监控服务器上以客户端形式运行的数据收集器的统称。**可以直接把数据发送给 Elasticsearch或者通过 Logstash 发送给Elasticsearch**，然后进行后续的日志分析。

Beats由packetbeat、`filebeat`、metricbeat、winlogbeat组成，其中**filebeat主要用于监控、收集服务器日志文件。**

## 2. 基于filebeat收集系统日志 ##

**实验拓扑图：**  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70]  
**①下载filebeat软件包**

**下载地址**：  
`https://elasticsearch.cn/download/`  
`https://www.elastic.co/cn/downloads/`  
`https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.10.2-x86_64.rpm`

**②检查ELK集群是否安装好**

# master节点-elasticsearch
    [root@master ~]# netstat -lntp | grep java
    tcp6       0      0 192.168.80.136:9200     :::*                    LISTEN      21107/java          
    tcp6       0      0 192.168.80.136:9300     :::*                    LISTEN      21107/java         
    # master节点-kibana
    [root@master ~]# netstat -lntp | grep 5601
    tcp        0      0 192.168.80.136:5601     0.0.0.0:*               LISTEN      8941/node    
    
    # node节点-elasticsearch和logstash
    [root@node ~]# netstat -lntp | grep java
    tcp6       0      0 192.168.80.133:9200     :::*                    LISTEN      20954/java          
    tcp6       0      0 :::10514                :::*                    LISTEN      8694/java           
    tcp6       0      0 192.168.80.133:9300     :::*                    LISTEN      20954/java          
    tcp6       0      0 192.168.80.133:9600     :::*                    LISTEN      8694/java

**访问**：`http://192.168.80.136:5601/app/home#/`  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 1]  
**③安装并启动filebeat**

[root@node ~]# ll filebeat-7.10.2-x86_64.rpm 
    -rw-r--r-- 1 root root 33798676 1月  20 14:45 filebeat-7.10.2-x86_64.rpm
    [root@node ~]# rpm -ivh filebeat-7.10.2-x86_64.rpm 
    [root@node ~]# vim /etc/filebeat/filebeat.yml 
     21 - type: log
     22 
     23   # Change to true to enable this input configuration.
     24   # enabled: false # 注释掉，不注释的话filebeat不会收集本地日志
     25 
     26   # Paths that should be crawled and fetched. Glob based paths.
     27   paths:  # 需要收集的日志的绝对路径
     28     - /var/log/messages
     29     - /var/log/secure
     
    177 output.elasticsearch:
    178   # Array of hosts to connect to.
    179   hosts: ["192.168.80.136:9200"]
    
    [root@node ~]# systemctl stop logstash.service 
    [root@node ~]# systemctl status logstash.service 
    
    [root@node ~]# systemctl start filebeat.service && systemctl status filebeat
    [root@node ~]# ps aux | grep filebeat
    root      23348 11.5  4.1 1375396 159124 ?      Ssl  15:20   0:05 /usr/share/filebeat/bin/filebeat --environment systemd -c /etc/filebeat/filebeat.yml --path.home /usr/share/filebeat --path.config /etc/filebeat --path.data /var/lib/filebeat --path.logs /var/log/filebeat
    root      23449  0.0  0.0 112724   988 pts/0    S+   15:21   0:00 grep --color=auto filebeat

**④在es服务器上查看filebeat的日志索引**

[root@master ~]# curl '192.168.80.136:9200/_cat/indices?v' | grep filebeat
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100  1290  100  1290    0     0  87588      0 --:--:-- --:--:-- --:--:-- 92142
    green  open   filebeat-7.10.2-2021.01.20-000001 XnXVOxU3R1C5pSJwQVaD5w   1   1       4970            0      1.8mb        808.7kb

**⑤在kibana上配置索引模式，可视化filebeat的日志**

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 2]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 3]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 4]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 5]

## 3. 搭建基于redis高速缓存的ELK架构 ##

**配置大致流程：**  
**①在node节点上，使用filebeat收集nginx的日志**

[root@node ~]# yum install -y nginx
    [root@node ~]# systemctl start nginx && systemctl enable nginx && systemctl status nginx
    [root@node ~]# netstat -antup | grep nginx
    tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      32251/nginx: master 
    tcp6       0      0 :::80                   :::*                    LISTEN      32251/nginx: master

**nginx的默认首页文件：**  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 6]

# 查看filebeat支持的模块，可以看到filebeat是支持收集很多种日志的
    [root@node ~]# filebeat modules enable nginx
    Enabled nginx
    
    [root@node ~]# filebeat modules list
    Enabled:
    nginx
    
    Disabled:
    activemq
    apache
    auditd
    aws
    azure
    barracuda
    bluecoat
    cef
    checkpoint
    cisco
    citrix
    coredns
    crowdstrike
    cyberark
    cylance
    elasticsearch
    envoyproxy
    f5
    fortinet
    googlecloud
    gsuite
    haproxy
    ibmmq
    icinga
    iis
    imperva
    infoblox
    iptables
    juniper
    kafka
    kibana
    logstash
    microsoft
    misp
    mongodb
    mssql
    mysql
    nats
    netflow
    netscout
    o365
    okta
    osquery
    panw
    postgresql
    proofpoint
    rabbitmq
    radware
    redis
    santa
    snort
    sonicwall
    sophos
    squid
    suricata
    symantec
    system
    tomcat
    traefik
    zeek
    zoom
    zscaler

**修改filebeat下的nginx配置文件：**

# 收集访问日志和错误日志，其实最好是仅收集一种日志
    [root@node ~]# vim /etc/filebeat/modules.d/nginx.yml 
      4 - module: nginx
      5   # Access logs
      6   access:
      7     enabled: true
      8     var.paths:["/var/log/nginx/access.log"]
      9     # Set custom paths for the log files. If left empty,
     10     # Filebeat will choose the paths depending on your OS.
     11     #var.paths:
     12 
     13   # Error logs
     14   error:
     15     enabled: true
     16     var.paths:["/var/log/nginx/error.log"]
    
    [root@node ~]# vim /etc/filebeat/filebeat.yml 
     97 filebeat.config.modules:
     98   # Glob pattern for configuration loading
     99   path: ${path.config}/modules.d/*.yml
    100 
    101   # Set to true to enable config reloading
    102   reload.enabled: true
    
    [root@node ~]# systemctl restart filebeat

**②在master节点上部署redis缓存filebeat收集的日志**

[root@master ~]# yum install redis -y
    [root@master ~]# vim /etc/redis.conf
      61 bind 0.0.0.0
      80 protected-mode no
    [root@master ~]# systemctl start redis && systemctl enable redis
    [root@master ~]# netstat -antup | grep redis
    tcp        0      0 0.0.0.0:6379            0.0.0.0:*               LISTEN      24416/redis-server

**③在node节点上，配置filebeat收集nginx日志并缓存到master节点上**

# 注释原有发送地址，配置redis服务器为日志发送地址
    [root@node ~]# vim /etc/filebeat/filebeat.yml
    177 # output.elasticsearch:
    178   # Array of hosts to connect to.
    179   # hosts: ["192.168.80.136:9200"]
    
    180   output.redis:
    181     hosts: ["192.168.80.136:6379"]
    182     key: "elk_redis"
    [root@node ~]# systemctl restart filebeat

**④在node节点上配置logstash，从redis中获取日志存入es服务器**

[root@node ~]# vim /etc/logstash/conf.d/redis.conf
    input{ 
      redis{       # logstash从redis获取日志
        host => "192.168.80.136"
        port => "6379"
        key => "elk_redis"
        data_type => "list"
      }
    }
    filter{ 
    }
    output{ 
      elasticsearch{     # logstash发送日志到es
        hosts => "192.168.80.136:9200"
        index => "redis-nginx-%{+YYYY.MM.dd}"
      }
    }
    [root@node ~]# systemctl restart logstash
    # 等待一会，看看logstash有没有获取到日志，可以看到是有的
    [root@node ~]# curl '192.168.80.136:9200/_cat/indices?v' | grep redis-nginx
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100  1548  100  1548    0     0  97665      0 --:--:-- --:--:-- --:--:--  100k
    green  open   redis-nginx-2021.01.26            Zv-zmbWtTwaXsqKkr03wIQ   1   1          5            0     75.5kb         32.9kb

**⑤配置kibana展示nginx日志**  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 7]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 8]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 9]

# 在node节点上重启下nginx服务，生成日志
    [root@node ~]# systemctl restart nginx.service

可以看到，**有相关日志被filebeat收集，并缓存到redis,由logstash从redis上获取，并传输给es,最后由kibana做展示。**  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 10]

## 4. ELK架构总结 ##

①`logstash`和`filebeat`都可以获取本地日志，并发送给`es`，最后通过`kibana`进行展示。在一台服务器上，不能同时启动logstash和filebeat来收集同一种日志。

②本文上述架构原理可以理解为：**先使用filebeat获取本地日志，并将本地日志存储在redis服务器上，之后通过logstash从redis服务器上获取并发送给es，最终由kibana展示日志数据。**

③**基于redis缓存服务的ELK架构**的优点为：**当某一应用服务器突然有大量的日志，这时若由logstash获取日志并发送给es，会导致logstash卡死，即logstash无法采集大量的日志。而若由filebeat获取日志并存储到redis中，即使应用服务器突然产生大量日志，都会被缓存到redis中，而logstash就可以有条不紊的从redis中获取数据并发送给es，而不至于卡死。**

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70]: /images/20221119/435c008e55584ec5a3ac9bd9b584c459.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 1]: /images/20221119/58495fac5cb344ecb66438e1302b9b1e.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 2]: /images/20221119/8188a833b91d429aa1d6615c07bef832.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 3]: /images/20221119/9ed0140fbdb14fca979353af58f27772.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 4]: /images/20221119/ae7e3ba72b3d4c07a2a600c5519b1878.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 5]: /images/20221119/3299276e6fbc4c2c8401e96766a57125.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 6]: /images/20221119/7ddfbc350a314c8d875badf86023a36a.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 7]: /images/20221119/9d9efb1c6d954d2d973cf24c8f287db6.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 8]: /images/20221119/a658af14265342c4bfcbc85f4a46f6c6.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 9]: /images/20221119/757720526968402d97569d4d0815ef88.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 10]: /images/20221119/26206cf191934b20912afb27fe35e9b6.png