ELK 7.5.0（十）ELK+Redis+Filebeat

分手后的思念是犯贱 2023-06-20 06:07 6阅读 0赞

## 一、实验环境 ##

主机名        IP
    es        192.168.14.210
    kibana    192.168.14.210
    logstash  192.168.14.211
    Redis     192.168.14.212
    Filebeat  192.168.14.213
    nginx     192.168.14.213
    secure    192.168.14.213

## **Filebeat - -> Redis - ->Logstash - ->elasticsearch - ->kibana** ##

## 二、安装部署（内容比较多，已经分多个章节） ##

1、ELK（elasticsearch+logstash+kibana）

具体查看第二、三、四章节

2、Filebeat安装

具体查看第六章节

## 三、安装Redis ##

1、官网下载

[root@redis ~]# wget http://download.redis.io/releases/redis-5.0.7.tar.gz

2、安装依赖环境

[root@redis ~]# yum install -y wget net-tools gcc gcc-c++ make tar openssl openssl-devel cmake

3、解压redis

[root@redis ~]# tar -zxvf redis-5.0.7.tar.gz  -C /usr/local/

4、编译安装

[root@redis ~]# cd /usr/local/redis-5.0.7/
    [root@redis redis-5.0.7]# make

5、（可选）根据需求拷贝启动程序和配置文件到指定目录

[root@redis redis-5.0.7]# mkdir -pv /usr/local/redis/conf /usr/local/redis/bin
    [root@redis redis-5.0.7]# cp src/redis* /usr/local/redis/bin/
    [root@redis redis-5.0.7]# cp redis.conf /usr/local/redis/conf

6、更改redis配置（redis默认是前端启动）

[root@redis ~]# vi /usr/local/redis/conf/redis.conf
    daemonize yes            #后台启动
    requirepass elkpwd       #密码

7、前端页面启动

[root@redis ~]# /usr/local/redis/bin/redis-server  /usr/local/redis/conf/redis.conf

## 四、Filebeat采集日志发送到redis ##

1、修改配置文件

[root@filebeat ~]# vi /usr/local/filebeat-7.5.0/filebeat.yml
    filebeat.inputs:
    - type: log
      enabled: true
      tail_files: true
      backoff: "1s"
      paths:
          - /usr/local/nginx/logs/access.json.log
      fields:
        filetype: nginx_access
      fields_under_root: true
    
    - type: log
      enabled: true
      tail_files: true
      backoff: "1s"
      paths:
          - /var/log/secure
      fields:
        filetype: linux_secure
      fields_under_root: true
    
    
    output:
      redis:
        hosts: ["192.168.14.212:6379"]     #redisIP地址
        password: 'elkpwd'                 #redis登录密码
        key: 'filebeat'                    #任意定义一个字段给logstash获取

2、重启服务（发现filebeat后台启动会自动退出，待解决问题）

[root@filebeat ~]# pkill filebeat
    [root@filebeat ~]# nohup filebeat -e -c /usr/local/filebeat-7.5.0/filebeat.yml > /tmp/filebeat.log 2>&1 &

## 五、logstash从redis中读取数据 ##

1、修改配置文件

[root@logstash ~]# vi /usr/local/logstash-7.5.0/config/logstash.conf 
    input {
        redis {
          host => '192.168.14.212'        #redis服务器IP
          port => 6379                    #redis端口
          key => "filebeat"               #filebeat自定义的值
          data_type => "list"             #以列表方式存储数据
          password => "elkpwd"            #redis登录密码
      }
    }
    
    filter {
      if [filetype] == "nginx_access" {
        json {
          source => "message"
          remove_field => ["message","@version","path","input","log","agent","ecs","tags"]
        }
      }
    }
    
    output{
      if [filetype] == "nginx_access" {
        elasticsearch {
          hosts => ["http://192.168.14.210:9200"]
          user => "elastic"
          password => "elkpwd"
          index => "nginx_access-%{+YYYY.MM.dd}"
        }
      }
      else if [filetype] == "linux_secure" {
        elasticsearch {
          hosts => ["http://192.168.14.210:9200"]
          user => "elastic"
          password => "elkpwd"
          index => "linux_secure-%{+YYYY.MM.dd}"
        }
      }
    }

2、重启服务

[root@logstash ~]# nohup logstash -f /usr/local/logstash-7.5.0/config/logstash.conf > /tmp/logstash.log 2> /tmp/logstash.log &

## 六、Kibana查看日志 ##

1、查看日志索引

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv_size_16_color_FFFFFF_t_70][]

2、访问nginx网页产生新的日志，查看日志信息

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv_size_16_color_FFFFFF_t_70 1][]

3、登录filebeat系统主机，产生日志信息

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv_size_16_color_FFFFFF_t_70 2][]

至此，Filebeat（多台） - -> Redis - -> Logstash（正则过滤）- -> Elasticsearch（存储） - ->Kibana（显示）的架构部署完成

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20191208171152521.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20191208171301397.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/20191208171335371.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv,size_16,color_FFFFFF,t_70