centos7安装filebeat6.2 收集java日志

短命女 2021-12-24 01:53 416阅读 0赞

#### 安装filebeat6.2 ####

$ cd /usr/local/src
    $ wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.2.1-linux-x86_64.tar.gz
    $ tar xf filebeat-6.2.1-linux-x86_64.tar.gz
    $ cd  filebeat-6.2.1-linux-x86_64
    $ cp -r filebeat /usr/local/bin

###### 创建filebeat配置文件 ######

$ mkdir /etc/filebeat
    $ cd /etc/filebeat/
    $ vi filebeat.yml     
    path.config: /etc/filebeat
    path.logs: /var/log/filebeat
    path.config: /etc/filebeat
    path.logs: /var/log/filebeat
    path.data: /var/lib/filebeat/data
    filebeat.registry_file: /var/lib/filebeat/registry
    filebeat.shutdown_timeout: 0
    logging.level: info
    logging.metrics.enabled: false
    logging.files.rotateeverybytes: 104857600
    logging.files.keepfiles: 10
    logging.files.permissions: 0600
    
    setup.template.name: "filebeat"
    setup.template.pattern: "filebeat-*"
    filebeat.config:
        prospectors:
            enabled: true
            path: ${path.config}/prospectors.d/*.yml
            reload.enabled: true
            reload.period: 10s
    
    output.logstash:  ##提前把logstash配置好,开启负载均衡模式
        hosts: ["10.100.200.47:5044","10.100.200.66:5044"]
        index: filebeat-%{+yyyy.MM.dd}
    
        loadbalance: true

##### 编辑 各个需要收集日志项目的配置文件，这里统一放到了prospectors.d中。 #####

示例：

$ mkdir prospectors.d/
    $ cd prospectors.d/
    $ cat preproduct-htjfapp-account-provider.yml
    - type: log
      enabled: true
      paths:
          - /data/WEBLOG/preproduct-xx-xx-provider/*/*.log
      scan_frequency: 10s
      fields_under_root: true
      
      
      fields:
          
          env: preproduct
          
          index: logstash-htjf
          
          logtype: preproduct-xx-xx-provider
          
          topic: logstash-htjf
     
          vm_host: 10.100.12.125
          
          
      tail_files: false
      close_inactive: 2h
      close_eof: false
      close_removed: true
      clean_removed: true
      close_renamed: false

##### 使用superviosrd 守护进程 #####

修改supervisord.conf配置文件

[program:filebeat]
    command=/usr/local/bin/filebeat  -c /etc/filebeat/filebeat.yml
    autostart=true
    autorestart=true
    startsecs=5
    priority=1
    user=root
    stopasgroup=true
    killasgroup=true

热加载supervisord

$ supervisorctl  update
    $ supervisorctl  status
    filebeat                         RUNNING   pid 16957, uptime 18:16:15

#### logstash 的配置文件（日志格式化） ####

$ cat htjf_spring_boot.conf
    input {
        beats { 
            port => "5044"
            codec=> multiline {
            pattern => "^%{YEAR}.*"  ##不是年份开头的行都归到上一行
            negate  => true
            what    => "previous"
            }
        }
    }
    
    filter {
       grok  {
            patterns_dir => [ "/etc/logstash/patterns.d" ]
            match => [ "message", "%{TIMESTAMP_ISO8601:timestamp}\s+\[%{THREADID:threadId}\]\s+\[%{THREADNAME:traceId}\]\s+%{LOGLEVEL:level}\s+%{JAVACLASS:javaclass}\s+\-\s+%{JAVAMESSAGE:javameassage}","message", "%{TIMESTAMP_ISO8601:timestamp}\s+\[%{THREADID_1:threadId}\]\s+%{LOGLEVEL:level}\s+%{JAVACLASS:javaclass}\s+\-\s+%{JAVAMESSAGE:javameassage}"]
            remove_field => [ "message","beat","timestamp","topic","hostname","name"]
       }
       date {
            match => ["time", "dd/MMM/yyyy:HH:mm:ss Z"]
            locale => "cn"
        }
    }
    
    output {
            elasticsearch {
              hosts => ["192.168.60.117:9200","192.168.60.118:9200","192.168.60.119:9200"]
              index => "logstash-htjf-filebeat-%{+YYYY-MM}"
              user => elastic
              password => xxx
            }
           stdout { codec => rubydebug }
    }

##### 启动logstash #####

$ /usr/share/logstash/bin/logstash  -f /etc/logstash/conf.d/htjf_spring_boot.conf --path.data=/var/log/htjf_spring_boot.log & > /dev/null

可以结合计划任务(crond) 执行logstash启动的守护进程

$ cat  /etc/logstash/scripts/logstash_auto.sh
    #!/bin/bash
    htjf_spring_boot_pid=`ps -ef |grep java |grep -v  grep|grep htjf_spring_boot |awk '{print $2}'`
    if [ -z $htjf_spring_boot_pid ];then
            /usr/share/logstash/bin/logstash  -f /etc/logstash/conf.d/htjf_spring_boot.conf --path.data=/var/log/htjf_spring_boot.log & > /dev/null
    fi
    
    $ crontab -l
    #logstash auto
    */5 * * * *  /bin/sh /etc/logstash/scripts/logstash_auto.sh
    
    $ /etc/init.d/crond  restart

#### kibana 展示收集日志 ####

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI1OTM0NDAx_size_16_color_FFFFFF_t_70]  
filebeat收集日志时，新增了字段，这里是vm\_host。kibana日志展示中出现了此字段 说明成功啦。。

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI1OTM0NDAx_size_16_color_FFFFFF_t_70]: /images/20211223/c709acd441f542bb83bb357a991b2a5e.png