ELK系列(六)、修改Nginx日志为Json格式并使用Logstash导入至ES

╰半橙微兮° 2023-02-11 13:29 12阅读 0赞

前面我们介绍了使用logstash监控nginx日志，nginx日志默认是httpd格式的日志，对于开发人员来说，json格式更加友好，因此本篇我们就介绍如何修改nginx的默认日志格式，以及如何使用logstash的插件解析json格式的日志，然后写入到ES中。

[ELK系列(一)、安装ElasticSearch+Logstash+Kibana+Filebeat-v7.7.0][ELK_ElasticSearch_Logstash_Kibana_Filebeat-v7.7.0]

[ELK系列(二)、在Kibana中使用RESTful操作ES库][ELK_Kibana_RESTful_ES]

[ELK系列(三)、安装Logstash插件及打包离线安装包][ELK_Logstash]

[ELK系列(四)、Logstash读取nginx日志写入ES中][ELK_Logstash_nginx_ES]

[ELK系列(五)、Logstash修改@timestamp时间为日志的产生时间][ELK_Logstash_timestamp]

\------------------------修改nginx日志为json格式并使用logstash读取到es内-----------------

# nginx配置 #

*vim /usr/local/nginx/conf/nginx.conf*

添加如下内容

log_format log_json '{"remote_addr": "$remote_addr", '
                        '"ident": "-", '
                        '"user": "$remote_user", '
                        '"timestamp": "$time_local", '
                        '"request": "$request", '
                        '"status": $status, '
                        '"bytes": $body_bytes_sent, '
                        '"referer": "$http_referer", '
                        '"agent": "$http_user_agent", '
                        '"x_forwarded": "$http_x_forwarded_for"'
                        ' }';
        access_log  logs/access-json.log  log_json;

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx_size_16_color_FFFFFF_t_70][]

# 重启nginx #

*sbin/nginx -s reload*

可以看到在nginx日志目录新多了一个access-json.log文件。

![2020052323110446.png][]

# logstash配置 #

*vim /opt/app/logstash-7.7.0/config/logstash\_nginxlog2es\_json.conf*

input {
       # 从文件读取日志信息
       file {
            path => "/usr/local/nginx/logs/access-json.log" #修改为json日志文件
            type => "nginx_access_json"
            start_position => "beginning"
        }
     }
    
    filter {
        json {
           source => "message"
           remove_field => ["beat","offset","tags","prospector"] #移除字段，不需要采集
        }
        date {
          match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"] #匹配timestamp字段
          target => "@timestamp"  #将匹配到的数据写到@timestamp字段中
        }
    }
    
    output {
           # 输出es
           elasticsearch {
              hosts => ["wykd:9200"]
              index => "nginx-%{+YYYY.MM.dd}"
          }
    
    }

# 重启logstash #

*bin/logstash -f config/logstash\_nginxlog2es\_json.conf*

# 验证ES内的数据 #

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx_size_16_color_FFFFFF_t_70 1][]

希望本文对你有帮助，请点个赞鼓励一下作者吧~ 谢谢！

[ELK_ElasticSearch_Logstash_Kibana_Filebeat-v7.7.0]: https://blog.csdn.net/wsdc0521/article/details/106236499
[ELK_Kibana_RESTful_ES]: https://blog.csdn.net/wsdc0521/article/details/106241596
[ELK_Logstash]: https://blog.csdn.net/wsdc0521/article/details/106292639
[ELK_Logstash_nginx_ES]: https://blog.csdn.net/wsdc0521/article/details/106306584
[ELK_Logstash_timestamp]: https://blog.csdn.net/wsdc0521/article/details/106307452
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20200523230810156.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx,size_16,color_FFFFFF,t_70
[2020052323110446.png]: https://img-blog.csdnimg.cn/2020052323110446.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20200523232253708.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx,size_16,color_FFFFFF,t_70