ELK系列(五)、Logstash修改@timestamp时间为日志的产生时间

ゞ浴缸里的玫瑰 2023-02-11 13:29 15阅读 0赞

上一篇讲了如何使用Logstash监控nginx日志并打印到控制台或导入到ES中，在kibana的discover中我们可以根据@timestamp时间对数据进行过滤和排序，但这里显示的@timestamp时间是指logstash读取到日志的时间而不是日志真正产生的时间，本篇就介绍如何配置使logstash在采集日志的过程中使用日志的真实时间戳作为消息体的时间戳，便于我们之后的分析

[ELK系列(一)、安装ElasticSearch+Logstash+Kibana+Filebeat-v7.7.0][ELK_ElasticSearch_Logstash_Kibana_Filebeat-v7.7.0]

[ELK系列(二)、在Kibana中使用RESTful操作ES库][ELK_Kibana_RESTful_ES]

[ELK系列(三)、安装Logstash插件及打包离线安装包][ELK_Logstash]

[ELK系列(四)、Logstash读取nginx日志写入ES中][ELK_Logstash_nginx_ES]

\-----------------------------------修改logstash消息体中的@timestamp为日志产生时间--------------

# 介绍 #

之前读取nginx日志的消息体如下，可以看到这里的@timestamp和message中的时间戳不一致，而timestamp和message中的时间是一致的，那么我们可以修改@timestamp指定为timestamp。

{
               "verb" => "GET",
           "@version" => "1",
           "referrer" => "\"-\"",
               "host" => "node01",
               "type" => "system",
          "timestamp" => "23/May/2020:21:00:19 +0800",
              "agent" => "\"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36\"",
        "httpversion" => "1.1",
               "path" => "/usr/local/nginx/logs/access.log",
         "@timestamp" => 2020-05-23T13:56:05.419Z,
           "clientip" => "192.168.142.1",
              "ident" => "-",
            "message" => "192.168.142.1 - - [23/May/2020:21:00:19 +0800] \"GET / HTTP/1.1\" 200 612 \"-\" \"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36\"",
           "response" => "200",
              "bytes" => "612",
            "request" => "/",
               "auth" => "-"
    }

# 配置 #

配置如下，只需要修改filter的部分即可：

*vim config/logstash\_nginxlog2es\_timestamp.conf*

input {
       # 从文件读取日志信息
       file {
            path => "/usr/local/nginx/logs/access.log"
            type => "nginx_access"
            start_position => "beginning"
        }
     }
    
    filter {
        grok {
          match => { "message" => "%{HTTPD_COMBINEDLOG}" }
        }
        date {
          match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"] #匹配timestamp字段
          target => "@timestamp"  #将匹配到的数据写到@timestamp字段中
        }
    }
    
    output {
           # 输出es
           elasticsearch {
              hosts => ["wykd:9200"]
              index => "nginx-%{+YYYY.MM.dd}"
          }
    
    }

# 验证 #

先删除之前在es中的索引以及sincedb文件，然后重新启动logstash：

*bin/logstash -f config/logstash\_nginxlog2es\_timestamp.conf*

在kibana中查看此时@timestamp已经是日志中的timestamp，该日志真正的产生时间，而不是logstash的采集时间了。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx_size_16_color_FFFFFF_t_70][]

希望本文对你有帮助，请点个赞鼓励一下作者吧~ 谢谢！

[ELK_ElasticSearch_Logstash_Kibana_Filebeat-v7.7.0]: https://blog.csdn.net/wsdc0521/article/details/106236499
[ELK_Kibana_RESTful_ES]: https://blog.csdn.net/wsdc0521/article/details/106241596
[ELK_Logstash]: https://blog.csdn.net/wsdc0521/article/details/106292639
[ELK_Logstash_nginx_ES]: https://blog.csdn.net/wsdc0521/article/details/106306584
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20200523224852996.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx,size_16,color_FFFFFF,t_70