filebeat直连elasticsearch利用pipeline提取message中的字段

客官°小女子只卖身不卖艺 2022-01-14 11:15 453阅读 0赞

[2019独角兽企业重金招聘Python工程师标准>>> ][2019_Python_] ![hot3.png][]

这里使用filebeat直连elasticsearch的形式完成数据传输，由于没有logstash，所有对于原始数据的过滤略显尴尬（logstash的filter非常强大）。 但是由于业务需求，还是需要将message（原始数据）中的某些字段进行提取，具体方式如下：

##### 1. /path/目录下建立pipeline.json文件 #####

{
      "description" : "test-pipeline",
      "processors" : [
        {
          "grok" :{
            "field" : "message",
            "patterns" : ["%{DATA:puid}\\\t%{DATA:datatime}\\\t\\\t%{DATA:content}"]
          }
        }
      ]
    }

##### 2. 将规则上传至elasticsearch中 #####

curl -H "Content-Type: application/json" -XPUT 'http://localhost:9200/_ingest/pipeline/test-pipeline' -d@/path/pipeline.json

##### 3. filebeat.yml中 #####

filebeat.prospectors:
     ******
     ******
    output.elasticsearch:
      hosts: ["localhost:9200"]
      # 加入如下行：
      pipeline: "test-pipeline"

##### 4. 测试数据 #####

f1b25095cc823e63389ff299622b7e85    2019/02/27 03:38:54     send packet! opcode:3 message is in lua8282
    f1b25095cc823e63389ff299622b7e85    2019/02/27 03:38:54     PacketManager::_onReceivedPacket opcode:3 size:27,rec_len:278282

##### 5. elasticsearch中数据结果 #####

[
        {
            "_score":1,
            "_type":"doc",
            "_id":"zWmLj2kB7ah0Pw2MmQGw",
            "_source":{
                "datatime":"2019/02/27 03:38:54",
                "log":{
                    "file":{
                        "path":"/path/test_1.log"
                    }
                },
                "beat":{
                    "hostname":":",
                    "name":":",
                    "version":"6.6.1"
                },
                "@timestamp":"2019-03-18T06:44:43.224Z",
                "host":{
                    "name":":"
                },
                "content":"",
                "source":"/path/test_1.log",
                "puid":"f1b25095cc823e63389ff299622b7e85",
                "offset":0,
                "input":{
                    "type":"log"
                },
                "message":"f1b25095cc823e63389ff299622b7e85 2019/02/27 03:38:54 send packet! opcode:3 message is in lua",
                "prospector":{
                    "type":"log"
                }
            },
            "_index":"test"
        },
        {
            "_score":1,
            "_type":"doc",
            "_id":"0GmLj2kB7ah0Pw2MmQGw",
            "_source":{
                "datatime":"2019/02/27 03:38:54",
                "log":{
                    "file":{
                        "path":"/path/test_1.log"
                    }
                },
                "beat":{
                    "hostname":":",
                    "name":":",
                    "version":"6.6.1"
                },
                "@timestamp":"2019-03-18T06:44:43.224Z",
                "host":{
                    "name":":"
                },
                "content":"",
                "source":"/path/test_1.log",
                "puid":"f1b25095cc823e63389ff299622b7e85",
                "offset":318,
                "input":{
                    "type":"log"
                },
                "message":"f1b25095cc823e63389ff299622b7e85 2019/02/27 03:38:54 PacketManager::_onReceivedPacket| ReceivedPacket size:27",
                "prospector":{
                    "type":"log"
                }
            },
            "_index":"test"
        }
    ]

##### 参考： #####

1.  [https://note.yuchaoshui.com/blog/post/yuziyue/filebeat-use-ingest-node-dealwith-log-then-load-into-elasticsearch][https_note.yuchaoshui.com_blog_post_yuziyue_filebeat-use-ingest-node-dealwith-log-then-load-into-elasticsearch]
2.  [http://www.axiaoxin.com/article/236/][http_www.axiaoxin.com_article_236]
3.  [https://blog.csdn.net/spring\_ming/article/details/62232331][https_blog.csdn.net_spring_ming_article_details_62232331]

\--Posted from Rpc

转载于:https://my.oschina.net/remainsu/blog/3024074

[2019_Python_]: https://my.oschina.net/u/2663968/blog/3061697
[hot3.png]: /images/20220114/9bf006eedc0a43c6a2088d3f748a5d65.png
[https_note.yuchaoshui.com_blog_post_yuziyue_filebeat-use-ingest-node-dealwith-log-then-load-into-elasticsearch]: https://note.yuchaoshui.com/blog/post/yuziyue/filebeat-use-ingest-node-dealwith-log-then-load-into-elasticsearch
[http_www.axiaoxin.com_article_236]: http://www.axiaoxin.com/article/236/
[https_blog.csdn.net_spring_ming_article_details_62232331]: https://blog.csdn.net/spring_ming/article/details/62232331