分布式环境中如何使用聚合日志系统ELK

雨点打透心脏的1/2处 2022-11-14 14:59 152阅读 0赞

## ELK简介 ##

ELK日志系统相信大家都不陌生了，如果你的系统是集群有多个实例，那么去后台看日志肯定不方便，因为前台访问时随机路由到后台app的，所以需要一个聚合的日志查询系统。ELK就是一个不错的选择。

ELK简单说的是：Elasticsearch + Logstash + Kibana。Logstash用于分析日志，获取想要的日志格式；Elasticsearch用于给日志创建索引；Kibana用于展现日志。

![图片][2594e17ced13b93730492a21ebf787bf.png]

这里我们还要增加一个采集软件：FileBeat，用于采集各app的日志。

系统机构图如下：

![20210330142217761.gif][]

## ELK配置 ##

安装ELK的环境需要安装JDK，这里我会说一些简单的配置，详细的安装大家可以自行上网搜索。有兴趣的童鞋可以看看这篇文章：

> “
> 
> https://blog.51cto.com/54dev/2570811?source=dra
> 
> ”

当我们解压了Logstash软件后，我们需要修改`startup.options`文件配置：

`cd logstash/config/startup.options`

LS_HOME=/home/elk/support/logstash
    LS_SETTINGS_DIR="${LS_HOME}/config"
    
    LS_OPTS="--path.settings ${LS_SETTINGS_DIR}"
    LS_JAVA_OPTS=""
    
    LS_PIDFILE=${LS_HOME}/logs/logstash.pid
    
    LS_USER=elk
    LS_GROUP=elk
    
    LS_GC_LOG_FILE=${LS_HOME}/logs/logstash-gc.log

我们还要给Logstash配置beats的端口，还有es的ip地址：

`cd logstash/config/logstash-aicrm-with-filebeat.yml`

input {
    beats {
        port => 5044
        }
    }
    
    output {
        elasticsearch {
        hosts => "192.168.205.129:9200"
        index => "aicrm-app-node"
        }
    }

对于Filebeat，我们需要配置日志格式以及Logstash的地址：

`cd filebeat/filebeat.yml`

filebeat:
    prospectors:
    -
    paths:
    - /home/elk/logs/xxx*.log
    type: log
    mutiline.pattern: '^\['
    mutiline.negate: true
    mutiline.match: after
    
    output:
    logstash:
    hosts: ["192.168.205.129:5044"]

最后我们在Kibana里面配置es的地址：

`cd kibana/config/kibana.yml`

...
    7 server.host: "0.0.0.0"
    8
    ...
    21 elasticsearch.url: "http://192.168.205.129:9200";
    22

配置好后，启动ELK

启动顺序为：elasticsearch ➡ logstash ➡ filebeat ➡ kibana

启动后的进程：

![图片][801a6b316561432a3a2548e130ef5e3d.png]

我们浏览器端访问：

http://192.168.205.129:5601/app/kibana

![20210330142217759.gif][]

这里我们需要创建ES索引

![20210330142217757.gif][]

之后我们就可以搜索日志了：

![图片][63b0e6f558cc0e56ea1f715c34a1a26d.png]

## 关于日志解析 ##

根据业务情况，会出现ELK解析多种格式的日志需求，这时需要在Logstash的配置文件中配置grok规则解析日志文件，grok解析建议使用在线工具测试。

在线Grok解析工具地址：https://grokdebug.herokuapp.com/?\#

注意，这个解析地址需要FQ才能访问。

解析样例：

在线测试样例：

![图片][f042d8b7d8ffce7cb3b70c6950a57899.png]

Grok的语句需要写在ELK的Logstash中的配置文件中，如下图：

![20210330142217767.gif][]

### 异常日志 ###

2018-11-09 23:01:18.766  [ERROR]  com.xxx.rpc.server.handler.ServerHandler - 调用com.xxx.search.server.SearchServer.search时发生错误！
    java.lang.reflect.InvocationTargetException
     at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
     at java.lang.reflect.Method.invoke(Method.java:597)

### grok解析 ###

%{TIMESTAMP_ISO8601:log_time} [%{DATA:log_level}] %{GREEDYDATA:message}

### 解析结果 ###

{
    "log_time": [
      [
        "2018-11-09 23:01:18.766"
      ]
    ],
    "YEAR": [
      [
        "2018"
      ]
    ],
    "MONTHNUM": [
      [
        "11"
      ]
    ],
    "MONTHDAY": [
      [
        "09"
      ]
    ],
    "HOUR": [
      [
        "23",
        null
      ]
    ],
    "MINUTE": [
      [
        "01",
        null
      ]
    ],
    "SECOND": [
      [
        "18.766"
      ]
    ],
    "ISO8601_TIMEZONE": [
      [
        null
      ]
    ],
    "log_level": [
      [
        "ERROR"
      ]
    ],
    "message": [
      [
        " com.xxx.rpc.server.handler.ServerHandler - 调用com.xxx.search.server.SearchServer.search时发生错误！"
      ]
    ]

我们再来解析如下日志：

<operation_in>请求报文：service_name接口名，sysfunc_id功能号，operator_id 操作员id，organ_id 机构号，request_seq 请求流水
    
    2018-11-12 15:03:41.388 639211542011357848  [DEBUG]  com.base.core.aop.http.HttpClient.send(HttpClient.java:128) - 
    reqid:b7fb8f90ddeb11e83d622c02b34132f7;AOP 发送信息: <?xml version="1.0" encoding="GBK"?><operation_in<service_name>BSM_SaleSystemLogin</service_name>     
    <sysfunc_id>91008027</sysfunc_id><request_type>1002</request_type><verify_code>304147201506190000000040</verify_code><operator_id>9991445</operator_id>              
     <organ_id>9999997</organ_id><request_time>20181112150341</request_time><request_seq>154200622111</request_seq><request_source>304147</request_source><request_target></request_target><msg_version>0100</msg_version><cont_version>0100</cont_version><access_token></access_token><content><request><msisdn>13xx6945211</msisdn><password>871221</password><portal_id>101704</portal_id><login_type>34</login_type><machine_mac>0000</machine_mac><machine_ip>120.33.xxx.198, 10.46.xxx.182, </machine_ip><machine_cpu></machine_cpu><machine_system_ver>12.0.1</machine_system_ver><machine_totalmemory></machine_totalmemory><machine_usablememory></machine_usablememory><machine_ie_ver></machine_ie_ver></request></content></operation_in>
    <operation_out>
    <operation_out><service_name>BSM_SaleSystemLogin</service_name><request_type>1002</request_type><sysfunc_id>91008027</sysfunc_id>
    <request_seq>15xxx0622111</request_seq><response_time>20181112150342</response_time><response_seq>471860579309</response_seq><request_source>304147</request_source><response><resp_type>0</resp_type><resp_code>0000</resp_code><resp_desc/></response><content><response><base_info><verifycode>173616671275425657328820</verifycode><operator_id>132394</operator_id><row><msisdn>13xxx945211</msisdn><role_id>6100004</role_id><owning_mode>1</owning_mode><status>1</status><inure_time>20170623145448</inure_time><expire_time>30000101000000</expire_time><request_source>0</request_source><modify_time>20170623145448</modify_time><modify_operator_id>4020205</modify_operator_id><modify_content>创建手机号码与角色对应关系

我们将此段日志在Grok网站进行解析，获得grok语句：

%{TIMESTAMP_ISO8601:log_time} %{DATA:serial_number} [%{DATA:log_level}] %{GREEDYDATA:message}<service_name>%{DATA:service_name}</service_name> <sysfunc_id>%{DATA:sysfunc_id}</sysfunc_id><request_type>%{DATA:other}</operator_id><organ_id>%{DATA:organ_id}</organ_id><request_time>%{DATA:request_time}</request_time><request_seq>%{DATA:request_seq}</request_seq><request_source>%{DATA:other}<operation_out>

解析结果如下：

{
      "log_time": [
        [
          "2018-11-12 15:03:41.388"
        ]
      ],
      "YEAR": [
        [
          "2018"
        ]
      ],
      "MONTHNUM": [
        [
          "11"
        ]
      ],
      "MONTHDAY": [
        [
          "12"
        ]
      ],
      "HOUR": [
        [
          "15",
          null
        ]
      ],
      "MINUTE": [
        [
          "03",
          null
        ]
      ],
      "SECOND": [
        [
          "41.388"
        ]
      ],
      "ISO8601_TIMEZONE": [
        [
          null
        ]
      ],
      "serial_number": [
        [
          "639211542011357848 "
        ]
      ],
      "log_level": [
        [
          "DEBUG"
        ]
      ],
      "message": [
        [
          " com.base.core.aop.http.HttpClient.send(HttpClient.java:128) - reqid:b7fb8f90ddeb11e83d622c02b34132f7;AOP 发送信息: <?xml version="1.0" encoding="GBK"?>   <operation_in"
        ]
      ],
      "service_name": [
        [
          "BSM_SaleSystemLogin"
        ]
      ],
      "sysfunc_id": [
        [
          "91008027"
        ]
      ],
      "other": [
        [
          "1002</request_type><verify_code>304147201506190000000040</verify_code><operator_id>9991445",
          "304147</request_source><request_target></request_target><msg_version>0100</msg_version><cont_version>0100</cont_version><access_token></access_token><content><request><msisdn>13xxx945211</msisdn><password>871221</password><portal_id>101704</portal_id><login_type>34</login_type><machine_mac>0000</machine_mac><machine_ip>120.33.xxx.198, 10.46.xxx.182, </machine_ip><machine_cpu></machine_cpu><machine_system_ver>12.0.1</machine_system_ver><machine_totalmemory></machine_totalmemory><machine_usablememory></machine_usablememory><machine_ie_ver></machine_ie_ver></request></content></operation_in>"
        ]
      ],
      "organ_id": [
        [
          "9999997"
        ]
      ],
      "request_time": [
        [
          "20181112150341"
        ]
      ],
      "request_seq": [
        [
          "154xxx622111"
        ]
      ]
    }

我们再来解析一条niginx的日志：

2018/11/01 23:30:39 [error] 15105#0: *397937824 connect() failed (111: Connection refused) while connecting to upstream, client: 10.48.xxx.3, server: 127.0.0.1, request: "POST /o2o_usercenter_svc/xxx/sysUserInfoService?req_sid=1612e430ddeb11e83d622c02b34132f7&syslogid=null HTTP/1.1", upstream: "http://127.0.0.1:xxx/o2o_usercenter_svc/remote/sysUserInfoService?req_sid=1612e430ddeb11e83d622c02b34132f7&syslogid=null", host: "10.46.xxx.155:xxx"

解析语句：

(?%{YEAR}[./-]%{MONTHNUM}[./-]%{MONTHDAY}[- ]%{TIME}) [%{LOGLEVEL:severity}] %{POSINT:pid}#%{NUMBER}: %{GREEDYDATA:errormessage}(?:, client: (?<remote_addr>%{IP}|%{HOSTNAME}))(?:, server: %{IPORHOST:server}?)(?:, request: %{QS:request})?(?:, upstream: (?\”%{URI}\”|%{QS}))?(?:, host: %{QS:request_host})?(?:, referrer: \”%{URI:referrer}\”)?

解析结果如下：

{
      "timestamp": [
        [
          "2018/11/01 23:30:39"
        ]
      ],
      "YEAR": [
        [
          "2018"
        ]
      ],
      "MONTHNUM": [
        [
          "11"
        ]
      ],
      "MONTHDAY": [
        [
          "01"
        ]
      ],
      "TIME": [
        [
          "23:30:39"
        ]
      ],
      "HOUR": [
        [
          "23"
        ]
      ],
      "MINUTE": [
        [
          "30"
        ]
      ],
      "SECOND": [
        [
          "39"
        ]
      ],
      "severity": [
        [
          "error"
        ]
      ],
      "pid": [
        [
          "15105"
        ]
      ],
      "NUMBER": [
        [
          "0"
        ]
      ],
      "BASE10NUM": [
        [
          "0"
        ]
      ],
      "errormessage": [
        [
          "*397937824 connect() failed (111: Connection refused) while connecting to upstream"
        ]
      ],
      "remote_addr": [
        [
          "10.48.xxx.3"
        ]
      ],
      "IP": [
        [
          "10.48.xxx.3",
          null,
          null,
          null
        ]
      ],
      "IPV6": [
        [
          null,
          null,
          null,
          null
        ]
      ],
      "IPV4": [
        [
          "10.48.xxx.3",
          null,
          null,
          null
        ]
      ],
      "HOSTNAME": [
        [
          null,
          "127.0.0.1",
          "127.0.0.1",
          null
        ]
      ],
      "server": [
        [
          "127.0.0.1"
        ]
      ],
     ...
    }

## Kibana图表面板 ##

![图片][618e793a687709308239fdcc7ca91e68.png]

上图中，我们可以在Kibana上面配置一些监控面板。比如配置异常日志监控。

关于ELK监控面板配置，有兴趣的童鞋可以看看这篇文章：

> “
> 
> https://blog.51cto.com/hnr520/1845900
> 
> ”

# **往期推荐** #

*  [我写出这样干净的代码，老板直夸我][Link 1]
 *  [云南丽江旅游攻略][Link 2]
 *  [使用ThreadLocal怕内存泄漏？][ThreadLocal]
 *  [Java进阶之路思维导图][Java]
 *  [程序员必看书籍推荐][Link 3]
 *  [3万字的Java后端面试总结（附PDF）][3_Java_PDF]

扫码二维码，获取更多精彩。或微信搜**Lvshen\_9**，可后台回复获取资料

> 回复"java" 获取java电子书；
> 
> 回复"python"获取python电子书；
> 
> 回复"算法"获取算法电子书；
> 
> 回复"大数据"获取大数据电子书；
> 
> 回复"spring"获取SpringBoot的学习视频。
> 
> 回复"面试"获取一线大厂面试资料
> 
> 回复"进阶之路"获取Java进阶之路的思维导图
> 
> 回复"手册"获取阿里巴巴Java开发手册(嵩山终极版)
> 
> 回复"总结"获取Java后端面试经验总结PDF版
> 
> 回复"Redis"获取Redis命令手册，和Redis专项面试习题（PDF）
> 
> 回复"并发导图"获取Java并发编程思维导图（xmind终极版）

另：点击【**我的福利**】有更多惊喜哦。

![format_png][]

[2594e17ced13b93730492a21ebf787bf.png]: /images/20221022/a0ccd385cda84d1b86ed33424fc6e9ac.png
[20210330142217761.gif]: /images/20221022/a1e63ebea0c5457abf60df5ec3e2e285.png
[801a6b316561432a3a2548e130ef5e3d.png]: /images/20221022/33099ac004644c569f171cd896657b21.png
[20210330142217759.gif]: /images/20221022/c5e5ea8a82194ca29b5c1b7140454126.png
[20210330142217757.gif]: /images/20221022/9182279216d841228f258af4fbf3de8f.png
[63b0e6f558cc0e56ea1f715c34a1a26d.png]: /images/20221022/7dce28349fb94af6887f08afcc39fc43.png
[f042d8b7d8ffce7cb3b70c6950a57899.png]: /images/20221022/fea5084299934b8684a8898ffde177e0.png
[20210330142217767.gif]: https://img-blog.csdnimg.cn/20210330142217767.gif
[618e793a687709308239fdcc7ca91e68.png]: https://img-blog.csdnimg.cn/img_convert/618e793a687709308239fdcc7ca91e68.png
[Link 1]: http://mp.weixin.qq.com/s?__biz=Mzg4NDMyNTIwMw==&mid=2247487966&idx=1&sn=7810bd4f62f2b92bec831db2e1f35b97&chksm=cfb8bfe2f8cf36f4295a3560fc8fde094bf73e8e187ff589c6559ae7ed38584d8c67c1c20330&scene=21#wechat_redirect
[Link 2]: http://mp.weixin.qq.com/s?__biz=Mzg4NDMyNTIwMw==&mid=2247487391&idx=1&sn=5847806bce7a105d6503eab52ea725be&chksm=cfb8a1a3f8cf28b517b36ded4dc1ec97334b0a685e5deb6f75d9a668c96d3fbc0c058e5432d8&scene=21#wechat_redirect
[ThreadLocal]: http://mp.weixin.qq.com/s?__biz=Mzg4NDMyNTIwMw==&mid=2247486830&idx=1&sn=3aa8095371b2e04dc0c954443a890ef4&chksm=cfb8a352f8cf2a444a4c54d19a5e74b03879c03e79209a1f7688fd9edd81edb35b43fea9578d&scene=21#wechat_redirect
[Java]: http://mp.weixin.qq.com/s?__biz=Mzg4NDMyNTIwMw==&mid=2247486701&idx=2&sn=8b6c8da8802d4fceb4ad06c229af5391&chksm=cfb8a2d1f8cf2bc71415098ecc807dfcb6bcfdde399ec92f79ead84f0f4352a5fdb7cf766593&scene=21#wechat_redirect
[Link 3]: http://mp.weixin.qq.com/s?__biz=Mzg4NDMyNTIwMw==&mid=2247486232&idx=1&sn=ac1ba6a9f543f7115098f820a7b41165&chksm=cfb8a524f8cf2c326861c7a0783a6ada2ba7a3e76c633cb9632af99f32de64a984fef5ba8a75&scene=21#wechat_redirect
[3_Java_PDF]: http://mp.weixin.qq.com/s?__biz=Mzg4NDMyNTIwMw==&mid=2247488123&idx=1&sn=9567232df10afbfede9688e4762c988c&chksm=cfb8bc47f8cf35512bced50381acb94713b51bb8691e895ba880a2b7863d7c882f4b54aec190&scene=21#wechat_redirect
[format_png]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X2pwZy84MjJ5d2djOWRDS3M4TnVYVW12SHN2QVBFODZOQnB2TUQ2OGliY1QwV2NKVnRpYWVPMFd0MENkWnhwcklSVDVVYjdXTzJWMmljQzRPOEYwUW5pYURDT2RKWkEvNjQw?x-oss-process=image/format,png