actuator+prometheus+grafana实现服务监控

素颜马尾好姑娘i 2022-12-05 10:16 453阅读 0赞

# actuator+prometheus+grafana实现服务监控 #

> 对Spring Boot监控能力有过了解的小伙伴都应该知道Spring Boot Actuator这个子项目，它为应用提供了强大的监控能力。从Spring Boot 2.x开始，Actuator将底层改为Micrometer，提供了更强、更灵活的监控能力。Micrometer是一个监控门面，可以类比成监控界的 Slf4j 。借助Micrometer，应用能够对接各种监控系统，例如本文所要介绍的：Prometheus

> Prometheus 是一个开源的服务监控系统和时间序列数据库。阿里的集群监控解决方案就是 Prometheus

## spring boot 配置 ##

### 第一步：添加maven依赖 ###

<dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-actuator</artifactId>
    </dependency>
    <dependency>
        <groupId>io.micrometer</groupId>
        <artifactId>micrometer-registry-prometheus</artifactId>
    </dependency>

> 如果使用InfluxDB进行数据存储可以使用 `micrometer-registry-influx`

### 第二步：配置 ###

# 启用 prometheus和health监控
    management.endpoints.web.exposure.include=prometheus,health
    management.endpoints.metrics.export.prometheus.enabled=true
    # 关闭elasticsearch客户端检查
    management.health.elasticsearch.enabled=false
    management.metrics.tags.application=${spring.application.name}

yml配置:

management:
      endpoints:
        web:
          exposure:
            # 将 Actuator 的 /actuator/prometheus 端点暴露出来
            include: 'prometheus,health'
      metrics:
        tags:
          # 为指标设置一个Tag，这里设置为应用名，Tag是Prometheus提供的一种能力，从而实现更加灵活的筛选
          application: ${ spring.application.name}

> 注意要加单引号

### 第三步：启动服务访问 /actuator/prometheus ###

url http://localhost:7005/actuator/prometheus

#### 监控项详解 ####

<table> 
 <thead> 
  <tr> 
   <th>序号</th> 
   <th>参数</th> 
   <th>参数说明</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>–</td> 
   <td>JVM</td> 
   <td>—</td> 
  </tr> 
  <tr> 
   <td>1</td> 
   <td>jvm.memory.max</td> 
   <td>JVM最大内存</td> 
  </tr> 
  <tr> 
   <td>2</td> 
   <td>jvm.memory.committed</td> 
   <td>JVM可用内存</td> 
  </tr> 
  <tr> 
   <td>3</td> 
   <td>jvm.memory.used</td> 
   <td>JVM已用内存</td> 
  </tr> 
  <tr> 
   <td>4</td> 
   <td>jvm.buffer.memory.used</td> 
   <td>JVM缓冲区已用内存</td> 
  </tr> 
  <tr> 
   <td>5</td> 
   <td>jvm.buffer.count</td> 
   <td>当前缓冲区数</td> 
  </tr> 
  <tr> 
   <td>6</td> 
   <td>jvm.threads.daemon</td> 
   <td>JVM守护线程数</td> 
  </tr> 
  <tr> 
   <td>7</td> 
   <td>jvm.threads.live</td> 
   <td>JVM当前活跃线程数</td> 
  </tr> 
  <tr> 
   <td>8</td> 
   <td>jvm.threads.peak</td> 
   <td>JVM峰值线程数</td> 
  </tr> 
  <tr> 
   <td>9</td> 
   <td>jvm.classes.loaded</td> 
   <td>加载classes数</td> 
  </tr> 
  <tr> 
   <td>10</td> 
   <td>jvm.classes.unloaded</td> 
   <td>未加载的classes数</td> 
  </tr> 
  <tr> 
   <td>11</td> 
   <td>jvm.gc.memory.allocated</td> 
   <td>GC时，年轻代分配的内存空间</td> 
  </tr> 
  <tr> 
   <td>12</td> 
   <td>jvm.gc.memory.promoted</td> 
   <td>GC时，老年代分配的内存空间</td> 
  </tr> 
  <tr> 
   <td>13</td> 
   <td>jvm.gc.max.data.size</td> 
   <td>GC时，老年代的最大内存空间</td> 
  </tr> 
  <tr> 
   <td>14</td> 
   <td>jvm.gc.live.data.size</td> 
   <td>FullGC时，老年代的内存空间</td> 
  </tr> 
  <tr> 
   <td>15</td> 
   <td>jvm.gc.pause</td> 
   <td>GC耗时</td> 
  </tr> 
  <tr> 
   <td>–</td> 
   <td>TOMCAT</td> 
   <td>—</td> 
  </tr> 
  <tr> 
   <td>16</td> 
   <td>tomcat.sessions.created</td> 
   <td>tomcat已创建session数</td> 
  </tr> 
  <tr> 
   <td>17</td> 
   <td>tomcat.sessions.expired</td> 
   <td>tomcat已过期session数</td> 
  </tr> 
  <tr> 
   <td>18</td> 
   <td>tomcat.sessions.active.current</td> 
   <td>tomcat活跃session数</td> 
  </tr> 
  <tr> 
   <td>19</td> 
   <td>tomcat.sessions.active.max</td> 
   <td>tomcat最多活跃session数</td> 
  </tr> 
  <tr> 
   <td>20</td> 
   <td>tomcat.sessions.alive.max.second</td> 
   <td>tomcat最多活跃session数持续时间</td> 
  </tr> 
  <tr> 
   <td>21</td> 
   <td>tomcat.sessions.rejected</td> 
   <td>超过session最大配置后，拒绝的session个数</td> 
  </tr> 
  <tr> 
   <td>22</td> 
   <td>tomcat.global.error</td> 
   <td>错误总数</td> 
  </tr> 
  <tr> 
   <td>23</td> 
   <td>tomcat.global.sent</td> 
   <td>发送的字节数</td> 
  </tr> 
  <tr> 
   <td>24</td> 
   <td>tomcat.global.request.max</td> 
   <td>request最长时间</td> 
  </tr> 
  <tr> 
   <td>25</td> 
   <td>tomcat.global.request</td> 
   <td>全局request次数和时间</td> 
  </tr> 
  <tr> 
   <td>26</td> 
   <td>tomcat.global.received</td> 
   <td>全局received次数和时间</td> 
  </tr> 
  <tr> 
   <td>27</td> 
   <td>tomcat.servlet.request</td> 
   <td>servlet的请求次数和时间</td> 
  </tr> 
  <tr> 
   <td>28</td> 
   <td>tomcat.servlet.error</td> 
   <td>servlet发生错误总数</td> 
  </tr> 
  <tr> 
   <td>29</td> 
   <td>tomcat.servlet.request.max</td> 
   <td>servlet请求最长时间</td> 
  </tr> 
  <tr> 
   <td>30</td> 
   <td>tomcat.threads.busy</td> 
   <td>tomcat繁忙线程</td> 
  </tr> 
  <tr> 
   <td>31</td> 
   <td>tomcat.threads.current</td> 
   <td>tomcat当前线程数</td> 
  </tr> 
  <tr> 
   <td>32</td> 
   <td>tomcat.threads.config.max</td> 
   <td>tomcat配置的线程最大数</td> 
  </tr> 
  <tr> 
   <td>33</td> 
   <td>tomcat.cache.access</td> 
   <td>tomcat读取缓存次数</td> 
  </tr> 
  <tr> 
   <td>34</td> 
   <td>tomcat.cache.hit</td> 
   <td>tomcat缓存命中次数</td> 
  </tr> 
  <tr> 
   <td>–</td> 
   <td>CPU</td> 
   <td>—</td> 
  </tr> 
  <tr> 
   <td>35</td> 
   <td>system.cpu.count</td> 
   <td>CPU数量</td> 
  </tr> 
  <tr> 
   <td>36</td> 
   <td>system.load.average.1m</td> 
   <td>load average</td> 
  </tr> 
  <tr> 
   <td>37</td> 
   <td>system.cpu.usage</td> 
   <td>系统CPU使用率</td> 
  </tr> 
  <tr> 
   <td>38</td> 
   <td>process.cpu.usage</td> 
   <td>当前进程CPU使用率</td> 
  </tr> 
  <tr> 
   <td>39</td> 
   <td>http.server.requests</td> 
   <td>http请求调用情况</td> 
  </tr> 
  <tr> 
   <td>40</td> 
   <td>process.uptime</td> 
   <td>应用已运行时间</td> 
  </tr> 
  <tr> 
   <td>41</td> 
   <td>process.files.max</td> 
   <td>允许最大句柄数</td> 
  </tr> 
  <tr> 
   <td>42</td> 
   <td>process.start.time</td> 
   <td>应用启动时间点</td> 
  </tr> 
  <tr> 
   <td>43</td> 
   <td>process.files.open</td> 
   <td>当前打开句柄数</td> 
  </tr> 
 </tbody> 
</table>

返回内容：（监控项和监控数值）

# HELP jvm_threads_states_threads The current number of threads having NEW state
    # TYPE jvm_threads_states_threads gauge
    jvm_threads_states_threads{application="feedback",state="new",} 0.0
    jvm_threads_states_threads{application="feedback",state="timed-waiting",} 17.0
    jvm_threads_states_threads{application="feedback",state="blocked",} 0.0
    jvm_threads_states_threads{application="feedback",state="waiting",} 24.0
    jvm_threads_states_threads{application="feedback",state="terminated",} 0.0
    jvm_threads_states_threads{application="feedback",state="runnable",} 19.0
    # HELP jvm_gc_max_data_size_bytes Max size of old generation memory pool
    # TYPE jvm_gc_max_data_size_bytes gauge
    jvm_gc_max_data_size_bytes{application="feedback",} 5.703204864E9
    # HELP tomcat_sessions_active_max_sessions  
    # TYPE tomcat_sessions_active_max_sessions gauge
    tomcat_sessions_active_max_sessions{application="feedback",} 0.0
    # HELP jvm_memory_committed_bytes The amount of memory in bytes that is committed for the Java virtual machine to use
    # TYPE jvm_memory_committed_bytes gauge
    jvm_memory_committed_bytes{application="feedback",area="nonheap",id="Metaspace",} 6.4094208E7
    jvm_memory_committed_bytes{application="feedback",area="nonheap",id="Code Cache",} 1.6973824E7
    jvm_memory_committed_bytes{application="feedback",area="heap",id="PS Survivor Space",} 2.0447232E7
    jvm_memory_committed_bytes{application="feedback",area="heap",id="PS Old Gen",} 3.89021696E8
    jvm_memory_committed_bytes{application="feedback",area="heap",id="PS Eden Space",} 4.5350912E8
    jvm_memory_committed_bytes{application="feedback",area="nonheap",id="Compressed Class Space",} 8519680.0
    # HELP jvm_memory_max_bytes The maximum amount of memory in bytes that can be used for memory management
    # TYPE jvm_memory_max_bytes gauge
    jvm_memory_max_bytes{application="feedback",area="nonheap",id="Metaspace",} -1.0
    jvm_memory_max_bytes{application="feedback",area="nonheap",id="Code Cache",} 2.5165824E8
    jvm_memory_max_bytes{application="feedback",area="heap",id="PS Survivor Space",} 2.0447232E7
    jvm_memory_max_bytes{application="feedback",area="heap",id="PS Old Gen",} 5.703204864E9
    jvm_memory_max_bytes{application="feedback",area="heap",id="PS Eden Space",} 2.809659392E9
    jvm_memory_max_bytes{application="feedback",area="nonheap",id="Compressed Class Space",} 1.073741824E9
    # HELP jvm_buffer_total_capacity_bytes An estimate of the total capacity of the buffers in this pool
    # TYPE jvm_buffer_total_capacity_bytes gauge
    jvm_buffer_total_capacity_bytes{application="feedback",id="direct",} 81936.0
    jvm_buffer_total_capacity_bytes{application="feedback",id="mapped",} 0.0
    # HELP tomcat_sessions_active_current_sessions  
    # TYPE tomcat_sessions_active_current_sessions gauge
    tomcat_sessions_active_current_sessions{application="feedback",} 0.0
    # HELP system_cpu_count The number of processors available to the Java virtual machine
    # TYPE system_cpu_count gauge
    system_cpu_count{application="feedback",} 6.0
    # HELP process_cpu_usage The "recent cpu usage" for the Java Virtual Machine process
    # TYPE process_cpu_usage gauge
    process_cpu_usage{application="feedback",} 0.001260906050531724
    # HELP system_cpu_usage The "recent cpu usage" for the whole system
    # TYPE system_cpu_usage gauge
    system_cpu_usage{application="feedback",} 0.06223864591061535
    # HELP process_uptime_seconds The uptime of the Java virtual machine
    # TYPE process_uptime_seconds gauge
    process_uptime_seconds{application="feedback",} 6079.537
    # HELP jvm_gc_live_data_size_bytes Size of old generation memory pool after a full GC
    # TYPE jvm_gc_live_data_size_bytes gauge
    jvm_gc_live_data_size_bytes{application="feedback",} 3.5917448E7
    # HELP tomcat_sessions_expired_sessions_total  
    # TYPE tomcat_sessions_expired_sessions_total counter
    tomcat_sessions_expired_sessions_total{application="feedback",} 0.0
    # HELP jvm_threads_live_threads The current number of live threads including both daemon and non-daemon threads
    # TYPE jvm_threads_live_threads gauge
    jvm_threads_live_threads{application="feedback",} 60.0
    # HELP http_server_requests_seconds  
    # TYPE http_server_requests_seconds summary
    http_server_requests_seconds_count{application="feedback",exception="None",method="GET",outcome="SUCCESS",status="200",uri="/actuator",} 1.0
    http_server_requests_seconds_sum{application="feedback",exception="None",method="GET",outcome="SUCCESS",status="200",uri="/actuator",} 0.0041683
    http_server_requests_seconds_count{application="feedback",exception="None",method="PUT",outcome="SUCCESS",status="200",uri="/feedback",} 1.0
    http_server_requests_seconds_sum{application="feedback",exception="None",method="PUT",outcome="SUCCESS",status="200",uri="/feedback",} 60.8246484
    http_server_requests_seconds_count{application="feedback",exception="None",method="GET",outcome="SUCCESS",status="200",uri="/actuator/prometheus",} 326.0
    http_server_requests_seconds_sum{application="feedback",exception="None",method="GET",outcome="SUCCESS",status="200",uri="/actuator/prometheus",} 19.7430817
    http_server_requests_seconds_count{application="feedback",exception="None",method="GET",outcome="CLIENT_ERROR",status="404",uri="/**",} 5.0
    http_server_requests_seconds_sum{application="feedback",exception="None",method="GET",outcome="CLIENT_ERROR",status="404",uri="/**",} 0.1397898
    # HELP http_server_requests_seconds_max  
    # TYPE http_server_requests_seconds_max gauge
    http_server_requests_seconds_max{application="feedback",exception="None",method="GET",outcome="SUCCESS",status="200",uri="/actuator",} 0.0
    http_server_requests_seconds_max{application="feedback",exception="None",method="PUT",outcome="SUCCESS",status="200",uri="/feedback",} 0.0
    http_server_requests_seconds_max{application="feedback",exception="None",method="GET",outcome="SUCCESS",status="200",uri="/actuator/prometheus",} 0.0560053
    http_server_requests_seconds_max{application="feedback",exception="None",method="GET",outcome="CLIENT_ERROR",status="404",uri="/**",} 0.0
    # HELP jvm_threads_peak_threads The peak live thread count since the Java virtual machine started or peak was reset
    # TYPE jvm_threads_peak_threads gauge
    jvm_threads_peak_threads{application="feedback",} 60.0
    # HELP jvm_threads_daemon_threads The current number of live daemon threads
    # TYPE jvm_threads_daemon_threads gauge
    jvm_threads_daemon_threads{application="feedback",} 47.0
    # HELP tomcat_sessions_created_sessions_total  
    # TYPE tomcat_sessions_created_sessions_total counter
    tomcat_sessions_created_sessions_total{application="feedback",} 0.0
    # HELP jvm_gc_memory_allocated_bytes_total Incremented for an increase in the size of the young generation memory pool after one GC to before the next
    # TYPE jvm_gc_memory_allocated_bytes_total counter
    jvm_gc_memory_allocated_bytes_total{application="feedback",} 8.452706E8
    # HELP tomcat_sessions_rejected_sessions_total  
    # TYPE tomcat_sessions_rejected_sessions_total counter
    tomcat_sessions_rejected_sessions_total{application="feedback",} 0.0
    # HELP jvm_gc_pause_seconds Time spent in GC pause
    # TYPE jvm_gc_pause_seconds summary
    jvm_gc_pause_seconds_count{action="end of minor GC",application="feedback",cause="Metadata GC Threshold",} 1.0
    jvm_gc_pause_seconds_sum{action="end of minor GC",application="feedback",cause="Metadata GC Threshold",} 0.021
    jvm_gc_pause_seconds_count{action="end of minor GC",application="feedback",cause="Allocation Failure",} 2.0
    jvm_gc_pause_seconds_sum{action="end of minor GC",application="feedback",cause="Allocation Failure",} 0.028
    jvm_gc_pause_seconds_count{action="end of major GC",application="feedback",cause="Metadata GC Threshold",} 1.0
    jvm_gc_pause_seconds_sum{action="end of major GC",application="feedback",cause="Metadata GC Threshold",} 0.113
    # HELP jvm_gc_pause_seconds_max Time spent in GC pause
    # TYPE jvm_gc_pause_seconds_max gauge
    jvm_gc_pause_seconds_max{action="end of minor GC",application="feedback",cause="Metadata GC Threshold",} 0.0
    jvm_gc_pause_seconds_max{action="end of minor GC",application="feedback",cause="Allocation Failure",} 0.0
    jvm_gc_pause_seconds_max{action="end of major GC",application="feedback",cause="Metadata GC Threshold",} 0.0
    # HELP jvm_gc_memory_promoted_bytes_total Count of positive increases in the size of the old generation memory pool before GC to after GC
    # TYPE jvm_gc_memory_promoted_bytes_total counter
    jvm_gc_memory_promoted_bytes_total{application="feedback",} 1.9524136E7
    # HELP jvm_buffer_memory_used_bytes An estimate of the memory that the Java virtual machine is using for this buffer pool
    # TYPE jvm_buffer_memory_used_bytes gauge
    jvm_buffer_memory_used_bytes{application="feedback",id="direct",} 81936.0
    jvm_buffer_memory_used_bytes{application="feedback",id="mapped",} 0.0
    # HELP jvm_classes_unloaded_classes_total The total number of classes unloaded since the Java virtual machine has started execution
    # TYPE jvm_classes_unloaded_classes_total counter
    jvm_classes_unloaded_classes_total{application="feedback",} 6.0
    # HELP tomcat_sessions_alive_max_seconds  
    # TYPE tomcat_sessions_alive_max_seconds gauge
    tomcat_sessions_alive_max_seconds{application="feedback",} 0.0
    # HELP process_start_time_seconds Start time of the process since unix epoch.
    # TYPE process_start_time_seconds gauge
    process_start_time_seconds{application="feedback",} 1.599642895483E9
    # HELP logback_events_total Number of error level events that made it to the logs
    # TYPE logback_events_total counter
    logback_events_total{application="feedback",level="error",} 0.0
    logback_events_total{application="feedback",level="debug",} 0.0
    logback_events_total{application="feedback",level="info",} 236.0
    logback_events_total{application="feedback",level="trace",} 0.0
    logback_events_total{application="feedback",level="warn",} 964.0
    # HELP jvm_buffer_count_buffers An estimate of the number of buffers in the pool
    # TYPE jvm_buffer_count_buffers gauge
    jvm_buffer_count_buffers{application="feedback",id="direct",} 11.0
    jvm_buffer_count_buffers{application="feedback",id="mapped",} 0.0
    # HELP jvm_memory_used_bytes The amount of used memory
    # TYPE jvm_memory_used_bytes gauge
    jvm_memory_used_bytes{application="feedback",area="nonheap",id="Metaspace",} 6.0684336E7
    jvm_memory_used_bytes{application="feedback",area="nonheap",id="Code Cache",} 1.6949312E7
    jvm_memory_used_bytes{application="feedback",area="heap",id="PS Survivor Space",} 6370080.0
    jvm_memory_used_bytes{application="feedback",area="heap",id="PS Old Gen",} 3.5945144E7
    jvm_memory_used_bytes{application="feedback",area="heap",id="PS Eden Space",} 2.6841788E8
    jvm_memory_used_bytes{application="feedback",area="nonheap",id="Compressed Class Space",} 7853464.0
    # HELP jvm_classes_loaded_classes The number of classes that are currently loaded in the Java virtual machine
    # TYPE jvm_classes_loaded_classes gauge
    jvm_classes_loaded_classes{application="feedback",} 11494.0

## 配置启动Prometheus ##

![Prometheus官方文档][Prometheus]

### 第一步：配置Prometheus ###

创建prometheus.yml配置文件并且如输入如下配置：

scrape_configs:
    # 任意写，建议英文，不要包含特殊字符
    - job_name: 'feedback'
      # 多久采集一次数据
      scrape_interval: 15s
      # 采集时的超时时间
      scrape_timeout: 10s
      # 采集的路径是啥
      metrics_path: '/actuator/prometheus'
      # 采集服务的地址，设置成上面Spring Boot应用所在服务器的具体地址。
      static_configs:
      - targets: ['192.168.2.137:8080']

### 第二步：Docker启动Prometheus ###

docker run -d \
    	--name prometheus \
        -p 9090:9090 \
        -v /home/docker/conf/prometheus.yml:/etc/prometheus/prometheus.yml \
        prom/prometheus

### 第三步：配置并查询服务监控信息 ###

#### 查看服务信息 ####

访问：

http://192.168.1.233:9090/targets

如下图：

![24151ccfb4dea15bdfbfb5b4dd5ec996.png][]

#### 查看配置已守护线程线程数（jvm\_threads\_daemon\_threads） ####

![f06221be50615f43970207c4ae53dd2c.png][]

1.  选择 `Graph` 图表
2.  选择 `jvm_threads_daemon_threads` JVM守护线程数
3.  选择 `Graph` TAB

## 安装Grafana可视化 ##

> Grafana 是一款采用 GO 语言编写的开源应用，支持跨平台度量分析和可视化 + 告警。可以通过将采集的数据查询然后可视化地展示，并及时通知。Grafana 支持多种数据源和展示方式，总而言之是一款强大酷炫的监控指标可视化工具。

更多内容参考：[官方文档][Link 1]

### docker安装Grafana ###

docker run -d --name grafana -p 3000:3000 grafana/grafana

### 登录Grafana ###

> 登录地址：http://192.168.1.233:3000/login （IP根据自己情况修改）

> 用户名/密码 ： admin/admin

![ff500f6073bbd792e85b8b5390d3bffe.png][]

### 添加数据源 ###

![c5ae1830c08fba7575c47af1e9c88a16.png][]

![62380f2ff4f8f04ae5303a00f0249549.png][]

### 添加面板（dashboard） ###

![8230528b68047a48b666381def2ec5c5.png][]

![5ee8552976b9b7495569bec4b09c066f.png][]

[Prometheus]: /images/20221123/d96ab0c1e8004c9a83b0548e2fbd00f0.png
[24151ccfb4dea15bdfbfb5b4dd5ec996.png]: /images/20221123/e37724535121472cb6d1b6a463fb4466.png
[f06221be50615f43970207c4ae53dd2c.png]: /images/20221123/0429d4c2e2c04fc5b016e883f68221ba.png
[Link 1]: https://grafana.com/docs/grafana/latest/
[ff500f6073bbd792e85b8b5390d3bffe.png]: /images/20221123/2fa47af853e8419fa444929aa8bb5edc.png
[c5ae1830c08fba7575c47af1e9c88a16.png]: /images/20221123/672593c62a514e9b92ab96a89d133bad.png
[62380f2ff4f8f04ae5303a00f0249549.png]: /images/20221123/ffcb5f93e2d84a0ebc14a5d0544ed564.png
[8230528b68047a48b666381def2ec5c5.png]: /images/20221123/13292f9bc0814944a8e5842d1361070b.png
[5ee8552976b9b7495569bec4b09c066f.png]: /images/20221123/97d05f58133c4da2b2c5080b98c9806c.png