prometheus+grafana 搭建企业级监控系统（三） altermanage 监控通知企业微信

拼搏现实的明天。 2022-12-22 12:28 359阅读 0赞

## 前言： ##

从下面的架构图看出，当prometheus拉取信息时，可以通过配置rules（规则）预警，把符合预警信息的指标push给altermanager，

altermanager然后把这些指标通过邮件，webhook，微信推送（企业）等推送给相关人员。

这就是一个完整的企业监控系统。

![e217ce37fb4801789c11f72c3af4dbff.png][]

## altermanager安装： ##

从prometheus  [https://prometheus.io/download/][https_prometheus.io_download] 官网下载 altermanager

比如我是推送企业微信，编辑 alertmanager/alertmanager.yml

global:
      resolve_timeout: 5m
    
    templates:   # 告警模板
      - './template/test.tmpl'
    
    route:
      group_by: ['alertname'] # 分组标签
      group_wait: 10s         #告警等待时间。告警产生后等待10s，如果有同组告警一起发出
      group_interval: 10s     # 两组告警的间隔时间 
      repeat_interval: 1m     # 重复告警的时间间隔，减少相同邮件的发送频率，此处设计测试为1分钟
      #receiver: 'web.hook'
      receiver: 'wechat'      #设置默认接收者
    
    receivers:
    - name: 'wechat'
      wechat_configs:
        - send_resolved: true
          agent_id: '1000242'     # 自建应用的agentId
          to_user: '@all'  # 接收告警消息的人员Id   
          api_secret: '*************' # 自建应用的secret
          corp_id: '********'  # 企业ID

然后修改prometheus的配置文件  vim  prometheus.yml

# Alertmanager configuration
    alerting:
      alertmanagers:
      - static_configs:
        #- targets:
        - targets: ['localhost:9093']
          # - alertmanager:9093
    
    # Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
    rule_files:
      - "/usr/local/prometheus/prometheus/rules/*.yml"
      # - "first_rules.yml"
      # - "second_rules.yml"

cd /usr/local/prometheus/prometheus/rules 下创建各个预警规则。

mysql-alert.yml

groups:                                                                                                                                         
    - name: MySQLStatsAlert
      rules:
      - alert: MySQL is down
        expr: mysql_up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "实例 {
       { $labels.instance }} MySQL is down"
          description: "MySQL 数据库挂了. 请立即采取行动!"
      - alert: open files high
        expr: mysql_global_status_innodb_num_open_files > (mysql_global_variables_open_files_limit) * 0.75
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "Instance {
       { $labels.instance }} open files high"
          description: "Open files is high. Please consider increasing open_files_limit."
      - alert: Read buffer size is bigger than max. allowed packet size
        expr: mysql_global_variables_read_buffer_size > mysql_global_variables_slave_max_allowed_packet 
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "Instance {
       { $labels.instance }} Read buffer size is bigger than max. allowed packet size"
          description: "Read buffer size (read_buffer_size) is bigger than max. allowed packet size (max_allowed_packet).This can break your replication."
      - alert: Sort buffer possibly missconfigured
        expr: mysql_global_variables_innodb_sort_buffer_size <256*1024 or mysql_global_variables_read_buffer_size > 4*1024*1024 
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "Instance {
       { $labels.instance }} Sort buffer possibly missconfigured"
          description: "Sort buffer size is either too big or too small. A good value for sort_buffer_size is between 256k and 4M."

redis-alert.yml

groups:
    - name: redis_alert
      rules:
      - alert: redis is down
        expr: redis_up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "实例 {
       { $labels.instance }} redis is down"
          description: "redis 挂了. 请立即采取行动!"
    ### 内存 ###
    # 默认内存告警策略
      - alert: redis内存95%
        expr: ((floor(redis_memory_used_rss_bytes / redis_memory_max_bytes * 100) >= 95) or (floor(redis_mem_use_ratio) >= 95)) and ((redis_memory_max_bytes <= 1024 * 1024 * 1024 * 4) or (redis_mem_total_size <= 4))
        for: 3m
        labels:
          severity: warning
        annotations:
          description: "[{
       { $labels.alias }}],地址:[{
       { $labels.addr }}],告警值为:[{
       { $value }}%],告警初始时长为3分钟."
      - alert: redis内存98%
        expr: ((floor(redis_memory_used_rss_bytes / redis_memory_max_bytes * 100) >= 98) or (floor(redis_mem_use_ratio) >= 98)) and ((redis_memory_max_bytes > 1024 * 1024 * 1024 * 4) or (redis_mem_total_size > 4))
        for: 3m
        labels:
          severity: warning
        annotations:
          description: "[{
       { $labels.alias }}],地址:[{
       { $labels.addr }}],告警值为:[{
       { $value }}%],告警初始时长为3分钟."

然后启动alertmanager和prometheus。

## 效果 ##

关掉 redis或者是mysql ,微信会受到以下消息

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NocmVrMTE_size_16_color_FFFFFF_t_70][]

[e217ce37fb4801789c11f72c3af4dbff.png]: /images/20221120/21b8b44400034bf5909620129e6b3831.png
[https_prometheus.io_download]: https://prometheus.io/download/
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NocmVrMTE_size_16_color_FFFFFF_t_70]: /images/20221120/efe9c726c6994327beb040d273fb6a19.png