canal prometheus性能监控

超、凢脫俗 2022-08-31 09:48 292阅读 0赞

--------------------

## canal prometheus性能监控 ##

**官网：[https://github.com/alibaba/canal/wiki/Prometheus-QuickStart][https_github.com_alibaba_canal_wiki_Prometheus-QuickStart]**

## \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* ##

**相关操作**

**创建 mysql**

docker run -it -d --net fixed3 --ip 192.168.57.2 --privileged=true \
    --name mysql -e  MYSQL_ROOT_PASSWORD=123456 mysql
    
    
    # 创建用户、并授权
    mysql> create user canal identified with mysql_native_password by "123456";
    Query OK, 0 rows affected (0.01 sec)
    
    mysql> GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%';
     value int not null);
    Query OK, 0 rows affected (0.00 sec)
    
    mysql> flush privileges;
    Query OK, 0 rows affected (0.00 sec)

**创建 canal server**

docker run -it -d --net fixed3 --ip 192.168.57.3 \
    -p 11111:11111 --name canal-server \
    -e canal.instance.master.address=192.168.57.2:3306  \
    -e canal.instance.dbUsername=canal \
    -e canal.instance.dbPassword=123456 canal/canal-server

**创建 prometheus：拉取指标数据**

docker run -it -d --net fixed3 --ip 192.168.57.4 \
    -v /usr/canal/prom/prometheus.yml:/etc/prometheus/prometheus.yml \
    --name prom prom/prometheus
    
    
    *****************
    prometheus.yml：配置文件
    
    # my global config
    global:
      scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
      evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
      # scrape_timeout is set to the global default (10s).
    
    # Alertmanager configuration
    alerting:
      alertmanagers:
      - static_configs:
        - targets:
          # - alertmanager:9093
    
    # Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
    rule_files:
      # - "first_rules.yml"
      # - "second_rules.yml"
    
    # A scrape configuration containing exactly one endpoint to scrape:
    # Here it's Prometheus itself.
    scrape_configs:
      # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
      - job_name: 'prometheus'
    
        # metrics_path defaults to '/metrics'
        # scheme defaults to 'http'.
    
        static_configs:
        - targets: ['192.168.57.4:9090']
    
    # 添加canal serer指标任务
      - job_name: 'canal'
        static_configs:
        - targets: ['192.168.57.3:11112']

**创建 grafana：可视化展示数据**

docker run -it -d --net fixed3 --ip 192.168.57.5 --name grafana grafana/grafana

## \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* ##

**数据展示**

**添加数据源**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzMTYyNQ_size_16_color_FFFFFF_t_70][]

**数据源配置：prometheus**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzMTYyNQ_size_16_color_FFFFFF_t_70 1][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzMTYyNQ_size_16_color_FFFFFF_t_70 2][]

**导入dashboard模板：[canal/conf/metrics/Canal\_instances\_tmpl.json][canal_conf_metrics_Canal_instances_tmpl.json]**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzMTYyNQ_size_16_color_FFFFFF_t_70 3][]

**点击 import**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzMTYyNQ_size_16_color_FFFFFF_t_70 4][]

**将json数据 复制到import via panel json框后，点击load**

**选择数据源prometheus**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzMTYyNQ_size_16_color_FFFFFF_t_70 5][]

**数据展示**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzMTYyNQ_size_16_color_FFFFFF_t_70 6][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzMTYyNQ_size_16_color_FFFFFF_t_70 7][]**说明：由于没有创建canal client，所以client部分显示no data **

## \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* ##

**指标数据说明**

**原始指标数据**

canal_instance：instance基本信息
    canal_instance_transactions：instance接收的transaction数
    canal_instance_subscriptions：instance接受的订阅数
    
    # parser基本指标
    canal_instance_parser_mode：instance解析模式(是否开启parallel解析)
    canal_instance_publish_blocking_time：dump线程提交到异步解析队列的阻塞时间(仅parallel解析模式)
    canal_instance_received_binlog_bytes：instance接收binlog字节数
    
    # sink基本指标
    canal_instance_sink_blocking_time：sink线程put数据至store的阻塞时间
    
    # store基本指标
    canal_instance_store：instance store基本信息
    canal_instance_store_produce_seq：instance store接收到的events sequence number
    canal_instance_store_produce_mem：instance store接收到的所有events占用内存总量
    canal_instance_store_consume_seq：instance store成功消费的events sequence number
    canal_instance_store_consume_mem：instance store成功消费的所有events占用内存总量
    
    # client基本指标
    canal_instance_client_packets：instance client请求次数的计数
    canal_instance_client_bytes：向instance client发送数据包的字节数
    canal_instance_client_empty_batches：向instance client发送get接口的空结果数
    canal_instance_client_request_error：instance client请求失败计数
    canal_instance_client_request_latency：instance client请求的响应延时
    
    
    # put、get、ack基本指标
    canal_instance_put_rows：store put操作完成的table rows
    canal_instance_get_rows：client get请求返回的table rows
    canal_instance_ack_rows：client ack操作释放的table rows
    
    
    # delay基本指标
    canal_instance_traffic_delay：canal server与MySQL master的延时
    canal_instance_put_delay：store put操作events的延时
    canal_instance_get_delay：client get请求返回events的延时
    canal_instance_ack_delay：client ack操作释放events的延时

**\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\***

**监控展示指标**

**basics：基本指标**

destination：监控的instance name
    parallel parser：event parser解析模式
    batch mode：instance存储模式，ITEMSIZE、MEMSIZE
    buffer size：存储数或者存储大小
    
    # 存储数为num
    batch mode: ITEMSIZE
    buffer size：num
    
    # 存储大小为：num * buffer.memunit（默认为1kb）
    batch mode: MEMSIZE
    buffer size：num

**network bandwidth：instance输入、输出网络带宽**

# dump线程读取binlog占用带宽
    inbound：rate(canal_instance_received_binlog_bytes{destination="example"}[2m]) / 1024
    
    # 向client发送数据占用带宽
    outbound：rate(canal_instance_client_bytes{destination="example"}[2m]) / 1024

**delay：延时指标**

master：canal server与MySQL master之间的延时，canal_instance_traffic_delay{destination="example"} / 1000
    
    put：store put操作延时，canal_instance_put_delay{destination="example"} / 1000
    get：cleint get操作延时，canal_instance_get_delay{destination="example"} / 1000
    ack：client ack操作延时，canal_instance_ack_delay{destination="example"} / 1000

**blocking：阻塞指标**

dump：dump线程blocking时间占比，clamp_max(rate(canal_instance_sink_blocking_time{destination="example"}[2m]), 1000) / 10
    sink：sink线程blocking时间占比，clamp_max(rate(canal_instance_publish_blocking_time{destination="example"}[2m]), 1000) / 10
    
    出现下述情况，则原因可能为：
    dump blocking ratio、sink blocking ratio占比高：client消费速度慢
    dump blocking ratio占比高、sink blocking ratio占比低：canal server parser解析数据慢

**TPS（table rows）：行记录put、get、ack指标**

put：store put行记录指标，rate(canal_instance_put_rows{destination="example"}[2m])
    get：client get行记录指标，rate(canal_instance_get_rows{destination="example"}[2m])
    ack：client ack行记录指标，rate(canal_instance_ack_rows{destination="example"}[2m])

**TPS(MySQL Transaction)：canal server处理transaction指标**

transactions：canal server处理transactions指标，rate(canal_instance_transactions{destination="example"}[2m])

**client requests：客户端请求指标**

clientack：client ack数，canal_instance_client_packets{destination="example", packetType="CLIENTACK"}
    get：client get数，canal_instance_client_packets{destination="example", packetType="GET"}
    subscription：客户端订阅数，canal_instance_subscriptions{destination="example",}

**client qps：客户端每秒操作数**

get：client每秒get数，rate(canal_instance_client_packets{destination="example",packetType="GET"}[2m])
    ack：cleint每秒ack数，rate(canal_instance_client_packets{destination="example",packetType="CLIENTACK"}[2m])

**empty packets：客户端空包指标**

empty：每秒空包数，rate(canal_instance_client_empty_batches{destination="example"}[2m])
    noempty：每秒非空包数，rate(canal_instance_client_packets{destination="example", packetType="GET"}[2m])

**response time：客户端请求响应时间指标**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzMTYyNQ_size_16_color_FFFFFF_t_70 8][]

**store remain events：内存中未ack的event数**

events：内存中未ack的event数，canal_instance_store_produce_seq{destination="example"} - canal_instance_store_consume_seq{destination="example"}

**store remain mem：内存中未ack的数据内存占用（kb）**

memsize：内存中未ack的数据内存占用，(canal_instance_store_produce_mem{destination="example"} - canal_instance_store_consume_mem{destination="example"}) / 1024

[https_github.com_alibaba_canal_wiki_Prometheus-QuickStart]: https://github.com/alibaba/canal/wiki/Prometheus-QuickStart
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzMTYyNQ_size_16_color_FFFFFF_t_70]: /images/20220829/627e0df946dc479081a3de6d41d3aaca.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzMTYyNQ_size_16_color_FFFFFF_t_70 1]: /images/20220829/d749e85b934c478fb258454573baf39a.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzMTYyNQ_size_16_color_FFFFFF_t_70 2]: /images/20220829/8c8b6e48f4c8471790a8bca9f85efddb.png
[canal_conf_metrics_Canal_instances_tmpl.json]: https://raw.githubusercontent.com/alibaba/canal/master/deployer/src/main/resources/metrics/Canal_instances_tmpl.json
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzMTYyNQ_size_16_color_FFFFFF_t_70 3]: /images/20220829/0765d19c7ec14c6d9040d22ab8d8a71d.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzMTYyNQ_size_16_color_FFFFFF_t_70 4]: /images/20220829/f6d300b8483a4184b0888387d407f6da.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzMTYyNQ_size_16_color_FFFFFF_t_70 5]: /images/20220829/69de467726e841bd907d912fffa87c84.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzMTYyNQ_size_16_color_FFFFFF_t_70 6]: /images/20220829/ad5b4f44171c4842adf733861b78633f.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzMTYyNQ_size_16_color_FFFFFF_t_70 7]: /images/20220829/cad87bbd22014cff85cbc2fc94c1dd3c.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzMTYyNQ_size_16_color_FFFFFF_t_70 8]: /images/20220829/063d362fa5984f7e803759815c53a8cc.png