Skywalking-04：扩展Metric监控信息

以你之姓@ 2022-09-01 14:57 277阅读 0赞

# 扩展 `Metric` 监控信息 #

## 官方文档 ##

[Source and Scope extension for new metrics][]

## 案例：`JVM Thread` 增加 `Metrics`  ##

### 修改 `Thread` 的定义 ###

在 `apm-protocol/apm-network/src/main/proto/language-agent/JVMMetric.proto` 协议文件中覆盖 `message Thread` 的定义

message Thread {
      int64 liveCount = 1;
      int64 daemonCount = 2;
      int64 peakCount = 3;
      int64 deadlocked = 4;
      int64 monitorDeadlocked = 5;
      int64 newThreadCount = 7;
      int64 runnableThreadCount = 8;
      int64 blockedThreadCount = 9;
      int64 waitThreadCount = 10;
      int64 timeWaitThreadCount = 11;
      int64 terminatedThreadCount = 12;
    }

重新构建 `apm-network` 项目

cd apm-protocol/apm-network
    mvn clean package -DskipTests=true

PS：可以安装 `Protocol Buffer Editor` 插件，支持 `Protocol Buffer` 语法

### 修改 `agent core` 中 `Thread Metrics` 的提供类 ###

直接使用如下代码覆盖 `org.apache.skywalking.apm.agent.core.jvm.thread.ThreadProvider` 类

/*
     * Licensed to the Apache Software Foundation (ASF) under one or more
     * contributor license agreements.  See the NOTICE file distributed with
     * this work for additional information regarding copyright ownership.
     * The ASF licenses this file to You under the Apache License, Version 2.0
     * (the "License"); you may not use this file except in compliance with
     * the License.  You may obtain a copy of the License at
     *
     *     http://www.apache.org/licenses/LICENSE-2.0
     *
     * Unless required by applicable law or agreed to in writing, software
     * distributed under the License is distributed on an "AS IS" BASIS,
     * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
     * See the License for the specific language governing permissions and
     * limitations under the License.
     *
     */
    
    package org.apache.skywalking.apm.agent.core.jvm.thread;
    
    import java.lang.management.ManagementFactory;
    import java.lang.management.ThreadInfo;
    import java.lang.management.ThreadMXBean;
    import java.util.Optional;
    
    import org.apache.skywalking.apm.network.language.agent.v3.Thread;
    
    public enum ThreadProvider {
        INSTANCE;
        private final ThreadMXBean threadMXBean;
        private static final long [] EMPTY_DEADLOCKED_THREADS = new long[0];
    
        ThreadProvider() {
            this.threadMXBean = ManagementFactory.getThreadMXBean();
        }
    
        public Thread getThreadMetrics() {
            int newThreadCount = 0;
            int runnableThreadCount = 0;
            int blockedThreadCount = 0;
            int waitThreadCount = 0;
            int timeWaitThreadCount = 0;
            int terminatedThreadCount = 0;
    
            // 基于线程状态信息增加对应状态的线程数
            ThreadInfo[] threadInfos = threadMXBean.getThreadInfo(threadMXBean.getAllThreadIds());
            if (threadInfos != null) {
                for (ThreadInfo threadInfo : threadInfos) {
                    if (threadInfo != null) {
                        switch (threadInfo.getThreadState()) {
                            case NEW:
                                newThreadCount++;
                                break;
                            case RUNNABLE:
                                runnableThreadCount++;
                                break;
                            case BLOCKED:
                                blockedThreadCount++;
                                break;
                            case WAITING:
                                waitThreadCount++;
                                break;
                            case TIMED_WAITING:
                                timeWaitThreadCount++;
                                break;
                            case TERMINATED:
                                terminatedThreadCount++;
                                break;
                            default:
                                break;
                        }
                    } else {
                        /*
                         * If a thread of a given ID is not alive or does not exist,
                         * the corresponding element in the returned array will,
                         * contain null,because is mut exist ,so the thread is terminated
                         */
                        terminatedThreadCount++;
                    }
                }
            }
    
            // 当前存活线程数
            int threadCount = threadMXBean.getThreadCount();
            // deamon线程数
            int daemonThreadCount = threadMXBean.getDaemonThreadCount();
            // 峰值线程数
            int peakThreadCount = threadMXBean.getPeakThreadCount();
            int deadlocked = Optional.ofNullable(threadMXBean.findDeadlockedThreads())
                    .orElse(EMPTY_DEADLOCKED_THREADS).length;
            int monitorDeadlocked = Optional.ofNullable(threadMXBean.findMonitorDeadlockedThreads())
                    .orElse(EMPTY_DEADLOCKED_THREADS).length;
            // 构建一个Thread对象，用于发送Thread Metric信息至OAP
            return Thread.newBuilder().setLiveCount(threadCount)
                    .setDaemonCount(daemonThreadCount)
                    .setPeakCount(peakThreadCount)
                    .setDeadlocked(deadlocked)
                    .setMonitorDeadlocked(monitorDeadlocked)
                    .setNewThreadCount(newThreadCount)
                    .setRunnableThreadCount(runnableThreadCount)
                    .setBlockedThreadCount(blockedThreadCount)
                    .setWaitThreadCount(waitThreadCount)
                    .setTimeWaitThreadCount(timeWaitThreadCount)
                    .setTerminatedThreadCount(terminatedThreadCount)
                    .build();
        }
    
    }

### 修改 `ServiceInstanceJVMThread`  ###

直接使用如下代码覆盖 `org.apache.skywalking.oap.server.core.source.ServiceInstanceJVMThread` 类， `ServiceInstanceJVMThread`继承了 `Source` 抽象类， `Source` 类是 `Skywalking` 中 [oal][] 体系，资源及范围的定义。

### 修改 `JVMSourceDispatcher`  ###

`org.apache.skywalking.oap.server.analyzer.provider.jvm.JVMSourceDispatcher` 是一个 `Source` 分发类，将从 `agent` 接收到的 `JVM` 相关 `Metrics` 拆分成对应的 `Source` 。例如： `ServiceInstanceJVMMemory` 、 `ServiceInstanceJVMThread` 。 修改方法`org.apache.skywalking.oap.server.analyzer.provider.jvm.JVMSourceDispatcher#sendToThreadMetricProcess` 为如下代码：

private void sendToThreadMetricProcess(String service,
                String serviceId,
                String serviceInstance,
                String serviceInstanceId,
                long timeBucket,
                Thread thread) {
            ServiceInstanceJVMThread serviceInstanceJVMThread = new ServiceInstanceJVMThread();
            serviceInstanceJVMThread.setId(serviceInstanceId);
            serviceInstanceJVMThread.setName(serviceInstance);
            serviceInstanceJVMThread.setServiceId(serviceId);
            serviceInstanceJVMThread.setServiceName(service);
            serviceInstanceJVMThread.setLiveCount(thread.getLiveCount());
            serviceInstanceJVMThread.setDaemonCount(thread.getDaemonCount());
            serviceInstanceJVMThread.setPeakCount(thread.getPeakCount());
            serviceInstanceJVMThread.setTimeBucket(timeBucket);
            serviceInstanceJVMThread.setDeadlocked(thread.getDeadlocked());
            serviceInstanceJVMThread.setMonitorDeadlocked(thread.getMonitorDeadlocked());
            serviceInstanceJVMThread.setNewThreadCount(thread.getNewThreadCount());
            serviceInstanceJVMThread.setRunnableThreadCount(thread.getRunnableThreadCount());
            serviceInstanceJVMThread.setBlockedThreadCount(thread.getBlockedThreadCount());
            serviceInstanceJVMThread.setWaitThreadCount(thread.getWaitThreadCount());
            serviceInstanceJVMThread.setTimeWaitThreadCount(thread.getTimeWaitThreadCount());
            serviceInstanceJVMThread.setTerminatedThreadCount(thread.getTerminatedThreadCount());
            sourceReceiver.receive(serviceInstanceJVMThread);
        }

### 在 `java-agent.oal` 增加相关指标 ###

在`oap-server/server-bootstrap/src/main/resources/oal/java-agent.oal` 添加如下语句

// 参考oal语法
    instance_jvm_thread_deadlocked = from(ServiceInstanceJVMThread.deadlocked).longAvg();
    instance_jvm_thread_monitor_deadlocked = from(ServiceInstanceJVMThread.monitorDeadlocked).longAvg();
    instance_jvm_thread_new_thread_count = from(ServiceInstanceJVMThread.newThreadCount).longAvg();
    instance_jvm_thread_runnable_thread_count = from(ServiceInstanceJVMThread.runnableThreadCount).longAvg();
    instance_jvm_thread_blocked_thread_count = from(ServiceInstanceJVMThread.blockedThreadCount).longAvg();
    instance_jvm_thread_wait_thread_count = from(ServiceInstanceJVMThread.waitThreadCount).longAvg();
    instance_jvm_thread_time_wait_thread_count = from(ServiceInstanceJVMThread.timeWaitThreadCount).longAvg();
    instance_jvm_thread_terminated_thread_count = from(ServiceInstanceJVMThread.terminatedThreadCount).longAvg();

### 修改 `apm.yml`  ###

在 `oap-server/server-bootstrap/src/main/resources/ui-initialized-templates/apm.yml` 文件的 `APM` 面板下的 `Instance` 项增加如下配置

{
      "width": 3,
      "title": "JVM Thread Count (Java Service)",
      "height": "250",
      "entityType": "ServiceInstance",
      "independentSelector": false,
      "metricType": "REGULAR_VALUE",
      "queryMetricType": "readMetricsValues",
      "chartType": "ChartLine",
      "metricName": "instance_jvm_thread_live_count, instance_jvm_thread_daemon_count, instance_jvm_thread_peak_count,instance_jvm_thread_deadlocked,instance_jvm_thread_monitor_deadlocked"
    },
    {
      "width": 3,
      "title": "JVM Thread State Count (Java Service)",
      "height": "250",
      "entityType": "ServiceInstance",
      "independentSelector": false,
      "metricType": "REGULAR_VALUE",
      "metricName": "instance_jvm_thread_new_thread_count,instance_jvm_thread_runnable_thread_count,instance_jvm_thread_blocked_thread_count,instance_jvm_thread_wait_thread_count,instance_jvm_thread_time_wait_thread_count,instance_jvm_thread_terminated_thread_count",
      "queryMetricType": "readMetricsValues",
      "chartType": "ChartBar"
    }

如果不清楚添加位置，可以直接使用如下配置，覆盖 `oap-server/server-bootstrap/src/main/resources/ui-initialized-templates/apm.yml`

# Licensed to the Apache Software Foundation (ASF) under one or more
    # contributor license agreements.  See the NOTICE file distributed with
    # this work for additional information regarding copyright ownership.
    # The ASF licenses this file to You under the Apache License, Version 2.0
    # (the "License"); you may not use this file except in compliance with
    # the License.  You may obtain a copy of the License at
    #
    #     http://www.apache.org/licenses/LICENSE-2.0
    #
    # Unless required by applicable law or agreed to in writing, software
    # distributed under the License is distributed on an "AS IS" BASIS,
    # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    # See the License for the specific language governing permissions and
    # limitations under the License.
    
    # UI templates initialized file includes the default template when the SkyWalking OAP starts up at the first time.
    #
    # Also, SkyWalking would detect the existing templates in the database, once they are missing, all templates in this file
    # could be added automatically.
    
    templates:
      - name: "APM"
        # The type includes DASHBOARD, TOPOLOGY_INSTANCE, TOPOLOGY_ENDPOINT.
        # DASHBOARD type templates could have multiple definitions, by using different names.
        # TOPOLOGY_INSTANCE, TOPOLOGY_ENDPOINT type templates should be defined once, as they are used in the topology page only.
        type: "DASHBOARD"
        # Configuration could be defined through UI, and use `export` to format in the standard JSON.
        configuration: |-
          [
            {
              "name": "APM",
              "type": "service",
              "children": [
                {
                  "name": "Global",
                  "children": [
                    {
                      "width": 3,
                      "title": "Services Load",
                      "height": "300",
                      "entityType": "Service",
                      "independentSelector": false,
                      "metricType": "REGULAR_VALUE",
                      "metricName": "service_cpm",
                      "queryMetricType": "sortMetrics",
                      "chartType": "ChartSlow",
                      "parentService": false,
                      "unit": "CPM - calls per minute"
                    },
                    {
                      "width": 3,
                      "title": "Slow Services",
                      "height": "300",
                      "entityType": "Service",
                      "independentSelector": false,
                      "metricType": "REGULAR_VALUE",
                      "metricName": "service_resp_time",
                      "queryMetricType": "sortMetrics",
                      "chartType": "ChartSlow",
                      "parentService": false,
                      "unit": "ms"
                    },
                    {
                      "width": 3,
                      "title": "Un-Health Services (Apdex)",
                      "height": "300",
                      "entityType": "Service",
                      "independentSelector": false,
                      "metricType": "REGULAR_VALUE",
                      "metricName": "service_apdex",
                      "queryMetricType": "sortMetrics",
                      "chartType": "ChartSlow",
                      "parentService": false,
                      "aggregation": "/",
                      "aggregationNum": "10000",
                      "sortOrder": "ASC"
                    },
                    {
                      "width": 3,
                      "title": "Slow Endpoints",
                      "height": "300",
                      "entityType": "Endpoint",
                      "independentSelector": false,
                      "metricType": "REGULAR_VALUE",
                      "metricName": "endpoint_avg",
                      "queryMetricType": "sortMetrics",
                      "chartType": "ChartSlow",
                      "parentService": false,
                      "unit": "ms"
                    },
                    {
                      "width": "6",
                      "title": "Global Response Latency",
                      "height": "280",
                      "entityType": "All",
                      "independentSelector": false,
                      "metricType": "LABELED_VALUE",
                      "metricName": "all_percentile",
                      "queryMetricType": "readLabeledMetricsValues",
                      "chartType": "ChartLine",
                      "metricLabels": "P50, P75, P90, P95, P99",
                      "labelsIndex": "0, 1, 2, 3, 4",
                      "unit": "percentile in ms"
                    },
                    {
                      "width": "6",
                      "title": "Global Heatmap",
                      "height": "280",
                      "entityType": "All",
                      "independentSelector": false,
                      "metricType": "HEATMAP",
                      "unit": "ms",
                      "queryMetricType": "readHeatMap",
                      "chartType": "ChartHeatmap",
                      "metricName": "all_heatmap"
                    }
                  ]
                },
                {
                  "name": "Service",
                  "children": [
                    {
                      "width": 3,
                      "title": "Service Apdex",
                      "height": "200",
                      "entityType": "Service",
                      "independentSelector": false,
                      "metricType": "REGULAR_VALUE",
                      "metricName": "service_apdex",
                      "queryMetricType": "readMetricsValue",
                      "chartType": "ChartNum",
                      "aggregation": "/",
                      "aggregationNum": "10000"
                    },
                    {
                      "width": 3,
                      "title": "Service Avg Response Time",
                      "height": "200",
                      "entityType": "Service",
                      "independentSelector": false,
                      "metricType": "REGULAR_VALUE",
                      "metricName": "service_resp_time",
                      "queryMetricType": "readMetricsValues",
                      "chartType": "ChartLine",
                      "unit": "ms"
                    },
                    {
                      "width": 3,
                      "title": "Successful Rate",
                      "height": "200",
                      "entityType": "Service",
                      "independentSelector": false,
                      "metricType": "REGULAR_VALUE",
                      "metricName": "service_sla",
                      "queryMetricType": "readMetricsValue",
                      "chartType": "ChartNum",
                      "unit": "%",
                      "aggregation": "/",
                      "aggregationNum": "100"
                    },
                    {
                      "width": 3,
                      "title": "Service Load",
                      "height": "200",
                      "entityType": "Service",
                      "independentSelector": false,
                      "metricType": "REGULAR_VALUE",
                      "metricName": "service_cpm",
                      "queryMetricType": "readMetricsValue",
                      "chartType": "ChartNum",
                      "unit": "CPM - calls per minute"
                    },
                    {
                      "width": 3,
                      "title": "Service Apdex",
                      "height": "200",
                      "entityType": "Service",
                      "independentSelector": false,
                      "metricType": "REGULAR_VALUE",
                      "metricName": "service_apdex",
                      "queryMetricType": "readMetricsValues",
                      "chartType": "ChartLine",
                      "aggregation": "/",
                      "aggregationNum": "10000"
                    },
                    {
                      "width": 3,
                      "title": "Service Response Time Percentile",
                      "height": "200",
                      "entityType": "Service",
                      "independentSelector": false,
                      "metricType": "LABELED_VALUE",
                      "metricName": "service_percentile",
                      "queryMetricType": "readLabeledMetricsValues",
                      "chartType": "ChartLine",
                      "unit": "ms",
                      "metricLabels": "P50, P75, P90, P95, P99",
                      "labelsIndex": "0, 1, 2, 3, 4"
                    },
                    {
                      "width": 3,
                      "title": "Successful Rate",
                      "height": "200",
                      "entityType": "Service",
                      "independentSelector": false,
                      "metricType": "REGULAR_VALUE",
                      "metricName": "service_sla",
                      "queryMetricType": "readMetricsValues",
                      "chartType": "ChartLine",
                      "unit": "%",
                      "aggregation": "/",
                      "aggregationNum": "100"
                    },
                    {
                      "width": 3,
                      "title": "Service Load",
                      "height": "200",
                      "entityType": "Service",
                      "independentSelector": false,
                      "metricType": "REGULAR_VALUE",
                      "metricName": "service_cpm",
                      "queryMetricType": "readMetricsValues",
                      "chartType": "ChartLine",
                      "unit": "CPM - calls per minute"
                    },
                    {
                      "width": "4",
                      "title": "Service Instances Load",
                      "height": "280",
                      "entityType": "ServiceInstance",
                      "independentSelector": false,
                      "metricType": "REGULAR_VALUE",
                      "metricName": "service_instance_cpm",
                      "queryMetricType": "sortMetrics",
                      "chartType": "ChartSlow",
                      "parentService": true,
                      "unit": "CPM - calls per minute"
                    },
                    {
                      "width": "4",
                      "title": "Slow Service Instance",
                      "height": "280",
                      "entityType": "ServiceInstance",
                      "independentSelector": false,
                      "metricType": "REGULAR_VALUE",
                      "metricName": "service_instance_resp_time",
                      "queryMetricType": "sortMetrics",
                      "chartType": "ChartSlow",
                      "parentService": true,
                      "unit": "ms"
                    },
                    {
                      "width": "4",
                      "title": "Service Instance Successful Rate",
                      "height": "280",
                      "entityType": "ServiceInstance",
                      "independentSelector": false,
                      "metricType": "REGULAR_VALUE",
                      "metricName": "service_instance_sla",
                      "queryMetricType": "sortMetrics",
                      "chartType": "ChartSlow",
                      "parentService": true,
                      "unit": "%",
                      "aggregation": "/",
                      "aggregationNum": "100",
                      "sortOrder": "ASC"
                    }
                  ]
                },
                {
                  "name": "Instance",
                  "children": [
                    {
                      "width": "3",
                      "title": "Service Instance Load",
                      "height": "250",
                      "entityType": "ServiceInstance",
                      "independentSelector": false,
                      "metricType": "REGULAR_VALUE",
                      "metricName": "service_instance_cpm",
                      "queryMetricType": "readMetricsValues",
                      "chartType": "ChartLine",
                      "unit": "CPM - calls per minute"
                    },
                    {
                      "width": 3,
                      "title": "Service Instance Throughput",
                      "height": "250",
                      "entityType": "ServiceInstance",
                      "independentSelector": false,
                      "metricType": "REGULAR_VALUE",
                      "metricName": "service_instance_throughput_received,service_instance_throughput_sent",
                      "queryMetricType": "readMetricsValues",
                      "chartType": "ChartLine",
                      "unit": "Bytes"
                    },
                    {
                      "width": "3",
                      "title": "Service Instance Successful Rate",
                      "height": "250",
                      "entityType": "ServiceInstance",
                      "independentSelector": false,
                      "metricType": "REGULAR_VALUE",
                      "metricName": "service_instance_sla",
                      "queryMetricType": "readMetricsValues",
                      "chartType": "ChartLine",
                      "unit": "%",
                      "aggregation": "/",
                      "aggregationNum": "100"
                    },
                    {
                      "width": "3",
                      "title": "Service Instance Latency",
                      "height": "250",
                      "entityType": "ServiceInstance",
                      "independentSelector": false,
                      "metricType": "REGULAR_VALUE",
                      "metricName": "service_instance_resp_time",
                      "queryMetricType": "readMetricsValues",
                      "chartType": "ChartLine",
                      "unit": "ms"
                    },
                    {
                      "width": 3,
                      "title": "JVM CPU (Java Service)",
                      "height": "250",
                      "entityType": "ServiceInstance",
                      "independentSelector": false,
                      "metricType": "REGULAR_VALUE",
                      "metricName": "instance_jvm_cpu",
                      "queryMetricType": "readMetricsValues",
                      "chartType": "ChartLine",
                      "unit": "%",
                      "aggregation": "+",
                      "aggregationNum": ""
                    },
                    {
                      "width": 3,
                      "title": "JVM Memory (Java Service)",
                      "height": "250",
                      "entityType": "ServiceInstance",
                      "independentSelector": false,
                      "metricType": "REGULAR_VALUE",
                      "metricName": "instance_jvm_memory_heap, instance_jvm_memory_heap_max,instance_jvm_memory_noheap, instance_jvm_memory_noheap_max",
                      "queryMetricType": "readMetricsValues",
                      "chartType": "ChartLine",
                      "unit": "MB",
                      "aggregation": "/",
                      "aggregationNum": "1048576"
                    },
                    {
                      "width": 3,
                      "title": "JVM GC Time",
                      "height": "250",
                      "entityType": "ServiceInstance",
                      "independentSelector": false,
                      "metricType": "REGULAR_VALUE",
                      "metricName": "instance_jvm_young_gc_time, instance_jvm_old_gc_time",
                      "queryMetricType": "readMetricsValues",
                      "chartType": "ChartLine",
                      "unit": "ms"
                    },
                    {
                      "width": 3,
                      "title": "JVM GC Count",
                      "height": "250",
                      "entityType": "ServiceInstance",
                      "independentSelector": false,
                      "metricType": "REGULAR_VALUE",
                      "queryMetricType": "readMetricsValues",
                      "chartType": "ChartBar",
                      "metricName": "instance_jvm_young_gc_count, instance_jvm_old_gc_count"
                    },
                    {
                      "width": 3,
                      "title": "JVM Thread Count (Java Service)",
                      "height": "250",
                      "entityType": "ServiceInstance",
                      "independentSelector": false,
                      "metricType": "REGULAR_VALUE",
                      "queryMetricType": "readMetricsValues",
                      "chartType": "ChartLine",
                      "metricName": "instance_jvm_thread_live_count, instance_jvm_thread_daemon_count, instance_jvm_thread_peak_count,instance_jvm_thread_deadlocked,instance_jvm_thread_monitor_deadlocked"
                    },
                    {
                      "width": 3,
                      "title": "JVM Thread State Count (Java Service)",
                      "height": "250",
                      "entityType": "ServiceInstance",
                      "independentSelector": false,
                      "metricType": "REGULAR_VALUE",
                      "metricName": "instance_jvm_thread_new_thread_count,instance_jvm_thread_runnable_thread_count,instance_jvm_thread_blocked_thread_count,instance_jvm_thread_wait_thread_count,instance_jvm_thread_time_wait_thread_count,instance_jvm_thread_terminated_thread_count",
                      "queryMetricType": "readMetricsValues",
                      "chartType": "ChartBar"
                    },
                    {
                      "width": 3,
                      "title": "CLR CPU  (.NET Service)",
                      "height": "250",
                      "entityType": "ServiceInstance",
                      "independentSelector": false,
                      "metricType": "REGULAR_VALUE",
                      "metricName": "instance_clr_cpu",
                      "queryMetricType": "readMetricsValues",
                      "chartType": "ChartLine",
                      "unit": "%"
                    },
                    {
                      "width": 3,
                      "title": "CLR GC (.NET Service)",
                      "height": "250",
                      "entityType": "ServiceInstance",
                      "independentSelector": false,
                      "metricType": "REGULAR_VALUE",
                      "metricName": "instance_clr_gen0_collect_count, instance_clr_gen1_collect_count, instance_clr_gen2_collect_count",
                      "queryMetricType": "readMetricsValues",
                      "chartType": "ChartBar"
                    },
                    {
                      "width": 3,
                      "title": "CLR Heap Memory (.NET Service)",
                      "height": "250",
                      "entityType": "ServiceInstance",
                      "independentSelector": false,
                      "metricType": "REGULAR_VALUE",
                      "metricName": "instance_clr_heap_memory",
                      "queryMetricType": "readMetricsValues",
                      "chartType": "ChartLine",
                      "unit": "MB",
                      "aggregation": "/",
                      "aggregationNum": "1048576"
                    },
                    {
                      "width": 3,
                      "title": "CLR Thread (.NET Service)",
                      "height": "250",
                      "entityType": "ServiceInstance",
                      "independentSelector": false,
                      "metricType": "REGULAR_VALUE",
                      "queryMetricType": "readMetricsValues",
                      "chartType": "ChartLine",
                      "metricName": "instance_clr_available_completion_port_threads,instance_clr_available_worker_threads,instance_clr_max_completion_port_threads,instance_clr_max_worker_threads"
                    }
                  ]
                },
                {
                  "name": "Endpoint",
                  "children": [
                    {
                      "width": "4",
                      "title": "Endpoint Load in Current Service",
                      "height": "280",
                      "entityType": "Endpoint",
                      "independentSelector": false,
                      "metricType": "REGULAR_VALUE",
                      "metricName": "endpoint_cpm",
                      "queryMetricType": "sortMetrics",
                      "chartType": "ChartSlow",
                      "parentService": true,
                      "unit": "CPM - calls per minute"
                    },
                    {
                      "width": "4",
                      "title": "Slow Endpoints in Current Service",
                      "height": "280",
                      "entityType": "Endpoint",
                      "independentSelector": false,
                      "metricType": "REGULAR_VALUE",
                      "queryMetricType": "sortMetrics",
                      "chartType": "ChartSlow",
                      "metricName": "endpoint_avg",
                      "unit": "ms",
                      "parentService": true
                    },
                    {
                      "width": "4",
                      "title": "Successful Rate in Current Service",
                      "height": "280",
                      "entityType": "Endpoint",
                      "independentSelector": false,
                      "metricType": "REGULAR_VALUE",
                      "metricName": "endpoint_sla",
                      "queryMetricType": "sortMetrics",
                      "chartType": "ChartSlow",
                      "aggregation": "/",
                      "aggregationNum": "100",
                      "parentService": true,
                      "unit": "%",
                      "sortOrder": "ASC"
                    },
                    {
                      "width": 3,
                      "title": "Endpoint Load",
                      "height": 350,
                      "entityType": "Endpoint",
                      "independentSelector": false,
                      "metricType": "REGULAR_VALUE",
                      "metricName": "endpoint_cpm",
                      "queryMetricType": "readMetricsValues",
                      "chartType": "ChartLine"
                    },
                    {
                      "width": 3,
                      "title": "Endpoint Avg Response Time",
                      "height": 350,
                      "entityType": "Endpoint",
                      "independentSelector": false,
                      "metricType": "REGULAR_VALUE",
                      "metricName": "endpoint_avg",
                      "queryMetricType": "readMetricsValues",
                      "chartType": "ChartLine",
                      "unit": "ms"
                    },
                    {
                      "width": 3,
                      "title": "Endpoint Response Time Percentile",
                      "height": 350,
                      "entityType": "Endpoint",
                      "independentSelector": false,
                      "metricType": "LABELED_VALUE",
                      "metricName": "endpoint_percentile",
                      "queryMetricType": "readLabeledMetricsValues",
                      "chartType": "ChartLine",
                      "metricLabels": "P50, P75, P90, P95, P99",
                      "labelsIndex": "0, 1, 2, 3, 4",
                      "unit": "ms"
                    },
                    {
                      "width": 3,
                      "title": "Endpoint Successful Rate",
                      "height": 350,
                      "entityType": "Endpoint",
                      "independentSelector": false,
                      "metricType": "REGULAR_VALUE",
                      "metricName": "endpoint_sla",
                      "queryMetricType": "readMetricsValues",
                      "chartType": "ChartLine",
                      "unit": "%",
                      "aggregation": "/",
                      "aggregationNum": "100"
                    }
                  ]
                }
              ]
            }
          ]
        # Activated as the DASHBOARD type, makes this templates added into the UI page automatically.
        # False means providing a basic template, user needs to add it manually.
        activated: true
        # True means wouldn't show up on the dashboard. Only keeps the definition in the storage.
        disabled: false

### 效果展示 ###

![file][]

![file][file 1]

# 代码贡献 #

*  [Add some new thread metric and class metric to JVMMetric \#7230][Add some new thread metric and class metric to JVMMetric _7230]
 *  [add some new thread metric and class metric to JVMMetric \#52][add some new thread metric and class metric to JVMMetric _52]
 *  [Remove Terminated State and New State in JVMMetric (\#7230) \#53][Remove Terminated State and New State in JVMMetric _7230_ _53]
 *  [Add some new thread metric and class metric to JVMMetric (\#7230) \#7243][Add some new thread metric and class metric to JVMMetric _7230_ _7243]

# 总结 #

`Metric` 如何扩展，网上基本上没案例，都是看官方文档和源码了解。对应这种非常热门的开源项目，还是看官方文档和源码更稳。

# 参考文档 #

*  [Java ManagementFactory解析][Java ManagementFactory]
 *  [编程中使用ThreadMXBean类来检测死锁][ThreadMXBean]
 *  [Source and Scope extension for new metrics][]
 *  [Observability Analysis Language][]

> 分享并记录所学所见

[Source and Scope extension for new metrics]: https://skywalking.apache.org/docs/main/v8.4.0/en/guides/source-extension
[oal]: https://skywalking.apache.org/docs/main/v8.4.0/en/concepts-and-designs/oal/
[file]: /images/20220829/ed1d4a07e7014fa6901d191067565311.png
[file 1]: /images/20220829/8558c6a157a140a2973f18913de3973c.png
[Add some new thread metric and class metric to JVMMetric _7230]: https://github.com/apache/skywalking/issues/7230
[add some new thread metric and class metric to JVMMetric _52]: https://github.com/apache/skywalking-data-collect-protocol/pull/52
[Remove Terminated State and New State in JVMMetric _7230_ _53]: https://github.com/apache/skywalking-data-collect-protocol/pull/53
[Add some new thread metric and class metric to JVMMetric _7230_ _7243]: https://github.com/apache/skywalking/pull/7243
[Java ManagementFactory]: https://www.jianshu.com/p/5d854245051d
[ThreadMXBean]: https://blog.csdn.net/beckham1984/article/details/44419855
[Observability Analysis Language]: https://skywalking.apache.org/docs/main/v8.4.0/en/concepts-and-designs/oal