ES 按照每隔几分钟，几小时，几天统计折线图

旧城等待， 2022-06-02 07:37 309阅读 0赞

公司应用es有一段时间，今天接触了一个相对复杂的业务，针对每隔几分钟，几小时，几天进行统计折线图，具体逻辑如下图：

![SouthEast][]

如图，系统必须要支持查询，每小时（每隔10分钟），每日（每隔4小时统计），每周（每隔1日），每月（每隔5日）进行统计，找到最大值显示到折线图上。

首先4张图像使用term聚合，每张图像上有两条线，表示cpu和内存，也属于term聚合方式，整个折线图采用dateHistogram聚合方式。

使用语句如下：

GET /system-audit1/auditEvent/_search
    {
      "aggs": {
        "sales": {
          "terms": {
            "field": "psName.keyword"
          },"aggs": {
            "type": {
              "terms": {
                "field": "type.keyword"
              },"aggs": {
            "staticTime": {
              "date_histogram": {
                "field": "statisticTime",
                "interval": "4h"
              },"aggs": {
                "maxValue": {
                  "max": {
                    "field": "value"
                  }
                }
              }
            }
          }
            }
          }
        }
      }
    }

执行结果：

"aggregations": {
        "sales": {
          "doc_count_error_upper_bound": 0,
          "sum_other_doc_count": 0,
          "buckets": [
            {
              "key": "192.168.1.241:es",
              "doc_count": 7516,
              "type": {
                "doc_count_error_upper_bound": 0,
                "sum_other_doc_count": 0,
                "buckets": [
                  {
                    "key": "cpu",
                    "doc_count": 3763,
                    "staticTime": {
                      "buckets": [
                        {
                          "key_as_string": "2018-01-05T16:00:00.000Z",
                          "key": 1515168000000,
                          "doc_count": 2067,
                          "maxValue": {
                            "value": 23.100000381469727
                          }
                        },
                        {
                          "key_as_string": "2018-01-05T20:00:00.000Z",
                          "key": 1515182400000,
                          "doc_count": 132,
                          "maxValue": {
                            "value": 22.799999237060547
                          }
                        },
                        {
                          "key_as_string": "2018-01-06T00:00:00.000Z",
                          "key": 1515196800000,
                          "doc_count": 0,
                          "maxValue": {
                            "value": null
                          }
                        }...

java代码实现：

List<SystemDistribution> list = new ArrayList<>(); //统计最终的数据
    BoolQueryBuilder boolQueryBuilder=QueryBuilders.boolQuery();
    boolQueryBuilder.must(QueryBuilders.rangeQuery("createTime").lte(endTime).gt(startTime)); //createTime是YYYYMMDDHHMMSSSSS格式字符串
    DateHistogramInterval dateHistogramInterval=getDateHistogramInterval(timeType); //聚合时间类型
    TermsAggregationBuilder termAggregation=AggregationBuilders.terms("psName").field("psName.keyword"); //服务器名称聚合
    TermsAggregationBuilder typeAggregation=AggregationBuilders.terms("type").field("type.keyword");
    AggregationBuilder timeAggregation =
            AggregationBuilders
                    .dateHistogram("agg")
                    .field("statisticTime")//统计时间聚合
                    .dateHistogramInterval(dateHistogramInterval);
    MaxAggregationBuilder maxAggregation = AggregationBuilders.max("maxValue").field("value");//最大值聚合
    timeAggregation.subAggregation(maxAggregation);
    typeAggregation.subAggregation(timeAggregation);
    termAggregation.subAggregation(typeAggregation);
    SearchResponse response = client.prepareSearch(INDEX_NAME).setTypes(TYPE)
        .setQuery(boolQueryBuilder).addAggregation(termAggregation).execute().actionGet();
    Terms genders = response.getAggregations().get("psName");
    for (Terms.Bucket entry : genders.getBuckets()) {
      SystemDistribution systemDistribution=new SystemDistribution();
      String psName=entry.getKey().toString();
      systemDistribution.setHostName(psName);
      Terms typeTerm = entry.getAggregations().get("type");
      List<RiskStatisticsVo> memRiskStatistics=new ArrayList<>();
      List<RiskStatisticsVo> cpuRiskStatisTics=new ArrayList<>();
      for (Terms.Bucket entry1 : typeTerm.getBuckets()) {
        String type = entry1.getKeyAsString(); // Key as String  2017-12-27T00:00:00.000Z
        Histogram histogram=entry1.getAggregations().get("agg");
        for(Histogram.Bucket entry2 : histogram.getBuckets()){
          RiskStatisticsVo riskStatisticsVo=new RiskStatisticsVo();
          riskStatisticsVo.setRiskType(type);
          String statisTime=entry2.getKeyAsString();
          Max max=entry2.getAggregations().get("maxValue");
          Double maxValue=max.getValue();
          if(maxValue.equals(Double.NEGATIVE_INFINITY)){ //如果为无穷大，赋值为0
            maxValue=0.0;
          }//-Infinity
          riskStatisticsVo.setStatisticTime(formatReturnTime(statisTime,timeType));//2018-01-08T11:00:00.000Z
          riskStatisticsVo.setCount(maxValue.toString());
          if("mem".equals(type)){
            memRiskStatistics.add(riskStatisticsVo);
          }else{
            cpuRiskStatisTics.add(riskStatisticsVo);
          }
        }
      }
      systemDistribution.setCpuStatisticList(cpuRiskStatisTics);
      systemDistribution.setEmeStatisticList(memRiskStatistics);
      list.add(systemDistribution);
    }
    return list;

获得时间类型：

private DateHistogramInterval getDateHistogramInterval(String dateType) {
        if(StatisticTimeTypeEnum.HOUR.getName().equals(dateType)){
          return DateHistogramInterval.minutes(10);//统计一个小时内数据，每隔10分钟一个显示
        }else if(StatisticTimeTypeEnum.Day.getName().equals(dateType)){
          return DateHistogramInterval.hours(4); //统计每日，每隔4小时统计
        }else if(StatisticTimeTypeEnum.WEEK.getName().equals(dateType)){
          return DateHistogramInterval.days(1); //每周，统计每天的数据统计
        }else{
          return DateHistogramInterval.days(5); //每月，每隔5天一个统计数据
        }
      }

格式化返回时间：

private String formatReturnTime(String time,String dateType){
        if(StatisticTimeTypeEnum.HOUR.getName().equals(dateType)){
          return time.substring(11,16);
        }else if(StatisticTimeTypeEnum.Day.getName().equals(dateType)){
          return time.substring(8,10)+"日"+time.substring(11,13)+"时";
        }else{
          return time.substring(8,10)+"日";
        }
      }

[SouthEast]: /images/20220602/406b280885bd4387b10bdbd19e898d44.png