【elasticsearch】关于elasticsearch的max_result_window限制问题的解决方式思考

快来打我* 2024-03-23 17:30 52阅读 0赞

> 事情起因：我们使用es作为日志搜索引擎，客户收集到的业务日志非常之大，每次查询后，返回页数较多，由于我们web界面限制每页返回150条，当客户翻到66页之后就会报错。

#### 文章目录 ####

前言

二、实验

1.默认生成20条数据

2.默认查询

3.Search after查询

4.Search after二次查询

总结

--------------------

**前言**

报错信息如下：

*  Elasticsearch limits the search result to 10000 messages. With a page size of 150 messages, you can use the first 66 pages. Unable to perform search query: Elasticsearch exception \[type=illegal\_argument\_exception, reason=Result window is too large, from + size must be less than or equal to: \[10000\] but was \[34050\]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the \[index.max\_result\_window\] index level setting.\].

--------------------

**一、Search after介绍**

使用Search after作为查询，默认情况下我们使用Scroll search作为查询，可以指定From + size 大小的数据批量返回，这是为了防止因为请求过多的结果而导致内存和CPU资源过度消耗。

查询结果的分页可以通过使用`from` and `size` 来完成，但是当达到深度分页时，成本变得令人望而却步。 默认为 10000 是一种保护措施，搜索请求占用的堆内存和时间成正比。 建议使用 [scroll][] API 来实现高效的深度滚动，但滚动上下文成本高昂，而且成本不高，建议将其用于实时用户请求。 **Search after**通过提供实时游标来规避此问题。 这个想法是使用上一页的结果来帮助检索下一页。

## 二、实验 ##

### 1.默认生成20条数据 ###

for i in `seq 1 20`
    do 
        echo 192.168.113.195 -d -P 54000 "zhongguoren $i"
        sleep 1
    done

### 2.默认查询 ###

GET index_1/_search
    {
      "query": {
        "match": {
          "message": "zhongguoren"
        }
      },
      "sort": [
        {
          "timestamp": "asc"
        }
      ]
    }

默认返回20条数据，通过默认查询知识确认，每条信息的流入时间和条数，目前按照1-20进行排序，没有重复的时间戳。

### 3.**Search after**查询 ###

index_1/_search
    {
      "size": 10,
      "query": {
        "match": {
          "message": "zhongguoren"
        }
      },
      "sort": [
        {
          "timestamp": "asc"
        },
        {
          "_id": "asc"
        }
      ]
    }

先查询前10条，这个和之前默认查询有个区别，就是排序时通过timestamp和\_id进行同时排序，在查询结果中得到一个sort字段，值为

"sort": \[ [\-][-]  
1680770417895,  
"a894bf71-d456-11ed-aa52-000c29f6b211"  
\]

查询后结果值为：

{ - 
      "took": 223,
      "timed_out": false,
      "_shards": { - 
        "total": 4,
        "successful": 4,
        "skipped": 0,
        "failed": 0
      },
      "hits": { - 
        "total": { - 
          "value": 20,
          "relation": "eq"
        },
        "max_score": null,
        "hits": [ - 
          { - 
            "_index": "12_149",
            "_type": "_doc",
            "_id": "2df06a80-d456-11ed-aa52-000c29f6b211",
            "_score": null,
            "_source": { - 
              "elap_accounted_message_size": 140,
              "elap_source_input": "636c90d3246ca975ec999b24",
              "streams": [ - 
                "6969696969697379736c6f67",
                "6969696969727379736c6f67"
              ],
              "elap_remote_port": 35834,
              "elap_source_node": "5bb5d963-7fdd-4e2d-bfdf-a5069bb07097",
              "source": "192.168.113.195",
              "elap_message_id": "01GXAVC899ZZXDJXF453F0RVXJ",
              "message": "<5>Apr 6 16:36:52 root: zhongguoren 1",
              "elap_remote_ip": "192.168.113.195",
              "timestamp": "2023-04-06 08:36:52.135"
            },
            "sort": [ - 
              1680770212135,
              "2df06a80-d456-11ed-aa52-000c29f6b211"
            ]
          },
          { - 
            "_index": "12_149",
            "_type": "_doc",
            "_id": "5244ccf0-d456-11ed-aa52-000c29f6b211",
            "_score": null,
            "_source": { - 
              "elap_accounted_message_size": 140,
              "elap_source_input": "636c90d3246ca975ec999b24",
              "streams": [ - 
                "6969696969697379736c6f67",
                "6969696969727379736c6f67"
              ],
              "elap_remote_port": 52885,
              "elap_source_node": "5bb5d963-7fdd-4e2d-bfdf-a5069bb07097",
              "source": "192.168.113.195",
              "elap_message_id": "01GXAVE3XCWZK71D74YVF58PZ2",
              "message": "<5>Apr 6 16:37:53 root: zhongguoren 2",
              "elap_remote_ip": "192.168.113.195",
              "timestamp": "2023-04-06 08:37:53.086"
            },
            "sort": [ - 
              1680770273086,
              "5244ccf0-d456-11ed-aa52-000c29f6b211"
            ]
          },
          { - 
            "_index": "12_149",
            "_type": "_doc",
            "_id": "5e46d610-d456-11ed-aa52-000c29f6b211",
            "_score": null,
            "_source": { - 
              "elap_accounted_message_size": 140,
              "elap_source_input": "636c90d3246ca975ec999b24",
              "streams": [ - 
                "6969696969697379736c6f67",
                "6969696969727379736c6f67"
              ],
              "elap_remote_port": 57417,
              "elap_source_node": "5bb5d963-7fdd-4e2d-bfdf-a5069bb07097",
              "source": "192.168.113.195",
              "elap_message_id": "01GXAVEQFJP3CN17A2NG41YYHY",
              "message": "<5>Apr 6 16:38:13 root: zhongguoren 3",
              "elap_remote_ip": "192.168.113.195",
              "timestamp": "2023-04-06 08:38:13.232"
            },
            "sort": [ - 
              1680770293232,
              "5e46d610-d456-11ed-aa52-000c29f6b211"
            ]
          },
          { - 
            "_index": "12_149",
            "_type": "_doc",
            "_id": "644eb870-d456-11ed-aa52-000c29f6b211",
            "_score": null,
            "_source": { - 
              "elap_accounted_message_size": 140,
              "elap_source_input": "636c90d3246ca975ec999b24",
              "streams": [ - 
                "6969696969697379736c6f67",
                "6969696969727379736c6f67"
              ],
              "elap_remote_port": 52483,
              "elap_source_node": "5bb5d963-7fdd-4e2d-bfdf-a5069bb07097",
              "source": "192.168.113.195",
              "elap_message_id": "01GXAVF1BQNJZ48GW39G48DQNQ",
              "message": "<5>Apr 6 16:38:23 root: zhongguoren 4",
              "elap_remote_ip": "192.168.113.195",
              "timestamp": "2023-04-06 08:38:23.350"
            },
            "sort": [ - 
              1680770303350,
              "644eb870-d456-11ed-aa52-000c29f6b211"
            ]
          },
          { - 
            "_index": "12_149",
            "_type": "_doc",
            "_id": "a562e070-d456-11ed-aa52-000c29f6b211",
            "_score": null,
            "_source": { - 
              "elap_accounted_message_size": 140,
              "elap_source_input": "636c90d3246ca975ec999b24",
              "streams": [ - 
                "6969696969697379736c6f67",
                "6969696969727379736c6f67"
              ],
              "elap_remote_port": 45404,
              "elap_source_node": "5bb5d963-7fdd-4e2d-bfdf-a5069bb07097",
              "source": "192.168.113.195",
              "elap_message_id": "01GXAVJC1YT9SPVWG81CJGBMXG",
              "message": "<5>Apr 6 16:40:12 root: zhongguoren 5",
              "elap_remote_ip": "192.168.113.195",
              "timestamp": "2023-04-06 08:40:12.534"
            },
            "sort": [ - 
              1680770412534,
              "a562e070-d456-11ed-aa52-000c29f6b211"
            ]
          },
          { - 
            "_index": "12_149",
            "_type": "_doc",
            "_id": "a609f5e0-d456-11ed-aa52-000c29f6b211",
            "_score": null,
            "_source": { - 
              "elap_accounted_message_size": 140,
              "elap_source_input": "636c90d3246ca975ec999b24",
              "streams": [ - 
                "6969696969697379736c6f67",
                "6969696969727379736c6f67"
              ],
              "elap_remote_port": 33127,
              "elap_source_node": "5bb5d963-7fdd-4e2d-bfdf-a5069bb07097",
              "source": "192.168.113.195",
              "elap_message_id": "01GXAVJD1Y45HAS4N7ZN2YXWS0",
              "message": "<5>Apr 6 16:40:13 root: zhongguoren 6",
              "elap_remote_ip": "192.168.113.195",
              "timestamp": "2023-04-06 08:40:13.629"
            },
            "sort": [ - 
              1680770413629,
              "a609f5e0-d456-11ed-aa52-000c29f6b211"
            ]
          },
          { - 
            "_index": "12_149",
            "_type": "_doc",
            "_id": "a6a328a0-d456-11ed-aa52-000c29f6b211",
            "_score": null,
            "_source": { - 
              "elap_accounted_message_size": 140,
              "elap_source_input": "636c90d3246ca975ec999b24",
              "streams": [ - 
                "6969696969697379736c6f67",
                "6969696969727379736c6f67"
              ],
              "elap_remote_port": 41301,
              "elap_source_node": "5bb5d963-7fdd-4e2d-bfdf-a5069bb07097",
              "source": "192.168.113.195",
              "elap_message_id": "01GXAVJE1A0ER5S5BZ8DJKW2A6",
              "message": "<5>Apr 6 16:40:14 root: zhongguoren 7",
              "elap_remote_ip": "192.168.113.195",
              "timestamp": "2023-04-06 08:40:14.633"
            },
            "sort": [ - 
              1680770414633,
              "a6a328a0-d456-11ed-aa52-000c29f6b211"
            ]
          },
          { - 
            "_index": "12_149",
            "_type": "_doc",
            "_id": "a73c3451-d456-11ed-aa52-000c29f6b211",
            "_score": null,
            "_source": { - 
              "elap_accounted_message_size": 140,
              "elap_source_input": "636c90d3246ca975ec999b24",
              "streams": [ - 
                "6969696969697379736c6f67",
                "6969696969727379736c6f67"
              ],
              "elap_remote_port": 36916,
              "elap_source_node": "5bb5d963-7fdd-4e2d-bfdf-a5069bb07097",
              "source": "192.168.113.195",
              "elap_message_id": "01GXAVJF0PXMNPGKVZ7YEJ1JNY",
              "message": "<5>Apr 6 16:40:15 root: zhongguoren 8",
              "elap_remote_ip": "192.168.113.195",
              "timestamp": "2023-04-06 08:40:15.637"
            },
            "sort": [ - 
              1680770415637,
              "a73c3451-d456-11ed-aa52-000c29f6b211"
            ]
          },
          { - 
            "_index": "12_149",
            "_type": "_doc",
            "_id": "a7dfc750-d456-11ed-aa52-000c29f6b211",
            "_score": null,
            "_source": { - 
              "elap_accounted_message_size": 140,
              "elap_source_input": "636c90d3246ca975ec999b24",
              "streams": [ - 
                "6969696969697379736c6f67",
                "6969696969727379736c6f67"
              ],
              "elap_remote_port": 42778,
              "elap_source_node": "5bb5d963-7fdd-4e2d-bfdf-a5069bb07097",
              "source": "192.168.113.195",
              "elap_message_id": "01GXAVJG26BRZG09JSSA96S329",
              "message": "<5>Apr 6 16:40:16 root: zhongguoren 9",
              "elap_remote_ip": "192.168.113.195",
              "timestamp": "2023-04-06 08:40:16.708"
            },
            "sort": [ - 
              1680770416708,
              "a7dfc750-d456-11ed-aa52-000c29f6b211"
            ]
          },
          { - 
            "_index": "12_149",
            "_type": "_doc",
            "_id": "a894bf71-d456-11ed-aa52-000c29f6b211",
            "_score": null,
            "_source": { - 
              "elap_accounted_message_size": 141,
              "elap_source_input": "636c90d3246ca975ec999b24",
              "streams": [ - 
                "6969696969697379736c6f67",
                "6969696969727379736c6f67"
              ],
              "elap_remote_port": 53090,
              "elap_source_node": "5bb5d963-7fdd-4e2d-bfdf-a5069bb07097",
              "source": "192.168.113.195",
              "elap_message_id": "01GXAVJH78H4GCQ44KNZQ1MQ0H",
              "message": "<5>Apr 6 16:40:17 root: zhongguoren 10",
              "elap_remote_ip": "192.168.113.195",
              "timestamp": "2023-04-06 08:40:17.895"
            },
            "sort": [ - 
              1680770417895,
              "a894bf71-d456-11ed-aa52-000c29f6b211"
            ]
          }
        ]
      }
    }

### 4.**Search after二次**查询 ###

可以根据sort值带入到后续得search after中，以做到从某点进行再次查询。

GET index_1/_search
    {
      "size": 10,
      "query": {
        "match": {
          "message": "zhongguoren"
        }
      },
      "search_after": [
        1680770417895,
        "a894bf71-d456-11ed-aa52-000c29f6b211"
      ],
      "sort": [
        {
          "timestamp": "asc"
        },
        {
          "_id": "asc"
        }
      ]
    }

返回结果中，得到11-20条数据。

--------------------

## 总结 ##

我也尝试过，直接从16条的sort字段值进行查询，同样可以查询到17-20的值，所以，我们可以进行改造，默认仍然是10000条的max\_result\_window，兼顾小批量数据的查询和搜索，当查询从66页往后时，每次翻页或者跳页我们要能获取到超过10000的最后一条信息的id和时间戳，由此解决es深度查询翻页的问题。

**明天找产品聊聊实现逻辑。**

[scroll]: https://www.elastic.co/guide/en/elasticsearch/reference/7.9/paginate-search-results.html#scroll-search-results
[-]: http://192.168.113.138:9999/#