测试es from size error，from + size must be less than or equal to: [10000]异常问题-蒲公英云

测试es from size error，from + size must be less than or equal to: [10000]异常问题

清疚 2023-02-12 04:28 133阅读 0赞

测试案例

1，查询第三页每页3334

{
“page”:3,
“size”:3334
}

2，查询第一页，每页100001

{
“page”:1,
“size”:100001
}

3，查询第26页，每页200条

{
“page”:26,
“size”:200
}

以上查询都会报异常。

控制方案

如何控制发生异常，常见如

控制每页数量不大于10000，但是如情况1，3还是会发生异常。

1，可能会问，es默认最大查询量10000，每页小于10000啊。但是es分页，会首先取到所有数据，再分页

总量3*3334和26*200，总量都大于10000，所以异常。

2，虽然实际情况每页不会3334这么大，会像第3种，每页会查询200，然后前端对20条数据再次分页，当翻页总量大于200，后台es才会查询下一页。实际的操作也不会翻这么多页（前端分页200/size*26）。但是还是存在异常的可能性。以上两种只是控制

每页数据量10000以内，不能避免发生异常的可能性。

3，控制查询总量，有没有意义，比如我只限制查询总量5000条或者10000条

我觉得没意义，首先比如查询到100万条数据，返回总量肯定100万，你告诉查询5000条或者10000条肯定是错的；其次你也没法保证查询的总量是5000或1万。

推荐：使用search_after进行分页，参考：

https://mp.csdn.net/console/editor/html/106314361

https://blog.csdn.net/zzh920625/article/details/84593590

代码如下

@Override
  public Rpage<ResShopProductExtensionInfoDataVO> selectList(ReqShopProductExtensionInfoSearchVO req) {
    try {
      SearchRequest searchRequest = new SearchRequest(EsOperateTables.SHOP_PRODUCT_EXTENSION_INFO.getAlias());
      SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
      List<String> fields = Arrays.stream(ResShopProductExtensionInfoVO.class.getDeclaredFields())
          /*将类属性名转为es中带下划线的字段名*/
          .map(field -> BizStringUtils.humpToLine(field.getName()))
          .collect(Collectors.toList());
      sourceBuilder.fetchSource(fields.toArray(new String[fields.size()]), null);
      BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery();
      ShopProductExtensionInfoConverter.setSomeCondition(req, sourceBuilder, boolQueryBuilder);
      sourceBuilder.size(req.getSize() > 10000 ? 10000 : req.getSize());
      sourceBuilder.from((req.getPage() - 1) * req.getSize());
      searchRequest.source(sourceBuilder);
      log.info("terms查询DSL语句：" + searchRequest.toString());
      SearchResponse searchResponse = restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT);
      long totalHits = 0L;
      if (searchResponse != null) {
        totalHits = searchResponse.getHits().getTotalHits();
      }
      long maxCount = new Long(ConstantsUtils.QueryPartams.MAX_RESULT_COUNT).intValue();//100000
      long totalCount = totalHits;
      totalHits = totalHits <= maxCount ? totalHits : maxCount;
      List<Map<String, Object>> mapList = Lists.newArrayListWithCapacity(searchResponse.getHits().getHits().length);//每页200实际查询的量
      Arrays.stream(searchResponse.getHits().getHits()).forEach(hit -> mapList.add(hit.getSourceAsMap()));
      YhPageBase page = new YhPageBase(mapList, totalHits, req.getSize().longValue(), req.getPage().longValue());
      List<ResShopProductExtensionInfoVO> resList = new ArrayList<>();
      if (CollectionUtils.isNotEmpty(page.getList())) {
        resList = JSONObject.parseArray(JSONObject.toJSONString(page.getList()), ResShopProductExtensionInfoVO.class);
      }
      Page rpage = new Page();
      rpage.setTotalNum((int) totalCount);//总量
      rpage.setPageSize(req.getSize());
      rpage.setPageNo(req.getPage());
      rpage.setResult(adapter(resList, req));//每页实际查询量
      return Rpage.success(rpage);
    } catch (Exception e) {
      log.error("查询出单方式配置es信息出错", e);
    }
    return Rpage.success(new Page<>());
  }