【ElasticSearch】ES自动补全查询与Java接口实现

港控/mmm° 2023-10-13 14:27 5阅读 0赞

#### 文章目录 ####

*  1、安装拼音分词器
 *  2、自定义分词器
 *  3、completion suggester查询
 *  4、hotel索引库更新
 *  5、代码修改
 *  6、RestAPI实现自动补全
 *  7、需求：搜索框实现自动补全

自动补全就是当用户在搜索框输入字符时，我们应该提示出与该字符有关的搜索项。

![在这里插入图片描述][cca22b3280ee44cd9ae5ee183857a405.png]

## 1、安装拼音分词器 ##

要实现根据字母做补全，就必须对文档按照拼音分词。GitHub上有相关插件，地址：[https://github.com/medcl/elasticsearch-analysis-pinyin][https_github.com_medcl_elasticsearch-analysis-pinyin]，下载和ES对应的版本。

**安装步骤：**

*  解压

![在这里插入图片描述][13a18e06243d43f1985ff77bb8d756b7.png]

*  上传到虚拟机中，elasticsearch的plugin目录

![在这里插入图片描述][08cf0cf9f75141c9ad938a3959ade24c.png]

docker volume ls
    docker inspect volumeXXX

*  重启elasticsearch的容器

docker restar es

*  测试

![在这里插入图片描述][c28324248d224345b628b5a6529ef0b9.png]

但上面的拼音分词器也有很明显的缺陷，那就是没有进行词条切割，且汉字没了。

## 2、自定义分词器 ##

ES的分词器由三部分组成：

> **character filters**：在tokenizer之前对文本进行处理。例如删除字符、替换字符

> **tokenizer**：将文本按照一定的规则切割成词条（term）。例如keyword，就是不分词；还有ik\_smart

> **tokenizer filter**：将tokenizer输出的词条做进一步处理。例如大小写转换、同义词处理、拼音处理等

举个例子：

![在这里插入图片描述][3ae98bc9ebf14682886a26f29703ffd4.png]  
**在创建索引库时，** 可以通过settings来配置自定义的analyzer（分词器）：

PUT /test
    {
        
      "settings": {
        
        "analysis": {
        
          "analyzer": {
         // 自定义分词器
            "my_analyzer": {
          // 分词器名称
              "tokenizer": "ik_max_word",
              "filter": "pinyin"
            }
          }
         }
      }
    }
    # 不需要替换或者删除，就不加character filter了

再改下定义tokenizer filter时的各种属性：

PUT /test
    {
        
      "settings": {
        
        "analysis": {
        
          "analyzer": {
         // 自定义分词器
            "my_analyzer": {
          // 分词器名称
              "tokenizer": "ik_max_word",
              "filter": "py"
            }
          },
          "filter": {
         // 自定义tokenizer filter
            "py": {
         // 过滤器名称
              "type": "pinyin", // 过滤器类型，这里是pinyin
    		  "keep_full_pinyin": false,  //解决全分为单个字的问题
              "keep_joined_full_pinyin": true,  //全拼
              "keep_original": true,  //是否保留中文
              "limit_first_letter_length": 16,
              "remove_duplicated_term": true,
              "none_chinese_pinyin_tokenize": false
            }
          }
        }
      }
    }

![在这里插入图片描述][a970499fc44b4d148ea4faf77bef26f3.png]

测试下效果：

![在这里插入图片描述][46e9618ae78744baaefb4b77bc862dcd.png]

插入两条name同音文档的后，搜索：

![在这里插入图片描述][1fa0c2cad5154ad183b5be5a9570f162.png]

但此时搜一下中文看看：

![在这里插入图片描述][622aa5e262784214b22bddeeedf72f68.png]

很明显有问题，别人问狮子，你连同音词虱子都返回了。

拼音分词器适合在创建倒排索引时使用，但不能在搜索的时候使用。

创建倒排索引时，如下图：

![在这里插入图片描述][84fc5fb2e1844e31b58f67f085fa3a61.png]

但当使用拼音分词器来搜索时：

![在这里插入图片描述][f729335fc704400dbb582540c5902141.png]

要解决这个问题，可以使用两个分词器：

"analyzer": "my_analyzer",   # 创建倒排索引时使用
    "search_analyzer": "ik_smart" # 搜索时使用

PUT /test
    {
        
      "settings": {
        
        "analysis": {
        
          "analyzer": {
        
            "my_analyzer": {
        
              "tokenizer": "ik_max_word", "filter": "py"
            }
          },
          "filter": {
        
            "py": {
         ... }
          }
        }
      },
      "mappings": {
        
        "properties": {
        
          "name": {
        
            "type": "text",
            "analyzer": "my_analyzer",
            "search_analyzer": "ik_smart"  //!!!!
          }
        }
      }
    }

![在这里插入图片描述][161ab6956407424c95d44cbe3b725c61.png]

重新测试下搜索中文的场景：

![在这里插入图片描述][1459fb82c536445289e2964e846e29ad.png]  
小结：

![在这里插入图片描述][e3e107af6bbe4b1b996ba1e76ae4e0ff.png]

## 3、completion suggester查询 ##

ES提供[Completion Suggester][]查询 来实现自动补全功能，该查询会匹配以用户输入内容开头的词条并返回。此时，文档中字段的类型也有特殊要求：

*  参与补全查询的字段必须是completion类型。
 *  字段的内容一般是用来补全的多个词条形成的数组

// 创建索引库
    PUT test
    {
        
      "mappings": {
        
        "properties": {
        
          "title":{
        
            "type": "completion"   //注意字段类型为completion
          }
        }
      }
    }

// 示例数据
    POST test/_doc
    {
        
      "title": ["Sony", "WH-1000XM3"]
    }
    POST test/_doc
    {
        
      "title": ["SK-II", "PITERA"]
    }
    POST test/_doc
    {
        
      "title": ["Nintendo", "switch"]
    }

**completion suggester查询语法：**

// 自动补全查询
    GET /test/_search
    {
        
      "suggest": {
         //查询类型，用suggest 
        "title_suggest": {
         //给你的suggest查询起个名
          "text": "s", // 用户输入的关键字
          "completion": {
        
            "field": "title", // 补全查询的字段
            "skip_duplicates": true, // 跳过重复的
            "size": 10 // 获取前10条结果
          }
        }
      }
    }

运行DSL：

![在这里插入图片描述][13eff94d616f49dc868ab16304e5af04.png]

## 4、hotel索引库更新 ##

看下之前的索引库的结构：

![在这里插入图片描述][261139b763794370a26cb6be6364b1ba.png]  
执行DSL来更新hotel索引库：

PUT /hotel
    {
        
      "settings": {
        
        "analysis": {
        
          "analyzer": {
        
            "text_anlyzer": {
          //定义第一个分词器
              "tokenizer": "ik_max_word",  //切割用ik_max
              "filter": "py"  //转换用拼音
            },
            "completion_analyzer": {
           //定义第二个分词器，用于自动补全，不分词，直接转拼音
              "tokenizer": "keyword",  //分词用keyword，因为参与自动补全的是一个个词条，这些词条放在数组当中，本身就是个词条
              "filter": "py"
            }
          },
          "filter": {
          //定义上面的拼音filter
            "py": {
        
              "type": "pinyin",
              "keep_full_pinyin": false,
              "keep_joined_full_pinyin": true,
              "keep_original": true,
              "limit_first_letter_length": 16,
              "remove_duplicated_term": true,
              "none_chinese_pinyin_tokenize": false
            }
          }
        }
      },
      "mappings": {
        
        "properties": {
        
          "id":{
        
            "type": "keyword"
          },
          "name":{
        
            "type": "text",
            "analyzer": "text_anlyzer",  //用来创建倒排索引时分词
            "search_analyzer": "ik_smart",  //用来全文检索
            "copy_to": "all"
          },
          "address":{
        
            "type": "keyword",
            "index": false
          },
          "price":{
        
            "type": "integer"
          },
          "score":{
        
            "type": "integer"
          },
          "brand":{
        
            "type": "keyword",
            "copy_to": "all"
          },
          "city":{
        
            "type": "keyword"
          },
          "starName":{
        
            "type": "keyword"
          },
          "business":{
        
            "type": "keyword",
            "copy_to": "all"
          },
          "location":{
        
            "type": "geo_point"
          },
          "pic":{
        
            "type": "keyword",
            "index": false
          },
          "all":{
        
            "type": "text",
            "analyzer": "text_anlyzer",  //倒排索引分词
            "search_analyzer": "ik_smart"  //搜索分词
          },
          "suggestion":{
           //新加这个字段，用来做自动补全
              "type": "completion",  //类型为completion
              "analyzer": "completion_analyzer"  //不分词，直接转拼音
          }
        }
      }
    }

上面实现了：

*  修改hotel索引库结构，设置自定义拼音分词器
 *  修改索引库的name、all字段，使用自定义分词器
 *  索引库添加一个新字段suggestion，类型为completion类型，使用自定义的分词器

## 5、代码修改 ##

上面索引库更新后，上一节中的代码也要发生修改：

*  给HotelDoc类添加suggestion字段，内容包含brand、business
 *  重新导入数据到hotel库

//HotelDoc类修改
    @Data
    @NoArgsConstructor
    public class HotelDoc {
        
        private Long id;
        private String name;
        private String address;
        private Integer price;
        private Integer score;
        private String brand;
        private String city;
        private String starName;
        private String business;
        private String location;
        private String pic;
        //距离
        private Object distance;
        //是否充广告
        private Boolean isAD;
        //ES中的completion，后面存数组，这里可以对应成List
        private List<String> suggestion;
    
        public HotelDoc(Hotel hotel) {
        
            this.id = hotel.getId();
            this.name = hotel.getName();
            this.address = hotel.getAddress();
            this.price = hotel.getPrice();
            this.score = hotel.getScore();
            this.brand = hotel.getBrand();
            this.city = hotel.getCity();
            this.starName = hotel.getStarName();
            this.business = hotel.getBusiness();
            this.location = hotel.getLatitude() + ", " + hotel.getLongitude();
            this.pic = hotel.getPic();
            this.suggestion = Arrays.asList(this.brand,this.business);
    
        }
    }

注意上面的`Array.asList方法，使suggestion字段内容包含brand、business`

//运行从MySQL读数据，插入文档到ES的单元测试代码
    @SpringBootTest
    public class HotelDocumentTest {
        
    
        @Resource
        IHotelService iHotelService;
    
        private RestHighLevelClient client;
    
        @Test
        void testInit(){
        
    
            System.out.println(client);
        }
    
        @BeforeEach
        void setUp(){
        
            this.client = new RestHighLevelClient(RestClient.builder(
                    HttpHost.create("http://10.4.130.220:9200")
            ));
        }
    
        @AfterEach
        void tearDown() throws IOException {
        
            this.client.close();
        }
    
    
        @Test
        void testBulk() throws IOException {
        
            List<Hotel> hotels = iHotelService.list();
            BulkRequest request = new BulkRequest();
            for(Hotel hotel : hotels){
        
                HotelDoc hotelDoc = new HotelDoc(hotel);
                request.add(new IndexRequest("hotel")
                        .id(hotelDoc.getId().toString())
                        .source(JSON.toJSONString(hotelDoc),XContentType.JSON)
                );
            }
            client.bulk(request,RequestOptions.DEFAULT);
    
    
    
        }
    
    
    }

查看下文档数据：

![在这里插入图片描述][4c3e3e617b7649e3b1dc644235242a9f.png]

在suggestion字段发现有数据的商业区有多个：

![在这里插入图片描述][b7a2399c0072477db013a61cb95ae584.png]

修改下HotelDoc的有参构造，加判断逻辑，business字段有斜杠/时，分开后再放入suggestion

@Data
    @NoArgsConstructor
    public class HotelDoc {
        
        private Long id;
        private String name;
        private String address;
        private Integer price;
        private Integer score;
        private String brand;
        private String city;
        private String starName;
        private String business;
        private String location;
        private String pic;
        //距离
        private Object distance;
        //是否充广告
        private Boolean isAD;
        //ES中的completion，后面存数组，这里可以对应成List
        private List<String> suggestion;
    
        public HotelDoc(Hotel hotel) {
        
            this.id = hotel.getId();
            this.name = hotel.getName();
            this.address = hotel.getAddress();
            this.price = hotel.getPrice();
            this.score = hotel.getScore();
            this.brand = hotel.getBrand();
            this.city = hotel.getCity();
            this.starName = hotel.getStarName();
            this.business = hotel.getBusiness();
            this.location = hotel.getLatitude() + ", " + hotel.getLongitude();
            this.pic = hotel.getPic();
            if(this.business.contains("/")){
        
                //此时business有多个值，需要分开后放入suggestion
                String[] arr = this.business.split("/");
                //添加元素
                this.suggestion = new ArrayList<>();
                Collections.addAll(this.suggestion,arr);
                this.suggestion.add(this.brand);
            }else{
        
                this.suggestion = Arrays.asList(this.brand,this.business);
            }
    
        }
    }

重新运行单元测试，插入文档数据，可以看到切割完成了：

![在这里插入图片描述][2ef9c3dac0cc427ebb95677b33eb5e2e.png]

执行自动补全查询的DSL，比如搜索h：

![在这里插入图片描述][258ca906fee3421b842e1a7670d668e9.png]

## 6、RestAPI实现自动补全 ##

Java代码对比DSL来看，查询的实现是：

![在这里插入图片描述][ed186f897f0842649f689e61a82bd85d.png]

对响应结果的处理是：

![在这里插入图片描述][29da20d18c674a478f3660ec7f4f7a5c.png]

在单元测试中使用一下先：

@Test
        void testSuggest() throws IOException {
        
            //1、准备Request
            SearchRequest request = new SearchRequest("hotel");
            //2、准备DSL
            request.source()
                    .suggest(new SuggestBuilder().addSuggestion(
                            "mySuggestion",
                            SuggestBuilders.completionSuggestion("suggestion")
                                    .prefix("h")  //搜索的关键字，这里用prefix,即前置，给方法起名很灵性
                                    .skipDuplicates(true)
                                    .size(10)
                    ));
            //3、发起请求
            SearchResponse response = client.search(request,RequestOptions.DEFAULT);
            //4、解析结果
            Suggest suggest = response.getSuggest();
            //4.1 根据不全查询名称，获取查询结果
            CompletionSuggestion suggestion = suggest.getSuggestion("mySuggestion");
            //4.2 获取options
            List<CompletionSuggestion.Entry.Option> options = suggestion.getOptions();
            //4.3 遍历
            for (CompletionSuggestion.Entry.Option option : options) {
        
                String text = option.getText().toString();
                System.out.println(text);
    
            }
        }

运行得到以h开头的所有结果：

![在这里插入图片描述][396129783d3c4fe4a4ece0236eb7848e.png]

## 7、需求：搜索框实现自动补全 ##

看下前端页面，每当在输入框键入时，前端会发送ajax请求：

![在这里插入图片描述][fe25633f735a4e399e6c54252d95a1a5.png]

接下来完善这个接口，实现搜索框的自动补全：

*  controller接口定义

import cn.itcast.hotel.domain.dto.RequestParams;
    import cn.itcast.hotel.domain.vo.PageResult;
    import cn.itcast.hotel.service.IHotelService;
    import org.springframework.web.bind.annotation.*;
    
    import javax.annotation.Resource;
    import java.util.List;
    import java.util.Map;
    @RestController
    @RequestMapping("/hotel")
    public class HotelSearchController {
        
    
        @Resource
        IHotelService hotelService;
    
        @GetMapping("/suggestion")
        public List<String> getSuggestions(@RequestParam("key") String prefix){
        
            return hotelService.getSuggestion(prefix);
        }
    
    
    
    }

*  Service接口

public interface IHotelService extends IService<Hotel> {
        
     
        List<String> getSuggestion(String prefix);
    }

*  接口实现

@Override
    public List<String> getSuggestion(String prefix) {
        
        try {
        
            SearchRequest request = new SearchRequest("hotel");
            request.source().suggest(new SuggestBuilder().addSuggestion(
                    "mySuggestion",
                    SuggestBuilders.completionSuggestion("suggestion")
                            .prefix(prefix)
                            .skipDuplicates(true)
                            .size(15)
            ));
            SearchResponse response = client.search(request, RequestOptions.DEFAULT);
            Suggest suggest = response.getSuggest();
            CompletionSuggestion mySuggestion = suggest.getSuggestion("mySuggestion");
            List<CompletionSuggestion.Entry.Option> options = mySuggestion.getOptions();
            return options.stream()
                    .map(t -> t.getText().toString())
                    .collect(Collectors.toList());
        } catch (IOException e) {
        
            throw new RuntimeException();
        }
    }

*  关于client这个Bean，再补充下：

@SpringBootApplication
    public class HotelDemoApplication {
        
    
        public static void main(String[] args) {
        
            SpringApplication.run(HotelDemoApplication.class, args);
        }
    
        @Bean
        public RestHighLevelClient client(){
        
            return new RestHighLevelClient(RestClient.builder(
                    HttpHost.create("http://10.4.130.220:9200")
            ));
        }
    
    }

重启服务，看下效果：

![在这里插入图片描述][35b372eb4be44cf6bbf2c22911bc2477.png]

自动补全成功实现！！

[cca22b3280ee44cd9ae5ee183857a405.png]: https://img-blog.csdnimg.cn/cca22b3280ee44cd9ae5ee183857a405.png
[https_github.com_medcl_elasticsearch-analysis-pinyin]: https://github.com/medcl/elasticsearch-analysis-pinyin
[13a18e06243d43f1985ff77bb8d756b7.png]: https://img-blog.csdnimg.cn/13a18e06243d43f1985ff77bb8d756b7.png
[08cf0cf9f75141c9ad938a3959ade24c.png]: https://img-blog.csdnimg.cn/08cf0cf9f75141c9ad938a3959ade24c.png
[c28324248d224345b628b5a6529ef0b9.png]: https://img-blog.csdnimg.cn/c28324248d224345b628b5a6529ef0b9.png
[3ae98bc9ebf14682886a26f29703ffd4.png]: https://img-blog.csdnimg.cn/3ae98bc9ebf14682886a26f29703ffd4.png
[a970499fc44b4d148ea4faf77bef26f3.png]: https://img-blog.csdnimg.cn/a970499fc44b4d148ea4faf77bef26f3.png
[46e9618ae78744baaefb4b77bc862dcd.png]: https://img-blog.csdnimg.cn/46e9618ae78744baaefb4b77bc862dcd.png
[1fa0c2cad5154ad183b5be5a9570f162.png]: https://img-blog.csdnimg.cn/1fa0c2cad5154ad183b5be5a9570f162.png
[622aa5e262784214b22bddeeedf72f68.png]: https://img-blog.csdnimg.cn/622aa5e262784214b22bddeeedf72f68.png
[84fc5fb2e1844e31b58f67f085fa3a61.png]: https://img-blog.csdnimg.cn/84fc5fb2e1844e31b58f67f085fa3a61.png
[f729335fc704400dbb582540c5902141.png]: https://img-blog.csdnimg.cn/f729335fc704400dbb582540c5902141.png
[161ab6956407424c95d44cbe3b725c61.png]: https://img-blog.csdnimg.cn/161ab6956407424c95d44cbe3b725c61.png
[1459fb82c536445289e2964e846e29ad.png]: https://img-blog.csdnimg.cn/1459fb82c536445289e2964e846e29ad.png
[e3e107af6bbe4b1b996ba1e76ae4e0ff.png]: https://img-blog.csdnimg.cn/e3e107af6bbe4b1b996ba1e76ae4e0ff.png
[Completion Suggester]: https://www.elastic.co/guide/en/elasticsearch/reference/7.6/search-suggesters.html#completion-suggester
[13eff94d616f49dc868ab16304e5af04.png]: https://img-blog.csdnimg.cn/13eff94d616f49dc868ab16304e5af04.png
[261139b763794370a26cb6be6364b1ba.png]: https://img-blog.csdnimg.cn/261139b763794370a26cb6be6364b1ba.png
[4c3e3e617b7649e3b1dc644235242a9f.png]: https://img-blog.csdnimg.cn/4c3e3e617b7649e3b1dc644235242a9f.png
[b7a2399c0072477db013a61cb95ae584.png]: https://img-blog.csdnimg.cn/b7a2399c0072477db013a61cb95ae584.png
[2ef9c3dac0cc427ebb95677b33eb5e2e.png]: https://img-blog.csdnimg.cn/2ef9c3dac0cc427ebb95677b33eb5e2e.png
[258ca906fee3421b842e1a7670d668e9.png]: https://img-blog.csdnimg.cn/258ca906fee3421b842e1a7670d668e9.png
[ed186f897f0842649f689e61a82bd85d.png]: https://img-blog.csdnimg.cn/ed186f897f0842649f689e61a82bd85d.png
[29da20d18c674a478f3660ec7f4f7a5c.png]: https://img-blog.csdnimg.cn/29da20d18c674a478f3660ec7f4f7a5c.png
[396129783d3c4fe4a4ece0236eb7848e.png]: https://img-blog.csdnimg.cn/396129783d3c4fe4a4ece0236eb7848e.png
[fe25633f735a4e399e6c54252d95a1a5.png]: https://img-blog.csdnimg.cn/fe25633f735a4e399e6c54252d95a1a5.png
[35b372eb4be44cf6bbf2c22911bc2477.png]: https://img-blog.csdnimg.cn/35b372eb4be44cf6bbf2c22911bc2477.png