Elasticserach 自定义分词器

刺骨的言语ヽ痛彻心扉 2023-07-07 11:16 30阅读 0赞

### 1）环境准备 ###

启动Elasticsearch https://blog.csdn.net/qq\_36918149/article/details/104221934  
启动Kinbana https://blog.csdn.net/qq\_36918149/article/details/104224625

### 2）Character Filter ###

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2OTE4MTQ5_size_16_color_FFFFFF_t_70]  
演示一：  
\#去除html标签

POST _analyze
    { 
      "tokenizer":"keyword",
      "char_filter":["html_strip"],
      "text": "<b>hello world</b>"
    }

结果，html标签已经被去除  
![在这里插入图片描述][2020022221385935.png]  
演示二：  
\#使用char filter进行替换

POST _analyze
    { 
      "tokenizer": "standard",
      "char_filter": [
          { 
            "type" : "mapping",
            "mappings" : [ "- => _"]
          }
        ],
      "text": "123-456, I-test! test-990 650-555-1234"
    }

结果显示，“-” 已替换为“\_”，并进行了分词  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2OTE4MTQ5_size_16_color_FFFFFF_t_70 1]

### 3）Tokenizer ###

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2OTE4MTQ5_size_16_color_FFFFFF_t_70 2]  
演示：  
\#按路径分词

POST _analyze
    { 
      "tokenizer":"path_hierarchy",
      "text":"/user/ymruan/a/b/c/d/e"
    }

结果：路径被依次进行了分词  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2OTE4MTQ5_size_16_color_FFFFFF_t_70 3]

4）Token Filter  
![在这里插入图片描述][20200222214921671.png]  
演示一：

# whitespace与stop
    GET _analyze
    { 
      "tokenizer": "whitespace",
      "filter": ["stop","snowball"],
      "text": ["The rain in Spain falls mainly on the plain."]
    }

结果:连接词被去掉以后的分词结果  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2OTE4MTQ5_size_16_color_FFFFFF_t_70 4]

### 4）总结 ###

*  过滤器过滤顺序：char filter - >tokenizer -> token filter。
 *  本章节在实际应用中还需要，仔细看一下api文档。

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2OTE4MTQ5_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20200222214741389.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2OTE4MTQ5,size_16,color_FFFFFF,t_70
[2020022221385935.png]: https://img-blog.csdnimg.cn/2020022221385935.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2OTE4MTQ5_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20200222213531225.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2OTE4MTQ5,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2OTE4MTQ5_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/20200222214836312.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2OTE4MTQ5,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2OTE4MTQ5_size_16_color_FFFFFF_t_70 3]: https://img-blog.csdnimg.cn/20200222214153402.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2OTE4MTQ5,size_16,color_FFFFFF,t_70
[20200222214921671.png]: https://img-blog.csdnimg.cn/20200222214921671.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2OTE4MTQ5_size_16_color_FFFFFF_t_70 4]: https://img-blog.csdnimg.cn/20200222215611961.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2OTE4MTQ5,size_16,color_FFFFFF,t_70