Elasticsearch+Spring Boot集成实践

Bertha 。 2023-07-11 05:37 32阅读 0赞

# ELK-技术栈 #

## Elasticsearch ##

### 简介 ###

Elasticsearch 是一个分布式、RESTful 风格的搜索和数据分析引擎，能够解决不断涌现出的各种用例。 作为 Elastic Stack 的核心，它集中存储您的数据，帮助您发现意料之中以及意料之外的情况。

Elasticsearch是当前比较流行的开源的分布式搜索和数据分析引擎，具备**易使用**、**高性能**、**扩展性强**等特点。是ElasticStack的核心组件，以其为核心构建的ELK，已经是日志分析领域的事实标准。

与ES有关的关键字如下所示：

*  实时
 *  分布式
 *  搜索
 *  分析

### 发展历史 ###

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw_size_16_color_FFFFFF_t_70][]

*  作者Shay Banon
 *  时间2010年2月8日
 *  妻子 - 伦敦 - 搜索菜谱 - lucene
 *  历程
    
     *  ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw_size_16_color_FFFFFF_t_70 1][]

### 作用 ###

*  **全文搜索**
 *  **站内分析**

**Top5**典型应用场景

*  记录和日志分析
 *  采集和组合公共数据
 *  全文搜索
 *  事件数据和指标
 *  数据可视化

--------------------

ElasticSearch 在滴滴有着非常丰富的应用场景：

1.  为线上**核心搜索业务**提供引擎支持；
2.  作为 RDS 从库，海量数据检索需求；
3.  解决公司**海量日志**检索问题；
4.  为**安全场景**提供数据分析能力。

--------------------

### 百度指数 ###

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw_size_16_color_FFFFFF_t_70 2][]

## Kibana ##

Kibana 是为 Elasticsearch设计的开源分析和可视化平台。你可以使用 Kibana 来搜索，查看存储在 Elasticsearch 索引中的数据并与之交互。你可以很容易实现高级的数据分析和可视化，以图标的形式展现出来。

主要用途

*  实时监控
 *  问题分析

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw_size_16_color_FFFFFF_t_70 3][]

## Logstash ##

Logstash 是开源的服务器端数据处理管道，能够同时从多个来源采集数据，转换数据，然后将数据发送到您最喜欢的“存储库”中。 **集中、转换和存储数据**

Logstash是一个具有实时pipeline功能的开源数据收集引擎。Logstash可以动态的统一来自不同数据源的数据，并将数据规范化到你选择的目的地。虽然Logstash最初推动了日志收集方面的创新，但它的功能现在更丰富了。任何类型的事件都可以通过丰富的input,filter,output插件进行转换，简化抽取过程

**LogStash的数据源**

*  日志和指标
 *  web
 *  数据存储和流

# 环境搭建 #

[下载中心][Link 1]

ELK技术栈全方位提供了各种安装方式，

**rpm、deb、tar.gz、docker**，

可以**随心使用**。

## Elasticsearch ##

### 安装 ###

[安装并运行Elasticsearch][Elasticsearch]

cd elasticsearch-<version>
    ./bin/elasticsearch

如果你想把 Elasticsearch 作为一个守护进程在后台运行，那么可以在后面添加参数 `-d` 。 如果你是在 Windows 上面运行 Elasticseach，你应该运行 `bin\elasticsearch.bat` 而不是 `bin\elasticsearch` 。

*  如果你想把 Elasticsearch 作为一个守护进程在后台运行，那么可以在后面添加参数 `-d` 。
 *  如果你是在 Windows 上面运行 Elasticseach，你应该运行 `bin\elasticsearch.bat` 而不是 `bin\elasticsearch` 。-

### 配置项 ###

# ======================== Elasticsearch Configuration =========================
    #
    # NOTE: Elasticsearch comes with reasonable defaults for most settings.
    # Before you set out to tweak and tune the configuration, make sure you
    # understand what are you trying to accomplish and the consequences.
    #
    # The primary way of configuring a node is via this file. This template lists
    # the most important settings you may want to configure for a production cluster.
    #
    # Please consult the documentation for further information on configuration options:
    # https://www.elastic.co/guide/en/elasticsearch/reference/index.html
    #
    # ---------------------------------- Cluster -----------------------------------
    #
    # Use a descriptive name for your cluster:
    #
    cluster.name: my-application
    #
    # ------------------------------------ Node ------------------------------------
    #
    # Use a descriptive name for the node:
    #
    #node.name: node-1
    #
    # Add custom attributes to the node:
    #
    #node.attr.rack: r1
    #
    # ----------------------------------- Paths ------------------------------------
    #
    # Path to directory where to store the data (separate multiple locations by comma):
    #
    #path.data: /path/to/data
    #
    # Path to log files:
    #
    #path.logs: /path/to/logs
    #
    # ----------------------------------- Memory -----------------------------------
    #
    # Lock the memory on startup:
    #
    #bootstrap.memory_lock: true
    #
    # Make sure that the heap size is set to about half the memory available
    # on the system and that the owner of the process is allowed to use this
    # limit.
    #
    # Elasticsearch performs poorly when the system is swapping the memory.
    #
    # ---------------------------------- Network -----------------------------------
    #
    # Set the bind address to a specific IP (IPv4 or IPv6):
    #
    network.host: localhost
    #
    # Set a custom port for HTTP:
    #
    http.port: 9200
    #
    # For more information, consult the network module documentation.
    #
    # --------------------------------- Discovery ----------------------------------
    #
    # Pass an initial list of hosts to perform discovery when new node is started:
    # The default list of hosts is ["127.0.0.1", "[::1]"]
    #
    #discovery.zen.ping.unicast.hosts: ["host1", "host2"]
    #
    # Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
    #
    #discovery.zen.minimum_master_nodes: 3
    #
    # For more information, consult the zen discovery module documentation.
    #
    # ---------------------------------- Gateway -----------------------------------
    #
    # Block initial recovery after a full cluster restart until N nodes are started:
    #
    #gateway.recover_after_nodes: 3
    #
    # For more information, consult the gateway module documentation.
    #
    # ---------------------------------- Various -----------------------------------
    #
    # Require explicit names when deleting indices:
    #
    #action.destructive_requires_name: true

测试ES安装是否成功，可以使用

curl 'http://localhost:9200/?pretty'

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw_size_16_color_FFFFFF_t_70 4][]

ze则表示安装成功

## Kibana ##

### 安装 ###

[在Windows上安装Kibana][Windows_Kibana]

cd $KIBANA_HOME
    .\bin\kibana

可以访问浏览器: localhost:5601测试安装是否成功。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw_size_16_color_FFFFFF_t_70 5][]

### 配置 ###

Kibana 默认情况下从 `$KIBANA_HOME/config/kibana.yml` 加载配置文件。

<table> 
 <thead> 
  <tr> 
   <th>类型</th> 
   <th>描述</th> 
   <th>默认位置</th> 
   <th>设置</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td><strong>home</strong></td> 
   <td>Kibana home 目录或 <code>$KIBANA_HOME</code> 。</td> 
   <td>解压包时创建的目录</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td><strong>bin</strong></td> 
   <td>二进制脚本，包括 <code>kibana</code> 启动 Kibana 服务和 <code>kibana-plugin</code> 安装插件。</td> 
   <td><code>$KIBANA_HOME\bin</code></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td><strong>config</strong></td> 
   <td>配置文件包括 <code>kibana.yml</code> 。</td> 
   <td><code>$KIBANA_HOME\config</code></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td><strong>data</strong></td> 
   <td>Kibana 和其插件写入磁盘的数据文件位置。</td> 
   <td><code>$KIBANA_HOME\data</code></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td><strong>optimize</strong></td> 
   <td>编译过的源码。某些管理操作(如，插件安装)导致运行时重新编译源码。</td> 
   <td><code>$KIBANA_HOME\optimize</code></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td><strong>plugins</strong></td> 
   <td>插件文件位置。每一个插件都一个单独的二级目录。</td> 
   <td><code>$KIBANA_HOME\plugins</code></td> 
   <td></td> 
  </tr> 
 </tbody> 
</table>

# Kibana is served by a back end server. This setting specifies the port to use.
    #server.port: 5601
    
    # Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
    # The default is 'localhost', which usually means remote machines will not be able to connect.
    # To allow connections from remote users, set this parameter to a non-loopback address.
    server.host: "0.0.0.0"
    
    # Enables you to specify a path to mount Kibana at if you are running behind a proxy. This only affects
    # the URLs generated by Kibana, your proxy is expected to remove the basePath value before forwarding requests
    # to Kibana. This setting cannot end in a slash.
    #server.basePath: ""
    
    # The maximum payload size in bytes for incoming server requests.
    #server.maxPayloadBytes: 1048576
    
    # The Kibana server's name. This is used for display purposes.
    #server.name: "your-hostname"
    
    # The URL of the Elasticsearch instance to use for all your queries.
    elasticsearch.url: "http://localhost:9200"
    
    # When this setting's value is true Kibana uses the hostname specified in the server.host
    # setting. When the value of this setting is false, Kibana uses the hostname of the host
    # that connects to this Kibana instance.
    #elasticsearch.preserveHost: true
    
    # Kibana uses an index in Elasticsearch to store saved searches, visualizations and
    # dashboards. Kibana creates a new index if the index doesn't already exist.
    #kibana.index: ".kibana"
    
    # The default application to load.
    #kibana.defaultAppId: "discover"
    
    # If your Elasticsearch is protected with basic authentication, these settings provide
    # the username and password that the Kibana server uses to perform maintenance on the Kibana
    # index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
    # is proxied through the Kibana server.
    #elasticsearch.username: "user"
    #elasticsearch.password: "pass"
    
    # Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
    # These settings enable SSL for outgoing requests from the Kibana server to the browser.
    #server.ssl.enabled: false
    #server.ssl.certificate: /path/to/your/server.crt
    #server.ssl.key: /path/to/your/server.key
    
    # Optional settings that provide the paths to the PEM-format SSL certificate and key files.
    # These files validate that your Elasticsearch backend uses the same key files.
    #elasticsearch.ssl.certificate: /path/to/your/client.crt
    #elasticsearch.ssl.key: /path/to/your/client.key
    
    # Optional setting that enables you to specify a path to the PEM file for the certificate
    # authority for your Elasticsearch instance.
    #elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]
    
    # To disregard the validity of SSL certificates, change this setting's value to 'none'.
    #elasticsearch.ssl.verificationMode: full
    
    # Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of
    # the elasticsearch.requestTimeout setting.
    #elasticsearch.pingTimeout: 1500
    
    # Time in milliseconds to wait for responses from the back end or Elasticsearch. This value
    # must be a positive integer.
    #elasticsearch.requestTimeout: 30000
    
    # List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side
    # headers, set this value to [] (an empty list).
    #elasticsearch.requestHeadersWhitelist: [ authorization ]
    
    # Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten
    # by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration.
    #elasticsearch.customHeaders: {}
    
    # Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.
    #elasticsearch.shardTimeout: 0
    
    # Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying.
    #elasticsearch.startupTimeout: 5000
    
    # Specifies the path where Kibana creates the process ID file.
    #pid.file: /var/run/kibana.pid
    
    # Enables you specify a file where Kibana stores log output.
    #logging.dest: stdout
    
    # Set the value of this setting to true to suppress all logging output.
    #logging.silent: false
    
    # Set the value of this setting to true to suppress all logging output other than error messages.
    #logging.quiet: false
    
    # Set the value of this setting to true to log all events, including system usage information
    # and all requests.
    #logging.verbose: false
    
    # Set the interval in milliseconds to sample system and process performance
    # metrics. Minimum is 100ms. Defaults to 5000.
    #ops.interval: 5000
    
    # The default locale. This locale can be used in certain circumstances to substitute any missing
    # translations.
    #i18n.defaultLocale: "en"

通过选项**elasticsearch.url**来使得kibana连接elasticsearch。

## 脚本 ##

可以采用批处理方式来一键启动ES和Kibana

@echo off
    echo Starting Elasticsearch...
    D:
    cd D:\elasticsearch-6.0.0\bin
    start elasticsearch.bat
    echo Starting Kibana...
    cd D:\kibana-6.0.0-windows-x86_64\bin
    start kibana.bat
    echo "This is the first dos program"
    exit

# Elasticsearch介绍 #

## 检索工作原理 ##

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw_size_16_color_FFFFFF_t_70 6][]

## 基本组成 ##

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw_size_16_color_FFFFFF_t_70 7][]

## DSL（Domain Specified Language） ##

**DSL**：以极其高效的方式描述特定领域的对象、规则和运行方式的语言。

*  需要有特定的解释器与其配合。
 *  高效简洁的领域语言，与通用语言相比能极大降级理解和使用难度，同时极大提高开发效率的语言。
 *  能够描述特定领域的世界观和方法论的语言。

{ 
        "bool": { 
            "must":     {  "term": {  "folder": "inbox" }},
            "must_not": {  "term": {  "tag":    "spam"  }},
            "should": [
                        {  "term": {  "starred": true   }},
                        {  "term": {  "unread":  true   }}
            ]
        }
    }

更多的实践

## Elasticsearch和MySql ##

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw_size_16_color_FFFFFF_t_70 8][]

**区别**：

*  Elasticsearch是**面向文档**的
 *  MySQL关系型数据库，具有事务性，而ES没有事务性
 *  ES没有外键约束，MySQL支持表与表之间的外键约束
 *  ES采用**倒排索引**,关系型数据库Mysql采用的是B+树索引
 *  面对**大数据量**简单计算的时候ES的效率**远**高于Mysql传统数据库

`Type` 可以理解成关系数据库中Table。

**之前**的版本中，索引和文档中间还有个类型的概念，**每个索引下可以建立多个类型**，文档存储时需要指定index和type。**从6.0.0开始单个索引中只能有一个类型**，

**7.0.0以后将将不建议使用，8.0.0 以后完全不支持**。

{ 
      "error": { 
        "root_cause": [
          { 
            "type": "illegal_argument_exception",
            "reason": "Rejecting mapping update to [bigdata] as the final mapping would have more than 1 type: [product, item]"
          }
        ],
        "type": "illegal_argument_exception",
        "reason": "Rejecting mapping update to [bigdata] as the final mapping would have more than 1 type: [product, item]"
      },
      "status": 400
    }

Elasticsearch的强大之处就是可以\*\*\*模糊查询\*\*\*

Elasticsearch是专门做搜索的，就是为了解决上面所讲的问题而生的，换句话说：

*  Elasticsearch对模糊搜索非常擅长（搜索速度很快）
 *  从Elasticsearch搜索到的数据可以根据评分过滤掉大部分的，只要返回评分高的给用户就好了（原生就支持排序）
 *  没有那么准确的关键字也能搜出相关的结果（能匹配有相关性的记录）

# 整合 #

## SpringBoot整合 ##

### ElasticsearchTemplate类型 ###

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw_size_16_color_FFFFFF_t_70 9][]

ElasticsearchTemplate许可的主要操作位于接口ElasticsearchOperations中，该类型源码如下：

/* * Copyright 2013-2016 the original author or authors. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */
    package org.springframework.data.elasticsearch.core;
    
    import org.elasticsearch.action.update.UpdateResponse;
    import org.elasticsearch.client.Client;
    import org.elasticsearch.cluster.metadata.AliasMetaData;
    import org.elasticsearch.common.Nullable;
    import org.springframework.data.domain.Page;
    import org.springframework.data.elasticsearch.core.convert.ElasticsearchConverter;
    import org.springframework.data.elasticsearch.core.mapping.ElasticsearchPersistentEntity;
    import org.springframework.data.elasticsearch.core.query.*;
    import org.springframework.data.util.CloseableIterator;
    
    import java.util.LinkedList;
    import java.util.List;
    import java.util.Map;
    
    /** * ElasticsearchOperations * * @author Rizwan Idrees * @author Mohsin Husen * @author Kevin Leturc */
    public interface ElasticsearchOperations { 
    
    	/** * @return Converter in use */
    	ElasticsearchConverter getElasticsearchConverter();
    
    	/** * @return elasticsearch client */
    	Client getClient();
    
    	/** * Create an index for a class * * @param clazz * @param <T> */
    	<T> boolean createIndex(Class<T> clazz);
    
    	/** * Create an index for given indexName * * @param indexName */
    	boolean createIndex(String indexName);
    
    	/** * Create an index for given indexName and Settings * * @param indexName * @param settings */
    	boolean createIndex(String indexName, Object settings);
    
    	/** * Create an index for given class and Settings * * @param clazz * @param settings */
    	<T> boolean createIndex(Class<T> clazz, Object settings);
    
    	/** * Create mapping for a class * * @param clazz * @param <T> */
    	<T> boolean putMapping(Class<T> clazz);
    
    	/** * Create mapping for a given indexName and type * * @param indexName * @param type * @param mappings */
    	boolean putMapping(String indexName, String type, Object mappings);
    
    	/** * Create mapping for a class * * @param clazz * @param mappings */
    	<T> boolean putMapping(Class<T> clazz, Object mappings);
    
    
    	/** * Get mapping for a class * * @param clazz * @param <T> */
    	<T> Map getMapping(Class<T> clazz);
    
    	/** * Get mapping for a given indexName and type * * @param indexName * @param type */
    	Map getMapping(String indexName, String type);
    
    	/** * Get settings for a given indexName * * @param indexName */
    	Map getSetting(String indexName);
    
    	/** * Get settings for a given class * * @param clazz */
    	<T> Map getSetting(Class<T> clazz);
    
    
    	/** * Execute the query against elasticsearch and return the first returned object * * @param query * @param clazz * @return the first matching object */
    	<T> T queryForObject(GetQuery query, Class<T> clazz);
    
    	/** * Execute the query against elasticsearch and return the first returned object using custom mapper * * @param query * @param clazz * @param mapper * @return the first matching object */
    	<T> T queryForObject(GetQuery query, Class<T> clazz, GetResultMapper mapper);
    
    	/** * Execute the query against elasticsearch and return the first returned object * * @param query * @param clazz * @return the first matching object */
    	<T> T queryForObject(CriteriaQuery query, Class<T> clazz);
    
    	/** * Execute the query against elasticsearch and return the first returned object * * @param query * @param clazz * @return the first matching object */
    	<T> T queryForObject(StringQuery query, Class<T> clazz);
    
    	/** * Execute the query against elasticsearch and return result as {@link Page} * * @param query * @param clazz * @return */
    	<T> Page<T> queryForPage(SearchQuery query, Class<T> clazz);
    
    	/** * Execute the query against elasticsearch and return result as {@link Page} using custom mapper * * @param query * @param clazz * @return */
    	<T> Page<T> queryForPage(SearchQuery query, Class<T> clazz, SearchResultMapper mapper);
    
    	/** * Execute the query against elasticsearch and return result as {@link Page} * * @param query * @param clazz * @return */
    	<T> Page<T> queryForPage(CriteriaQuery query, Class<T> clazz);
    
    	/** * Execute the query against elasticsearch and return result as {@link Page} * * @param query * @param clazz * @return */
    	<T> Page<T> queryForPage(StringQuery query, Class<T> clazz);
    
    	/** * Execute the query against elasticsearch and return result as {@link Page} using custom mapper * * @param query * @param clazz * @return */
    	<T> Page<T> queryForPage(StringQuery query, Class<T> clazz, SearchResultMapper mapper);
    
    	/** * Executes the given {@link CriteriaQuery} against elasticsearch and return result as {@link CloseableIterator}. * <p> * Returns a {@link CloseableIterator} that wraps an Elasticsearch scroll context that needs to be closed in case of error. * * @param <T> element return type * @param query * @param clazz * @return * @since 1.3 */
    	<T> CloseableIterator<T> stream(CriteriaQuery query, Class<T> clazz);
    
    	/** * Executes the given {@link SearchQuery} against elasticsearch and return result as {@link CloseableIterator}. * <p> * Returns a {@link CloseableIterator} that wraps an Elasticsearch scroll context that needs to be closed in case of error. * * @param <T> element return type * @param query * @param clazz * @return * @since 1.3 */
    	<T> CloseableIterator<T> stream(SearchQuery query, Class<T> clazz);
    
    	/** * Executes the given {@link SearchQuery} against elasticsearch and return result as {@link CloseableIterator} using custom mapper. * <p> * Returns a {@link CloseableIterator} that wraps an Elasticsearch scroll context that needs to be closed in case of error. * * @param <T> element return type * @param query * @param clazz * @param mapper * @return * @since 1.3 */
    	<T> CloseableIterator<T> stream(SearchQuery query, Class<T> clazz, SearchResultMapper mapper);
    
    	/** * Execute the criteria query against elasticsearch and return result as {@link List} * * @param query * @param clazz * @param <T> * @return */
    	<T> List<T> queryForList(CriteriaQuery query, Class<T> clazz);
    
    	/** * Execute the string query against elasticsearch and return result as {@link List} * * @param query * @param clazz * @param <T> * @return */
    	<T> List<T> queryForList(StringQuery query, Class<T> clazz);
    
    	/** * Execute the search query against elasticsearch and return result as {@link List} * * @param query * @param clazz * @param <T> * @return */
    	<T> List<T> queryForList(SearchQuery query, Class<T> clazz);
    
    	/** * Execute the query against elasticsearch and return ids * * @param query * @return */
    	<T> List<String> queryForIds(SearchQuery query);
    
    	/** * return number of elements found by given query * * @param query * @param clazz * @return */
    	<T> long count(CriteriaQuery query, Class<T> clazz);
    
    	/** * return number of elements found by given query * * @param query * @return */
    	<T> long count(CriteriaQuery query);
    
    	/** * return number of elements found by given query * * @param query * @param clazz * @return */
    	<T> long count(SearchQuery query, Class<T> clazz);
    
    	/** * return number of elements found by given query * * @param query * @return */
    	<T> long count(SearchQuery query);
    
    	/** * Execute a multiGet against elasticsearch for the given ids * * @param searchQuery * @param clazz * @return */
    	<T> LinkedList<T> multiGet(SearchQuery searchQuery, Class<T> clazz);
    
    	/** * Execute a multiGet against elasticsearch for the given ids with MultiGetResultMapper * * @param searchQuery * @param clazz * @param multiGetResultMapper * @return */
    	<T> LinkedList<T> multiGet(SearchQuery searchQuery, Class<T> clazz, MultiGetResultMapper multiGetResultMapper);
    
    	/** * Index an object. Will do save or update * * @param query * @return returns the document id */
    	String index(IndexQuery query);
    
    	/** * Partial update of the document * * @param updateQuery * @return */
    	UpdateResponse update(UpdateQuery updateQuery);
    
    	/** * Bulk index all objects. Will do save or update * * @param queries */
    	void bulkIndex(List<IndexQuery> queries);
    
    	/** * Bulk update all objects. Will do update * * @param queries */
    	void bulkUpdate(List<UpdateQuery> queries);
    
    	/** * Delete the one object with provided id * * @param indexName * @param type * @param id * @return documentId of the document deleted */
    	String delete(String indexName, String type, String id);
    
    
    	/** * Delete all records matching the criteria * * @param clazz * @param criteriaQuery */
    	<T> void delete(CriteriaQuery criteriaQuery, Class<T> clazz);
    	/** * Delete the one object with provided id * * @param clazz * @param id * @return documentId of the document deleted */
    	<T> String delete(Class<T> clazz, String id);
    
    	/** * Delete all records matching the query * * @param clazz * @param query */
    	<T> void delete(DeleteQuery query, Class<T> clazz);
    
    	/** * Delete all records matching the query * * @param query */
    	void delete(DeleteQuery query);
    
    	/** * Deletes an index for given entity * * @param clazz * @param <T> * @return */
    	<T> boolean deleteIndex(Class<T> clazz);
    
    	/** * Deletes an index for given indexName * * @param indexName * @return */
    	boolean deleteIndex(String indexName);
    
    	/** * check if index is exists * * @param clazz * @param <T> * @return */
    	<T> boolean indexExists(Class<T> clazz);
    
    	/** * check if index is exists for given IndexName * * @param indexName * @return */
    	boolean indexExists(String indexName);
    
    	/** * check if type is exists in an index * * @param index * @param type * @return */
    	boolean typeExists(String index, String type);
    
    	/** * refresh the index * * @param indexName * */
    	void refresh(String indexName);
    
    	/** * refresh the index * * @param clazz * */
    	<T> void refresh(Class<T> clazz);
    
    	/** * Returns scrolled page for given query * * @param query The search query. * @param scrollTimeInMillis The time in millisecond for scroll feature * {@link org.elasticsearch.action.search.SearchRequestBuilder#setScroll(org.elasticsearch.common.unit.TimeValue)}. * @param clazz The class of entity to retrieve. * @return The scan id for input query. */
    	<T> Page<T> startScroll(long scrollTimeInMillis, SearchQuery query, Class<T> clazz);
    
    	/** * Returns scrolled page for given query * * @param query The search query. * @param scrollTimeInMillis The time in millisecond for scroll feature * {@link org.elasticsearch.action.search.SearchRequestBuilder#setScroll(org.elasticsearch.common.unit.TimeValue)}. * @param mapper Custom impl to map result to entities * @return The scan id for input query. */
    	<T> Page<T> startScroll(long scrollTimeInMillis, SearchQuery query, Class<T> clazz, SearchResultMapper mapper);
    
    	/** * Returns scrolled page for given query * * @param criteriaQuery The search query. * @param scrollTimeInMillis The time in millisecond for scroll feature * {@link org.elasticsearch.action.search.SearchRequestBuilder#setScroll(org.elasticsearch.common.unit.TimeValue)}. * @param clazz The class of entity to retrieve. * @return The scan id for input query. */
    	<T> Page<T> startScroll(long scrollTimeInMillis, CriteriaQuery criteriaQuery, Class<T> clazz);
    
    	/** * Returns scrolled page for given query * * @param criteriaQuery The search query. * @param scrollTimeInMillis The time in millisecond for scroll feature * {@link org.elasticsearch.action.search.SearchRequestBuilder#setScroll(org.elasticsearch.common.unit.TimeValue)}. * @param mapper Custom impl to map result to entities * @return The scan id for input query. */
    	<T> Page<T> startScroll(long scrollTimeInMillis, CriteriaQuery criteriaQuery, Class<T> clazz, SearchResultMapper mapper);
    
    
    	<T> Page<T> continueScroll(@Nullable String scrollId, long scrollTimeInMillis, Class<T> clazz);
    	<T> Page<T> continueScroll(@Nullable String scrollId, long scrollTimeInMillis, Class<T> clazz, SearchResultMapper mapper);
    	/** * Clears the search contexts associated with specified scroll ids. * * @param scrollId * */
    	<T> void clearScroll(String scrollId);
    
    	/** * more like this query to search for documents that are "like" a specific document. * * @param query * @param clazz * @param <T> * @return */
    	<T> Page<T> moreLikeThis(MoreLikeThisQuery query, Class<T> clazz);
    
    	/** * adding new alias * * @param query * @return */
    	Boolean addAlias(AliasQuery query);
    
    	/** * removing previously created alias * * @param query * @return */
    	Boolean removeAlias(AliasQuery query);
    
    	/** * get all the alias pointing to specified index * * @param indexName * @return */
    	List<AliasMetaData> queryForAlias(String indexName);
    
    
    	<T> T query(SearchQuery query, ResultsExtractor<T> resultsExtractor);
    
    
    	ElasticsearchPersistentEntity getPersistentEntityFor(Class clazz);
    }

### POM ###

在POM文件中，需要引入**spring-boot-starter-data-elasticsearch**依赖。

<dependencies>
          <dependency>
              <groupId>org.springframework.boot</groupId>
              <artifactId>spring-boot-starter-web</artifactId>
          </dependency>
          <dependency>
              <groupId>org.springframework.boot</groupId>
              <artifactId>spring-boot-starter-test</artifactId>
              <scope>test</scope>
          </dependency>
          <dependency>
              <groupId>junit</groupId>
              <artifactId>junit</artifactId>
              <version>4.12</version>
              <scope>test</scope>
          </dependency>
          
          <dependency>
              <groupId>org.springframework.boot</groupId>
              <artifactId>spring-boot-starter-data-elasticsearch</artifactId>
          </dependency>
    
          <dependency>
              <groupId>org.projectlombok</groupId>
              <artifactId>lombok</artifactId>
              <version>1.18.0</version>
              <scope>provided</scope>
          </dependency>
    
      </dependencies>
    
      <repositories>
        <repository>
          <id>releases</id>
          <name>Releases</name>
          <url>http://localhost:8081//repository/maven-central/</url>
          <releases>
            <enabled>true</enabled>
          </releases>
        </repository>
      </repositories>
      <pluginRepositories>
        <pluginRepository>
          <id>releases</id>
          <name>Releases</name>
          <url>http://localhost:8081//repository/maven-central/</url>
          <releases>
            <enabled>true</enabled>
          </releases>
        </pluginRepository>
      </pluginRepositories>

### 配置项application.yml ###

spring:
      data:
        elasticsearch:
          cluster-name: my-application
          cluster-nodes: 127.0.0.1:9300

> 9300为Java客户端使用的端口，9200为elasticsearch使用的端口

在Spring Boot中的自动装配工作时，这两个配置项映射的类型为

/** * Configuration properties for Elasticsearch. * * @author Artur Konczak * @author Mohsin Husen * @since 1.1.0 */
    @ConfigurationProperties(prefix = "spring.data.elasticsearch")
    public class ElasticsearchProperties { 
    
    	/** * Elasticsearch cluster name. */
    	private String clusterName = "elasticsearch";
    
    	/** * Comma-separated list of cluster node addresses. */
    	private String clusterNodes;

在此简要论述Spring Boot的自动装配原理：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw_size_16_color_FFFFFF_t_70 10][]

![自动配置][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw_size_16_color_FFFFFF_t_70 11]

而ElasticsearchTemplate自动装配到ioc容器中使用了如下的类型

*  ElasticSearchProperties
 *  **ElasticsearchAutoConfiguration**
 *  **ElasticsearchDataAutoConfiguration**
 *  ElasticsearchRepositoriesAutoConfiguration

ElasticsearchTemplate组件生成位于**ElasticsearchDataAutoConfiguration**

**ElasticsearchAutoConfiguration**类型声明如下：

/** * {@link org.springframework.boot.autoconfigure.EnableAutoConfiguration * Auto-configuration} for Elasticsearch. * * @author Artur Konczak * @author Mohsin Husen * @author Andy Wilkinson * @since 1.1.0 */
    @Configuration
    @ConditionalOnClass({  Client.class, TransportClientFactoryBean.class })
    @ConditionalOnProperty(prefix = "spring.data.elasticsearch", name = "cluster-nodes", matchIfMissing = false)
    @EnableConfigurationProperties(ElasticsearchProperties.class)
    public class ElasticsearchAutoConfiguration { 
        
        private final ElasticsearchProperties properties;
    
    	public ElasticsearchAutoConfiguration(ElasticsearchProperties properties) { 
    		this.properties = properties;
    	}

### 实体类Document ###

在使用ElasticsearchTemplate时，必须存在由\*\*@Document\*\*标注的实体类，来映射索引库中某类型的映射结构

package com.example.entity;
    
    import lombok.*;
    import lombok.experimental.Accessors;
    import org.springframework.data.annotation.Id;
    import org.springframework.data.elasticsearch.annotations.Document;
    
    /** * @author songquanheng * @Time: 2020/2/27-16:58 */
    @Accessors(chain = true)
    @RequiredArgsConstructor(staticName = "of")
    @ToString
    @Document(indexName = "bigdata", type = "product")
    @Data
    public class Product { 
        @Id
        private String id;
        private String name;
        private String author;
        private String version;
        private int age;
    }

实体类要注意\*\*@Document\*\*标注的作用为文档，在该实体类上，可以指定字段的类型、分析器、该实体类所属的索引库、类型等重要信息。

@Document(indexName = "item",type = "docs",shards = 1,replicas = 0)
    public class Item { 
        @Id
        private Long id;
        @Field(type = FieldType.Text,analyzer = "ik_max_word")
        private String title;
        @Field(type=FieldType.Keyword)
        private String category;
        @Field(type=FieldType.Keyword)
        private String brand;
        @Field(type=FieldType.Double)
        private Double price;
        @Field(index = false,type = FieldType.Keyword)
        private String images;
    }

另外，要注意FieldType枚举中keyword和text的区别

1.  text类型：**会分词**，**先把对象进行分词处理**，然后再再存入到es中。

当使用多个单词进行查询的时候，当然查不到已经分词过的内容！

1.  keyword：不分词，没有把es中的对象进行分词处理，而是存入了**整个对象**！

**这时候当然可以进行完整地查询**！默认是256个字符！

Spring Data通过注解来声明字段的映射属性，有下面的三个注解：
    
    @Document 作用在类，标记实体类为文档对象，一般有两个属性
    
    indexName：对应索引库名称
    
    type：对应在索引库中的类型
    
    shards：分片数量，默认5
    
    replicas：副本数量，默认1
    
    @Id 作用在成员变量，标记一个字段作为id主键
    
    @Field 作用在成员变量，标记为文档的字段，并指定字段映射属性：
    
    type：字段类型，是枚举：FieldType，可以是text、long、short、date、integer、object等
    
    text：存储数据时候，会自动分词，并生成索引
    
    keyword：存储数据时候，不会分词建立索引
    
    Numerical：数值类型，分两类
    
    基本数据类型：long、interger、short、byte、double、float、half_float
    
    浮点数的高精度类型：scaled_float
    
    需要指定一个精度因子，比如10或100。elasticsearch会把真实值乘以这个因子后存储，取出时再还原。
    
    Date：日期类型
    
    elasticsearch可以对日期格式化为字符串存储，但是建议我们存储为毫秒值，存储为long，节省空间。
    
    index：是否索引，布尔类型，默认是true
    
    store：是否存储，布尔类型，默认是false
    
    analyzer：分词器名称，这里的ik_max_word即使用ik分词器

### JUnit测试 ###

package com.example;
    
    import com.example.entity.Item;
    import com.example.entity.Product;
    import org.elasticsearch.index.query.IdsQueryBuilder;
    import org.elasticsearch.index.query.QueryBuilder;
    import org.junit.Test;
    import org.junit.runner.RunWith;
    import org.springframework.beans.factory.annotation.Autowired;
    import org.springframework.boot.test.context.SpringBootTest;
    import org.springframework.data.elasticsearch.core.ElasticsearchTemplate;
    import org.springframework.data.elasticsearch.core.query.NativeSearchQuery;
    import org.springframework.data.elasticsearch.core.query.SearchQuery;
    import org.springframework.test.context.junit4.SpringRunner;
    
    import java.util.List;
    
    import static org.junit.Assert.assertTrue;
    
    /** * @author songquanheng * @Time: 2020/2/27-12:12 */
    @RunWith(SpringRunner.class)
    @SpringBootTest
    public class ElasticsearchTest { 
        @Autowired
        ElasticsearchTemplate elasticsearchTemplate;
        @Test
        public void indexExists() { 
            assertTrue(elasticsearchTemplate.indexExists("bigdata"));
    
        }
    
        /** * 在elasticsearch添加索引 */
        @Test
        public void createIndex() { 
            boolean res = elasticsearchTemplate.createIndex(Item.class);
            assertTrue(elasticsearchTemplate.indexExists("bigdata"));
        }
    
        @Test
        public void deleteIndex() { 
            boolean smalldata = elasticsearchTemplate.deleteIndex(Item.class);
        }
    
        /** * 检索 */
        @Test
        public void query() { 
            QueryBuilder queryBuilder = new IdsQueryBuilder().addIds("1", "2");
            SearchQuery query = new NativeSearchQuery(queryBuilder);
            List<Product> products = elasticsearchTemplate.queryForList(query, Product.class);
            products.forEach(System.out::println);
        }
    }

## 实战 ##

### 创建索引 ###

#### elasticsearchTemplate ####

/** * Create an index for a class * * @param clazz * @param <T> */
    	<T> boolean createIndex(Class<T> clazz);
    
    	/** * Create an index for given indexName * * @param indexName */
    	boolean createIndex(String indexName);
    
    	/** * Create an index for given indexName and Settings * * @param indexName * @param settings */
    	boolean createIndex(String indexName, Object settings);
    
    	/** * Create an index for given class and Settings * * @param clazz * @param settings */
    	<T> boolean createIndex(Class<T> clazz, Object settings);

> 实例

/** * 在elasticsearch添加索引 */
        @Test
        public void createIndex() { 
            elasticsearchTemplate.createIndex(Product.class);
            assertTrue(elasticsearchTemplate.indexExists("bigdata"));
        }

#### curl ####

使用curl创建索引的请求如下：

curl -XPUT "http://localhost:9200/product"

在es中，索引库必须唯一存在，这意味着，如果用户想要创建两个相同的索引库，es会报错：

### 删除索引 ###

Elasticsearch可以删除索引，但是**不能像数据库一样直接删除type**,如果想要删除type有两种方式  
a.删除index,这样会把所有该index的所有的type都会删除  
b.重新创建一个新的type,使用新的type,这种方式安全一点  
如果一个index下面只有一个type，那么就可以直接删除index  
如果一个index下面有多个type ，那么删除一个type的时候就要考虑到其他的type

#### elasticsearchTemplate ####

/** * Deletes an index for given entity * * @param clazz * @param <T> * @return */
    	<T> boolean deleteIndex(Class<T> clazz);
    
    	/** * Deletes an index for given indexName * * @param indexName * @return */
    	boolean deleteIndex(String indexName);
    
    	/** * check if index is exists * * @param clazz * @param <T> * @return */
    	<T> boolean indexExists(Class<T> clazz);

实例

#### curl ####

删除索引的curl命令表达为

curl -XDELETE "http://localhost:9200/bigdata"

### 插入文档 ###

#### elasticsearchTemplate ####

在向索引库插入文档时，主要是调用elasticsearchTemplate方法的**index**操作

/** * Index an object. Will do save or update * * @param query * @return returns the document id */
    	String index(IndexQuery query);

实际操练：

/** * 添加文档到索引中 */
        @Test
        public void addDocuments() { 
            for (int i = 0; i < 10; i++) { 
                Product product = Product.of()
                        .setAge(i + 5)
                        .setAuthor(String.format("刘德华%s", i))
                        .setId(String.valueOf(i))
                        .setName("iPad series" + i);
                IndexQueryBuilder queryBuilder = new IndexQueryBuilder()
                        .withId(product.getId())
                        .withObject(product);
                String index = elasticsearchTemplate.index(queryBuilder.build());
                System.out.println(index);
            }
    
        }

另外，可以使用es提供的bulk操作**批量**添加文档。

/** * Bulk index all objects. Will do save or update * * @param queries */
    void bulkIndex(List<IndexQuery> queries);

#### curl请求 ####

> 保存时不带id，id会自动生成

curl -XPOST "http://localhost:9200/bigdata/product" -H 'Content-Type: application/json' -d'
    {
      "name": "pencial",
      "author": "sqh",
      "version": "2.3.5",
      "age":1
    }'

> 可以携带id保存，格式如下：

curl -XPOST "http://localhost:9200/bigdata/product/5" -H 'Content-Type: application/json' -d'
    {
      "name": "Lenovo",
      "author": "sqh",
      "version": "4.5",
      "age":7
    }'

响应结果如下：

{ 
      "_index": "bigdata",
      "_type": "product",
      "_id": "5",
      "_version": 1,
      "result": "created",
      "_shards": { 
        "total": 2,
        "successful": 1,
        "failed": 0
      },
      "_seq_no": 1,
      "_primary_term": 9
    }

如果上述同样的请求再连续发两次，则可以得到如下的响应：

{ 
      "_index": "bigdata",
      "_type": "product",
      "_id": "5",
      "_version": 3,
      "result": "updated",
      "_shards": { 
        "total": 2,
        "successful": 1,
        "failed": 0
      },
      "_seq_no": 3,
      "_primary_term": 9
    }

上述要关注的字段包括\*\*“result”: “updated”**、**"\_version": 3\*\*、**"\_id": “5”**,这表示对于同一个\_id文档的操作，elasticsearch会记录对于该文档的操作，并且记录版本信息，是一种内部管理措施。

### 更新文档 ###

> 在es中，文档是不可变的，它们不能被更改，只能被替换。

es中更新文档有两种策略

*  脚本更新
 *  局部更新文档

#### elasticsearchTemplate ####

在ElasticsearchTemplate类中，用来更新文档的操作为

/** * Partial update of the document * 文档的局部更新 * @param updateQuery * @return */
    	UpdateResponse update(UpdateQuery updateQuery);
    	/** * Bulk update all objects. Will do update * * @param queries */
    	void bulkUpdate(List<UpdateQuery> queries);

更新之前，0号文档内容如下：

![20200303232124338.png][]

@Test
        public void updateDoc() { 
            // 以doc的方式更新0号id
            Product product = Product.of()
                    .setId("0")
                    .setName("易小川")
                    .setAge(53)
                    .setVersion("2.5");
            // The number of object passed must be even but was [1]
            // 如果采用doc(product)会报出如上的错误，必须把product的属性
            // 平铺成key, value的形式
            Map<String, String> update = new HashMap<>();
            update.put("name", "易小川");
            Map<String, Object> map = new HashMap<>();
            map.put("id", 0);
            map.put("name", "易小川");
            map.put("age", 53);
            map.put("version", "2.5");
    
            UpdateRequest updateRequest = new UpdateRequest(BIGDATA, PRODUCT, "0")
                    .doc(map);
            UpdateQuery updateQuery = new UpdateQueryBuilder().
                    withId("0")
                    .withIndexName(BIGDATA)
                    .withType(PRODUCT)
                    .withUpdateRequest(updateRequest)
                    .build();
            UpdateResponse updateResponse = elasticsearchTemplate.update(updateQuery);
            System.out.println(updateResponse);
    
    
        }

![20200303232124495.png][]

#### curl ####

注意更新文档为POST请求。更新API还支持传递部分文档，该文档将合并到现有文档中（简单的递归合并，对象的内部合并，替换核心"键/值"和数组）。

![更新前][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw_size_16_color_FFFFFF_t_70 12]

curl -XPOST "http://localhost:9200/bigdata/product/0/_update" -H 'Content-Type: application/json' -d'
    {
      "doc":{
        "name": "狄仁杰"
      }
    }'

执行更新操作之后，通过kibana可以看到更新之后的结果为：

![20200303232124837.png][]

注意，更新操作不要使用如下的实践方式，通过PUT请求，会导致之前其他的字段删掉，而仅剩下此次操作的字段

curl -XPUT "http://localhost:9200/bigdata/product/0" -H 'Content-Type: application/json' -d'
    {
        "name": "黄飞鸿"
    }'

![20200303232124338.png][]

显然，这不是我们的本意。

### 删除文档 ###

#### elasticsearchTemplate ####

删除文档有关的操作如下所示：

/** * Delete the one object with provided id * * @param indexName * @param type * @param id * @return documentId of the document deleted */
    	String delete(String indexName, String type, String id);
    
    
    	/** * Delete all records matching the criteria * * @param clazz * @param criteriaQuery */
    	<T> void delete(CriteriaQuery criteriaQuery, Class<T> clazz);
    	/** * Delete the one object with provided id * * @param clazz * @param id * @return documentId of the document deleted */
    	<T> String delete(Class<T> clazz, String id);
    
    	/** * Delete all records matching the query * * @param clazz * @param query */
    	<T> void delete(DeleteQuery query, Class<T> clazz);
    
    	/** * Delete all records matching the query * * @param query */
    	void delete(DeleteQuery query);

从上述的API可以看出，删除操作可以通过id来进行，亦或者通过DeleteQuery来删除符合条件的文档。

实操：

// 删除年纪从8到10的文档
        @Test
        public void delete() { 
            QueryBuilder queryBuilder = new RangeQueryBuilder("age")
                    .from(8)
                    .to(10);
            DeleteQuery deleteQuery = new DeleteQuery();
            deleteQuery.setIndex(BIGDATA);
            deleteQuery.setType(PRODUCT);
            deleteQuery.setQuery(queryBuilder);
            elasticsearchTemplate.delete(deleteQuery);
        }

操作执行之后，age属于\[8, 10\]的记录都被删除了。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw_size_16_color_FFFFFF_t_70 13][]

#### curl请求 ####

删除文档对应REST请求中的DELETE请求

curl -XDELETE "http://localhost:9200/bigdata/product/2"

如果能找到待删除的文档，则删除的响应类似如下所示：

{ 
      "_index": "bigdata",
      "_type": "product",
      "_id": "2",
      "_version": 2,
      "result": "deleted",
      "_shards": { 
        "total": 2,
        "successful": 1,
        "failed": 0
      },
      "_seq_no": 4,
      "_primary_term": 1
    }

否则，意味着删除不存在的文档，则得到如下的响应：

{ 
      "_index": "bigdata",
      "_type": "product",
      "_id": "2",
      "_version": 1,
      "result": "not_found",
      "_shards": { 
        "total": 2,
        "successful": 1,
        "failed": 0
      },
      "_seq_no": 5,
      "_primary_term": 1
    }

**"result":"not\_found"**

### 查询文档 ###

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw_size_16_color_FFFFFF_t_70 14][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw_size_16_color_FFFFFF_t_70 15][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw_size_16_color_FFFFFF_t_70 16][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw_size_16_color_FFFFFF_t_70 17][]

**Query DSL** 是 elasticsearch 的核心，搜索方面的项目大部分时间都耗费在对查询结果的调优上。因此对 Query DSL 的理解越深入，越能节省项目时间，并给用户好的体验。

Elasticsearch提供基于JSON的完整查询DSL来定义查询。将Query DSL视为查询的AST（抽象语法树），由两种类型的子句组成：

*  叶子查询子句
    
     *  叶查询子句中寻找一个特定的值在某一特定领域，如 [`match`][match]，[`term`][term]或 [`range`][range]查询。这些查询可以自己使用。
 *  复合查询子句
    
     *  复合查询子句包装其他叶查询**或**复合查询，并用于以逻辑方式组合多个查询（例如 [`bool`][bool]或[`dis_max`][dis_max]查询），或更改其行为（例如 [`constant_score`][constant_score]查询）。

> 查询字句的行为取决于它在查询上下文还是过滤器上下文中使用。

*  查询上下文是有效每当查询子句被传递给一个`query`参数，如`query`该参数[`search`][search]的API。
 *  每当将查询子句传递到`filter` 参数（例如查询中的`filter`或`must_not`参数， [`bool`][bool]查询中的`filter`参数 [`constant_score`][constant_score]或[`filter`][filter]聚合）时， 过滤器上下文即有效。

**Query DSL实例**

*  该`title`字段包含单词`search`。
 *  该`content`字段包含单词`elasticsearch`。
 *  该`status`字段包含确切的单词`published`。
 *  该`publish_date`字段包含从2015年1月1日开始的日期。

GET / _search {  " query" ：{  
    
        
        "布尔" ：{   
          "必须" ：[ {  "匹配" ：{  "标题" ："搜索" }}， 
                           
            {  " match" ：{  " content" ：" Elasticsearch" }}       
          ]，" filter" ：[
            
            {  " term" ：{  " status" ：" published" }}，       
            {  " range" ：{  " publish_date" ：{  " gte" ：" 2015-01-01" }}}        
          ] } } }

#### elasticsearchTemplate ####

通过JavaAPI来操作ES与Query DSL一摸一样，只要能尽可能熟悉即可，可以说熟悉了DSL语言之后，es的操作和使用就算入门了。

常用的QueryBuilder如下图所示：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw_size_16_color_FFFFFF_t_70 18][]

在构造Query语法树主要使用了**NativeSearchQuery**类，该类用来聚拢Query、Filter、Sort、以及用于高亮的HightLightBuilder。

public NativeSearchQuery(QueryBuilder query) { 
    		this.query = query;
    	}
    
    	public NativeSearchQuery(QueryBuilder query, QueryBuilder filter) { 
    		this.query = query;
    		this.filter = filter;
    	}
    
    	public NativeSearchQuery(QueryBuilder query, QueryBuilder filter, List<SortBuilder> sorts) { 
    		this.query = query;
    		this.filter = filter;
    		this.sorts = sorts;
    	}
    
    	public NativeSearchQuery(QueryBuilder query, QueryBuilder filter, List<SortBuilder> sorts, HighlightBuilder.Field[] highlightFields) { 
    		this.query = query;
    		this.filter = filter;
    		this.sorts = sorts;
    		this.highlightFields = highlightFields;
    	}

在查询过程中主要使用的类型继承体系如下图所示：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw_size_16_color_FFFFFF_t_70 19][]

获取bigdata索引库下product类型的所有文档的Java代码如下：

// 用于获取所有的文档
    	@Test
        public void queryAll() { 
            SearchQuery searchQuery = new NativeSearchQuery(new MatchAllQueryBuilder());
            List<Product> products = elasticsearchTemplate.queryForList(searchQuery, Product.class);
            products.forEach(System.out::println);
        }

获取name中包含iPad，并且按照age降序排列

@Test
        public void queryBySort() { 
            List<SortBuilder> sortBuilders = new ArrayList<>();
            sortBuilders.add(SortBuilders.fieldSort("age").order(SortOrder.DESC));
            SearchQuery searchQuery = new NativeSearchQuery(new MatchQueryBuilder("name", "iPad"), new MatchAllQueryBuilder(), sortBuilders);
    
            List<Product> products = elasticsearchTemplate.queryForList(searchQuery, Product.class);
            products.forEach(System.out::println);
        }

以通配符的方式获取name中包含series1?的文档

@Test
        public void queryBySort() { 
            List<SortBuilder> sortBuilders = new ArrayList<>();
            sortBuilders.add(SortBuilders.fieldSort("age").order(SortOrder.DESC));
            SearchQuery searchQuery = new NativeSearchQuery(new WildcardQueryBuilder("name", "series1?"), new MatchAllQueryBuilder(), sortBuilders);
    
            List<Product> products = elasticsearchTemplate.queryForList(searchQuery, Product.class);
            System.out.println(products.size());
            products.forEach(System.out::println);
        }

#### curl-DSL ####

查询年龄从11到13岁的记录

curl -XGET "http://localhost:9200/bigdata/product/_search" -H 'Content-Type: application/json' -d'
    {
      "query": {
        "range": {
          "age": {
            "gte": 10,
            "lte": 13
          }
        }
      }
    }'

Elasticsearch提供了一种JSON样式的特定于域的语言，可用于执行查询。这称为[查询DSL][DSL]

查询类型字段包含经济的记录、记录排序以VideoSize字段降序排列，同时进行分页处理。

match_class_order_page = { 
        "query": { 
            "match": { 
                "class1": "经济"
            }
        },
        "sort": [
            { 
                "VideoSize": { 
                    "order": "desc"
                }
            }
        ],
        "from": 1,  # 从第几个数据开始
        "size": 5   # 每页数据个数
    }

**多条件**

must :: 多个查询条件的完全匹配,相当于 and。
    
    must_not :: 多个查询条件的相反匹配，相当于 not。
    
    should :: 至少有一个查询条件匹配, 相当于 or。
    
    这些参数可以分别继承一个过滤条件或者一个过滤条件的数组

must_should = { 
        "query": { 
            "bool":{ 
                "must":[
                    { 
                        "match": { 
                            "class1": "经济"
                        }
                    },
                ],
                "filter": { 
                    "range": { 
                        "VideoSize": { 
                            "gte": 3000,
                            "lte": 5000
                        }
                    }
                }
            },
        }
    }

返回指定字段的数据

all = { 
        "query": { 
            "match_all": { }
        },
        "_source": ['id', "class1", "VioceSize"]
    }

其他对于查询的笔记如下图所示：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw_size_16_color_FFFFFF_t_70 20][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw_size_16_color_FFFFFF_t_70 21][]

查询的具体种类如下图所示：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw_size_16_color_FFFFFF_t_70 22][]

# 分词 #

中文分词

*  **ik**
 *  庖丁解牛中文分词

# 总结 #

本文主要内容是梳理了ELK技术栈中ES的入门，核心的内容有：

1.  ES的应用场景、发展历史
2.  **ES、Kibana的安装和配置**
3.  **ES Query DSL的阐述**
4.  **Spring Boot与ES集成**。
5.  通过curl来操作ES

希望为之后使用ES打下基础。

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20200303232117305.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20200303232117705.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/20200303232117996.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw_size_16_color_FFFFFF_t_70 3]: https://img-blog.csdnimg.cn/20200303232118183.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw,size_16,color_FFFFFF,t_70
[Link 1]: https://elasticsearch.cn/download/
[Elasticsearch]: https://www.elastic.co/guide/cn/elasticsearch/guide/current/running-elasticsearch.html
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw_size_16_color_FFFFFF_t_70 4]: https://img-blog.csdnimg.cn/20200303232118378.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw,size_16,color_FFFFFF,t_70
[Windows_Kibana]: https://www.elastic.co/guide/cn/kibana/current/windows.html
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw_size_16_color_FFFFFF_t_70 5]: https://img-blog.csdnimg.cn/20200303232118571.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw_size_16_color_FFFFFF_t_70 6]: https://img-blog.csdnimg.cn/20200303232118855.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw_size_16_color_FFFFFF_t_70 7]: https://img-blog.csdnimg.cn/20200303232120810.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw_size_16_color_FFFFFF_t_70 8]: https://img-blog.csdnimg.cn/2020030323212162.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw_size_16_color_FFFFFF_t_70 9]: https://img-blog.csdnimg.cn/20200303232121221.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw_size_16_color_FFFFFF_t_70 10]: https://img-blog.csdnimg.cn/2020030323212369.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw_size_16_color_FFFFFF_t_70 11]: https://img-blog.csdnimg.cn/20200303232124111.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw,size_16,color_FFFFFF,t_70
[20200303232124338.png]: https://img-blog.csdnimg.cn/20200303232124338.png
[20200303232124495.png]: https://img-blog.csdnimg.cn/20200303232124495.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw_size_16_color_FFFFFF_t_70 12]: https://img-blog.csdnimg.cn/20200303232124671.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw,size_16,color_FFFFFF,t_70
[20200303232124837.png]: https://img-blog.csdnimg.cn/20200303232124837.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw_size_16_color_FFFFFF_t_70 13]: https://img-blog.csdnimg.cn/20200303232125160.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw_size_16_color_FFFFFF_t_70 14]: https://img-blog.csdnimg.cn/20200303232126555.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw_size_16_color_FFFFFF_t_70 15]: https://img-blog.csdnimg.cn/20200303232128188.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw_size_16_color_FFFFFF_t_70 16]: https://img-blog.csdnimg.cn/20200303232129788.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw_size_16_color_FFFFFF_t_70 17]: https://img-blog.csdnimg.cn/2020030323213156.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw,size_16,color_FFFFFF,t_70
[match]: https://www.elastic.co/guide/en/elasticsearch/reference/6.0/query-dsl-match-query.html
[term]: https://www.elastic.co/guide/en/elasticsearch/reference/6.0/query-dsl-term-query.html
[range]: https://www.elastic.co/guide/en/elasticsearch/reference/6.0/query-dsl-range-query.html
[bool]: https://www.elastic.co/guide/en/elasticsearch/reference/6.0/query-dsl-bool-query.html
[dis_max]: https://www.elastic.co/guide/en/elasticsearch/reference/6.0/query-dsl-dis-max-query.html
[constant_score]: https://www.elastic.co/guide/en/elasticsearch/reference/6.0/query-dsl-constant-score-query.html
[search]: https://www.elastic.co/guide/en/elasticsearch/reference/6.0/search-request-query.html
[filter]: https://www.elastic.co/guide/en/elasticsearch/reference/6.0/search-aggregations-bucket-filter-aggregation.html
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw_size_16_color_FFFFFF_t_70 18]: https://img-blog.csdnimg.cn/20200303232131315.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw_size_16_color_FFFFFF_t_70 19]: https://img-blog.csdnimg.cn/20200303232131505.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw,size_16,color_FFFFFF,t_70
[DSL]: https://www.elastic.co/guide/en/elasticsearch/reference/6.0/query-dsl.html
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw_size_16_color_FFFFFF_t_70 20]: https://img-blog.csdnimg.cn/20200303232132791.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw_size_16_color_FFFFFF_t_70 21]: https://img-blog.csdnimg.cn/20200303232135539.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw_size_16_color_FFFFFF_t_70 22]: https://img-blog.csdnimg.cn/20200303232135778.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xrMTQyNTAw,size_16,color_FFFFFF,t_70