elasticsearch5.3.0 bulk index 性能调优实践

快来打我* 2022-06-07 13:56 297阅读 0赞

> 导语： 腾讯云CDN上每天产生大量回源日志，回源日志通常用在问题定位的时候比较有用。这里使用filebeat+logstash+elasticsearch的方案收集、存储日志数据并提供查询。当前的使用场景里，每天有70亿条日志需要存储，属于写多读少的场景。本文整理了在搭建elasticsearch集群的时候需要注意的配置项，通过对这些配置项的调整，期望提高elasticsearch写入的性能。

## 一、集群基本信息 ##

### 1. 每天日志量70亿 ###

### 2. ES版本： 5.3.0 ###

### 3. 机器部署 ###

*  master node： 3台
 *  data node： TS60\*10
 *  client node： 2台

## 二、 写入优化措施及说明 ##

未进行配置优化之前，ES集群负载非常高，主要表现在磁盘IO上，写入的qps在2万/s左右  
为了提高写入的性能，笔者搜集了官方文档以及一些资料，有如下优化措施。

### 1. [index.refresh\_interval][index.refresh_interval]: "30s" ###

*  优化点： 减少刷新频率，降低潜在的写磁盘性能损耗
 *  官网解释： How often to perform a refresh operation, which makes recent changes to the index visible to search. Defaults to 1s. Can be set to -1 to disable refresh.

### 2. [translog优化][translog] ###

*  优化点： 减少写磁盘的频率
    
        python
        {
          "index": {
              "translog": {
                  "flush_threshold_size": "1gb",
                  "sync_interval": "30s",
                  "durability": "async"
              }
          }
        }
 *  Lucene只有在commit的时候才会把之前的变更持久化存储到磁盘（每次操作都写到磁盘的话，代价太大），在commit之前如果出现故障，上一次commit之后的变更都会丢失
 *  为了防止数据丢失，Lucene会把变更操作都记录在translog里，在出现故障的时候，从上次commit起记录在translog里的变更都可以恢复，尽量保证数据不丢失
 *  Lucene的flush操作就是执行一次commit，同时开始记录一个新的translog，所以translog是用来记录从上次commit到下一次commit之间的操作的
 *  flush操作的频率是通过translog的大小控制的，当translog大小达到一定值的时候就执行一次flush，对应参数为index.translog.flush\_threshold\_size，默认值是512mb，这里调整为1gb，减少flush的次数
 *  translog本身是文件，也需要存储到磁盘，它的存储方式通过index.translog.durability和index.translog.sync\_interval设定。默认情况下，index.translog.durability=request，意为每次请求都会把translog写到磁盘。这种设定可以降低数据丢失的风险，但是磁盘IO开销会较大
 *  这里采用异步方式持久化translog，每隔30秒写一次磁盘

### 3. index.store.throttle.type: "none" ###

*  优化点： 放开merge操作，加快合并，[防止因为merge滞后导致index操作被限速][merge_index]
 *  [1.4的文档][1.4]关于这个参数的解释：  
    index操作首先会生成很多小的segment，会有异步逻辑合并（merge）这些segment  
    merge操作比较消耗IO，当系统IO性能比较差的时候，merge会影响查询和索引的性能。  
    index.store.throttle.type和index.store.throttle.max\_bytes\_per\_sec可以在节点级或者index级限制merge操作消耗的磁盘带宽，防止因为merge导致磁盘高负载，影响其他操作
 *  另一篇关于[ES2.x index调优的文章][ES2.x index]里讲到，如果不关心查询的性能，可以把index.store.throttle.type设为none，意为不对merge操作限速  
    这个参数默认配置是针对merge操作限制使用磁盘带宽20MBps

### 4. [index.merge.scheduler.max\_thread\_count][index.merge.scheduler.max_thread_count]: 1 ###

*  优化点： 减少并发并发merge对磁盘的消耗
 *  index由多个shard组成，每个shard又分成很多segment，segment是index数据存储的最小单位
 *  执行索引操作时，[ES会先生成小的segment][ES_segment]
 *  segment比较多的时候会影响搜索性能（要查询很多segment），ES有离线的逻辑对小的segment进行合并，优化查询性能。但是合并过程中会消耗较多磁盘IO，会影响查询性能
 *  index.merge.scheduler.max\_thread\_count控制并发的merge线程数，如果存储是并发性能较好的SSD，可以用系统默认的max(1, min(4, availableProcessors / 2))，普通磁盘的话设为1

### 5. [indices.memory.index\_buffer\_size][indices.memory.index_buffer_size]: "20%" ###

*  优化点： 降低被动写磁盘的可能性
 *  该配置项指定了用于索引操作的内存大小，索引的结果先存在内存中，缓存空间满了的话，缓存的内容会以segment为单位写到磁盘。显然，增大缓存空间大小可以降低被动写磁盘的频率

## 三、 优化思路梳理 ##

### 1. 目标： 优化index性能 ###

### 2. 主要降低index性能的因素： 磁盘IO（基于观察，负载非常高） ###

### 3. 哪些操作在消耗磁盘IO ###

*  index refresh（不确定有没有写磁盘，待确认）
 *  Lucene flush
 *  translog persistant
 *  index buffer不足导致被动写磁盘
 *  segment merge

关于segment merge有两个调整，（1）减少并发merge的线程数，（2）放开merge的磁盘带宽限制。这里猜测是因为，merge操作是要做的，但是并发的merge比较耗磁盘IO，折中的方案是减少并发，加强单线程merge

## 四、 测试 ##

以上配置全部用上之后，集群负载瞬间降低了，但是不清楚是哪个配置，或者哪些配置的影响比较大，下面通过测试确认了这些配置的影响

### 1. 测试方法 ###

使用现网流量进行测试  
将上述全部优化项启用的时候作为基准，分别禁用单个优化项，观察ES写入性能和集群负载

### 2. 测试日期：20170923 ###

### 3. 测试数据 ###

<table style="border-collapse:collapse; border-spacing:0px; list-style:inherit; margin-top:0px; margin-bottom:16px; display:block; width:748.984px; overflow:auto; word-break:keep-all; color:rgb(102,102,102); font-family:&quot;helvetica neue&quot;,PingFangSC-Light,arial,&quot;hiragino sans gb&quot;,&quot;microsoft yahei ui&quot;,&quot;microsoft yahei&quot;,simsun,sans-serif; font-size:16px"> 
 <thead style="list-style:inherit"> 
  <tr style="list-style:inherit; border-top:1px solid rgb(204,204,204)"> 
   <th style="margin:0px; padding:6px 13px; list-style:inherit; background-color:rgb(248,248,248); border:1px solid rgb(221,221,221); word-wrap:break-word; word-break:inherit"> 对比组</th> 
   <th style="margin:0px; padding:6px 13px; list-style:inherit; background-color:rgb(248,248,248); border:1px solid rgb(221,221,221); word-wrap:break-word; word-break:inherit"> 时间</th> 
   <th style="margin:0px; padding:6px 13px; list-style:inherit; background-color:rgb(248,248,248); border:1px solid rgb(221,221,221); word-wrap:break-word; word-break:inherit"> qps</th> 
   <th style="margin:0px; padding:6px 13px; list-style:inherit; background-color:rgb(248,248,248); border:1px solid rgb(221,221,221); word-wrap:break-word; word-break:inherit"> 负载</th> 
   <th style="margin:0px; padding:6px 13px; list-style:inherit; background-color:rgb(248,248,248); border:1px solid rgb(221,221,221); word-wrap:break-word; word-break:inherit"> 说明</th> 
  </tr> 
 </thead> 
 <tbody style="list-style:inherit"> 
  <tr style="list-style:inherit; border-top:1px solid rgb(204,204,204)"> 
   <td style="margin:0px; padding:6px 13px; list-style:inherit; border:1px solid rgb(221,221,221); word-wrap:break-word"> 基准</td> 
   <td style="margin:0px; padding:6px 13px; list-style:inherit; border:1px solid rgb(221,221,221); word-wrap:break-word"> 14:30~15:00</td> 
   <td style="margin:0px; padding:6px 13px; list-style:inherit; border:1px solid rgb(221,221,221); word-wrap:break-word"> 61685</td> 
   <td style="margin:0px; padding:6px 13px; list-style:inherit; border:1px solid rgb(221,221,221); word-wrap:break-word"> 3</td> 
   <td style="margin:0px; padding:6px 13px; list-style:inherit; border:1px solid rgb(221,221,221); word-wrap:break-word"> -</td> 
  </tr> 
  <tr style="list-style:inherit; border-top:1px solid rgb(204,204,204)"> 
   <td style="margin:0px; padding:6px 13px; list-style:inherit; border:1px solid rgb(221,221,221); word-wrap:break-word"> 并发merge线程数设为默认（15:09）</td> 
   <td style="margin:0px; padding:6px 13px; list-style:inherit; border:1px solid rgb(221,221,221); word-wrap:break-word"> 15:30~16:00</td> 
   <td style="margin:0px; padding:6px 13px; list-style:inherit; border:1px solid rgb(221,221,221); word-wrap:break-word"> 64831</td> 
   <td style="margin:0px; padding:6px 13px; list-style:inherit; border:1px solid rgb(221,221,221); word-wrap:break-word"> 3</td> 
   <td style="margin:0px; padding:6px 13px; list-style:inherit; border:1px solid rgb(221,221,221); word-wrap:break-word"> 刚修改配置qps有个小凸尖，随后平稳</td> 
  </tr> 
  <tr style="list-style:inherit; border-top:1px solid rgb(204,204,204)"> 
   <td style="margin:0px; padding:6px 13px; list-style:inherit; border:1px solid rgb(221,221,221); word-wrap:break-word"> <span style="list-style:inherit; font-weight:700">禁用translog优化（16:07）</span></td> 
   <td style="margin:0px; padding:6px 13px; list-style:inherit; border:1px solid rgb(221,221,221); word-wrap:break-word"> 16:12~16:22</td> 
   <td style="margin:0px; padding:6px 13px; list-style:inherit; border:1px solid rgb(221,221,221); word-wrap:break-word"> <span style="list-style:inherit; font-weight:700">18399</span></td> 
   <td style="margin:0px; padding:6px 13px; list-style:inherit; border:1px solid rgb(221,221,221); word-wrap:break-word"> <span style="list-style:inherit; font-weight:700">39</span></td> 
   <td style="margin:0px; padding:6px 13px; list-style:inherit; border:1px solid rgb(221,221,221); word-wrap:break-word"> <span style="list-style:inherit; font-weight:700">qps暴跌，负载猛增</span></td> 
  </tr> 
  <tr style="list-style:inherit; border-top:1px solid rgb(204,204,204)"> 
   <td style="margin:0px; padding:6px 13px; list-style:inherit; border:1px solid rgb(221,221,221); word-wrap:break-word"> refresh interval使用默认值1s（16:28）</td> 
   <td style="margin:0px; padding:6px 13px; list-style:inherit; border:1px solid rgb(221,221,221); word-wrap:break-word"> 16:31~17:01</td> 
   <td style="margin:0px; padding:6px 13px; list-style:inherit; border:1px solid rgb(221,221,221); word-wrap:break-word"> 57012</td> 
   <td style="margin:0px; padding:6px 13px; list-style:inherit; border:1px solid rgb(221,221,221); word-wrap:break-word"> 5</td> 
   <td style="margin:0px; padding:6px 13px; list-style:inherit; border:1px solid rgb(221,221,221); word-wrap:break-word"> qps比基准微降，负载微曾</td> 
  </tr> 
  <tr style="list-style:inherit; border-top:1px solid rgb(204,204,204)"> 
   <td style="margin:0px; padding:6px 13px; list-style:inherit; border:1px solid rgb(221,221,221); word-wrap:break-word"> 开启merge限流（17:05）</td> 
   <td style="margin:0px; padding:6px 13px; list-style:inherit; border:1px solid rgb(221,221,221); word-wrap:break-word"> 17:10~17:20</td> 
   <td style="margin:0px; padding:6px 13px; list-style:inherit; border:1px solid rgb(221,221,221); word-wrap:break-word"> 61862</td> 
   <td style="margin:0px; padding:6px 13px; list-style:inherit; border:1px solid rgb(221,221,221); word-wrap:break-word"> 2.5</td> 
   <td style="margin:0px; padding:6px 13px; list-style:inherit; border:1px solid rgb(221,221,221); word-wrap:break-word"> 和基准持平</td> 
  </tr> 
 </tbody> 
</table>

### 4. 压测 ###

按照所有优化项开启的设定，使用[python api][]开启多线程向ES集群发起bulk index请求，同时观察kibana monitor界面，发现index qps达到30w/s，随后集群很快挂掉。

## 五、结论 ##

以上测试可以看到

*  translog优化禁用的时候集群负载最高，写入性能最差
 *  index refresh操作对写入的性能略有影响
 *  segment merge对写入性能影响非常小

## 附录 ##

### 1. ES存储节点完整配置 ###

cluster.routing.allocation.enable: all
    bootstrap.system_call_filter: false
    discovery.zen.ping.unicast.hosts: ["host0", "host1", "host2"]
    thread_pool.bulk.queue_size: 6000
    network.host: ${lanip}
    cluster.routing.allocation.node_concurrent_recoveries: 128
    path.data: 
       - /data1/cdn_log/data
       - /data2/cdn_log/data
       - /data3/cdn_log/data
       - /data4/cdn_log/data
       - /data5/cdn_log/data
       - /data6/cdn_log/data
       - /data7/cdn_log/data
       - /data8/cdn_log/data
       - /data9/cdn_log/data
       - /data10/cdn_log/data
       - /data11/cdn_log/data
       - /data12/cdn_log/data
    node.master: false
    path.logs: /data/log/hy-es
    discovery.zen.minimum_master_nodes: 2
    bootstrap.memory_lock: false
    node.data: true
    #node.attr.rack: r1
    http.port: 9200
    gateway.recover_after_data_nodes: 7
    node.name: node-5
    gateway.recover_after_master_nodes: 2
    cluster.name: your_cluster_name
    xpack.security.enabled: false
    action.destructive_requires_name: true
    indices.recovery.max_bytes_per_sec: 200mb
    indices.memory.index_buffer_size: 20%

### 2. index template配置 ###

{
      "order": 0,
      "template": "hy-log-*",
      "settings": {
        "index": {
          "refresh_interval": "30s",
          "number_of_shards": "20",
          "translog": {
            "flush_threshold_size": "1gb",
            "sync_interval": "30s",
            "durability": "async"
          },
          "number_of_replicas": "0"
        }
      },
      "mappings": {
        "_default_": {
          "dynamic_templates": [
            {
              "strings_as_keywords": {
                "mapping": {
                  "index": "not_analyzed"
                },
                "match_mapping_type": "string"
              }
            }
          ]
        }
      },
      "aliases": {}
    }

[index.refresh_interval]: https://www.elastic.co/guide/en/elasticsearch/reference/5.3/index-modules.html
[translog]: https://www.elastic.co/guide/en/elasticsearch/reference/5.3/index-modules-translog.html
[merge_index]: https://www.datadoghq.com/blog/elasticsearch-performance-scaling-problems/
[1.4]: https://www.elastic.co/guide/en/elasticsearch/reference/1.4/index-modules-store.html#store-throttling
[ES2.x index]: https://www.elastic.co/guide/en/elasticsearch/guide/current/indexing-performance.html
[index.merge.scheduler.max_thread_count]: https://www.elastic.co/guide/en/elasticsearch/reference/5.3/index-modules-merge.html
[ES_segment]: https://stackoverflow.com/questions/15426441/understanding-segments-in-elasticsearch
[indices.memory.index_buffer_size]: https://www.elastic.co/guide/en/elasticsearch/reference/5.3/indexing-buffer.html
[python api]: https://elasticsearch-py.readthedocs.io/en/master/