ES实时刷新与n秒刷新，压测性能耗损情况

野性酷女 2022-06-04 06:59 128阅读 0赞

ES压测  
机器配置： CPU：4核     内存：8G     两台集群机器  ，以下入ES都是单个操作，批量操作所有耗时会相对变短。  
  
一、10万数据  
1、50个线程（实时刷新，设置方法参考： [http://blog.csdn.net/wwd0501/article/details/78316581][http_blog.csdn.net_wwd0501_article_details_78316581]）

**①创建索引后，默认1s更新**

<table style="border-collapse:collapse;margin:0px;color:rgb(51,51,51);font-family:Arial, sans-serif;font-size:14px;"> 
 <tbody> 
  <tr> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">耗时</td> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">131437ms(2分钟多)&nbsp;&nbsp;</td> 
  </tr> 
  <tr> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">丢索引情况</td> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">无丢失&nbsp;</td> 
  </tr> 
  <tr> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">CPU情况</td> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">100%左右</td> 
  </tr> 
  <tr> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">内存使用情况</td> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">30%左右</td> 
  </tr> 
  <tr> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">异常情况</td> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">无</td> 
  </tr> 
 </tbody> 
</table>

**②创建索引后实时更新**

<table style="border-collapse:collapse;margin:0px;color:rgb(51,51,51);font-family:Arial, sans-serif;font-size:14px;"> 
 <tbody> 
  <tr> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">耗时</td> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">1647341ms(27分钟多)&nbsp;&nbsp;&nbsp;</td> 
  </tr> 
  <tr> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">丢索引情况</td> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">无丢失</td> 
  </tr> 
  <tr> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">CPU情况</td> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">CPU经常300%~380%</td> 
  </tr> 
  <tr> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">内存使用情况</td> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">30%左右</td> 
  </tr> 
  <tr> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">异常情况</td> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">无</td> 
  </tr> 
 </tbody> 
</table>

ES的实时刷新性能比默认的1s刷新性能下降10倍。  
  
2、100个线程（似乎100个线程没有全都用上）

**①创建索引后，默认1s更新**

<table style="border-collapse:collapse;margin:0px;color:rgb(51,51,51);font-family:Arial, sans-serif;font-size:14px;"> 
 <tbody> 
  <tr> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">耗时</td> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">140639ms&nbsp;&nbsp;</td> 
  </tr> 
  <tr> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">丢索引情况</td> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">无丢失&nbsp;</td> 
  </tr> 
  <tr> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">CPU情况</td> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">100%左右</td> 
  </tr> 
  <tr> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">内存使用情况</td> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">30%左右</td> 
  </tr> 
  <tr> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">异常情况</td> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">无</td> 
  </tr> 
 </tbody> 
</table>

**②创建索引后实时更新**

<table style="border-collapse:collapse;margin:0px;color:rgb(51,51,51);font-family:Arial, sans-serif;font-size:14px;"> 
 <tbody> 
  <tr> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">耗时</td> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">847928ms&nbsp;&nbsp;&nbsp;</td> 
  </tr> 
  <tr> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">丢索引情况</td> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">无丢失</td> 
  </tr> 
  <tr> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">CPU情况</td> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">CPU使用不断增加，后期维持在380%(CPU用尽)</td> 
  </tr> 
  <tr> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">内存使用情况</td> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">30%左右</td> 
  </tr> 
  <tr> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">异常情况</td> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">无</td> 
  </tr> 
 </tbody> 
</table>

3、设置30s更新，索引创建后，隔30s刷新（设置方式参考： [http://blog.csdn.net/wwd0501/article/details/78316581][http_blog.csdn.net_wwd0501_article_details_78316581]）

50个线程

<table style="border-collapse:collapse;margin:0px;"> 
 <tbody> 
  <tr> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">耗时</td> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">279409ms&nbsp;</td> 
  </tr> 
  <tr> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">丢索引情况</td> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">无丢失</td> 
  </tr> 
  <tr> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">CPU情况</td> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">基本都是10%以下，偶尔60%左右</td> 
  </tr> 
  <tr> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">内存使用情况</td> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">30%左右</td> 
  </tr> 
  <tr> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">异常情况</td> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">无</td> 
  </tr> 
 </tbody> 
</table>

二、100万数据  
1、50个线程

①创建索引后，默认1s更新

<table style="border-collapse:collapse;margin:0px;color:rgb(51,51,51);font-family:Arial, sans-serif;font-size:14px;"> 
 <tbody> 
  <tr> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">耗时</td> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">2246360ms&nbsp;&nbsp;</td> 
  </tr> 
  <tr> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">丢索引情况</td> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">无丢失&nbsp;</td> 
  </tr> 
  <tr> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">CPU情况</td> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">有段时间30%左右，大部分100%左右（波动至100%时，索引创建很快，有可能是网络不好导致请求没有到达ES）</td> 
  </tr> 
  <tr> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">内存使用情况</td> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">30%左右</td> 
  </tr> 
  <tr> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">异常情况</td> 
   <td style="border:1px solid rgb(221,221,221);padding:7px 10px;vertical-align:top;min-width:8px;">无</td> 
  </tr> 
 </tbody> 
</table>

②创建索引后实时更新  太耗时，没有压测完。

Java压测代码：

package test;
    
    import java.util.Random;
    import java.util.UUID;
    import java.util.concurrent.CountDownLatch;
    import java.util.concurrent.ExecutorService;
    import java.util.concurrent.Executors;
    import java.util.concurrent.atomic.AtomicLong;
    
    import org.elasticsearch.action.index.IndexResponse;
    import org.elasticsearch.action.support.WriteRequest.RefreshPolicy;
    import org.elasticsearch.client.Client;
    
    import product.Product;
    
    import com.alibaba.fastjson.JSON;
    
    /**
     * 
     * Description: ES实时刷新，压测性能耗损情况
     * 
     *     		//设置实时刷新
        		.setRefreshPolicy(RefreshPolicy.IMMEDIATE)
     * CreateTime: 2017年9月23日  下午2:20:14
     *
     */
    public class EsTest {
    
    	private static int count = 50;
    	private static ExecutorService cachedThreadPool = Executors.newFixedThreadPool(count);
    	
    	public static void main(String[] args) {
        	try {
        		System.out.println("es client init start");
        		long start = System.currentTimeMillis();
        		Client client = ElasticsearchConfiguration.getClient();
        		System.out.println("es client init end:"+(System.currentTimeMillis()-start)+"ms");
        		
        		System.out.println("====================test start");
        		start = System.currentTimeMillis();
        		AtomicLong productId=new AtomicLong();
        		CountDownLatch countDownLatch = new CountDownLatch(count);
        		for (int i = 0; i < count; i++) {
        			cachedThreadPool.execute(new Runnable() {
        				@Override
        				public void run() {
        					System.out.println(1);
        					addProduct(productId, client);
        					countDownLatch.countDown();
        				}
        			});
    			}
        		
            	countDownLatch.await();
                System.out.println("总耗时：" + (System.currentTimeMillis()-start) + "ms");
                
                cachedThreadPool.shutdown();
    		} catch (Exception e) {
    			e.printStackTrace();
    		}
        }
    	
    	private static void addProduct(AtomicLong productId,Client client) {
    		int i = 0;
    		Product product = null;
    		for (; i < 20000; i++) {
    			try {
    				product = new Product();
    				product.setProductID(productId.incrementAndGet());
    				product.setProductName(getProductName());
    				product.setProductBrief("商品简介:"+UUID.randomUUID());
    				product.setPlaceProvince("河北省");
    				product.setPlaceCity("固安");
    				product.setProducer("獐子岛");
    				
    				//设置实时刷新
    				IndexResponse response = client.prepareIndex("wareic-bak","product",product.getProductID().toString()).setSource(JSON.toJSONString(product)).setRefreshPolicy(RefreshPolicy.IMMEDIATE)
    						.get();  
    			} catch (Exception e) {
    				e.printStackTrace();
    			}
    		}
    		System.out.println(i);
    	}
        
        private static String getProductName(){
        	String[]productNameArray={"西红柿番茄","西葫芦","宜家清新简约马克杯","螃蟹","文件柜","海鲜","电热水壶"};
        	return productNameArray[new Random().nextInt(6)]+new Random().nextInt(100000000);
        }
    }

ElasticsearchConfiguration类

import org.elasticsearch.client.Client;
    import org.elasticsearch.client.transport.TransportClient;
    import org.elasticsearch.common.settings.Settings;
    import org.elasticsearch.common.transport.InetSocketTransportAddress;
    import org.elasticsearch.transport.client.PreBuiltTransportClient;
    
    import java.net.InetAddress;
    import java.net.URI;
    import java.net.UnknownHostException;
    import java.util.Arrays;
    import java.util.List;
    
    public class ElasticsearchConfiguration {
    
        private static TransportClient client;
    
        private static String clusterName;
        private static List<String> clusterNodes;
    
        static {
            clusterName = "shopmall-es";
            clusterNodes = Arrays.asList("http://172.16.32.69:9300","http://172.16.32.48:9300");
        }
    
        private ElasticsearchConfiguration() {
        }
    
        public static Client getClient() throws UnknownHostException {
            Settings settings = Settings.builder().put("cluster.name", clusterName).build();
            client = new PreBuiltTransportClient(settings);
            for (String node : clusterNodes) {
                URI host = URI.create(node);
                client.addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName(host.getHost()), host.getPort()));
            }
            return client;
        }
    
        public static void close() {
            client.close();
        }
    }

Product类

package test;
    
    
    public class Product {
        /**
         * 商品ID
         */
        private Long productID;
    
        /**
         * 商品编码
         */
        private String productCode;
    
        /**
         * 商品名称
         */
        private String productName;
    
        /**
         * 商品简介
         */
        private String productBrief;
    
        /**
         * 商品主图URL地址
         */
        private String imgUrl;
    
        /**
         * 商品产地省
         */
        private String placeProvince;
    
        /**
         * 商品产地市
         */
        private String placeCity;
    
        /**
         * 生产厂家
         */
        private String producer;
    
    	public Long getProductID() {
    		return productID;
    	}
    
    	public void setProductID(Long productID) {
    		this.productID = productID;
    	}
    
    	public String getProductCode() {
    		return productCode;
    	}
    
    	public void setProductCode(String productCode) {
    		this.productCode = productCode;
    	}
    
    	public String getProductName() {
    		return productName;
    	}
    
    	public void setProductName(String productName) {
    		this.productName = productName;
    	}
    
    	public String getProductBrief() {
    		return productBrief;
    	}
    
    	public void setProductBrief(String productBrief) {
    		this.productBrief = productBrief;
    	}
    
    	public String getImgUrl() {
    		return imgUrl;
    	}
    
    	public void setImgUrl(String imgUrl) {
    		this.imgUrl = imgUrl;
    	}
    
    	public String getPlaceProvince() {
    		return placeProvince;
    	}
    
    	public void setPlaceProvince(String placeProvince) {
    		this.placeProvince = placeProvince;
    	}
    
    	public String getPlaceCity() {
    		return placeCity;
    	}
    
    	public void setPlaceCity(String placeCity) {
    		this.placeCity = placeCity;
    	}
    
    	public String getProducer() {
    		return producer;
    	}
    
    	public void setProducer(String producer) {
    		this.producer = producer;
    	}
    
    }

[http_blog.csdn.net_wwd0501_article_details_78316581]: http://blog.csdn.net/wwd0501/article/details/78316581