centos7搭建日志服务elasticsearch6.3.2

Love The Way You Lie 2022-05-16 07:51 556阅读 0赞

## centOS7 搭建日志服务 ##

### 版本 ###

elasticsearch-6.3.2、kibana-6.3.2、filebeat-6.3.2

### 参考链接 ###

[elasticsearch6.3.1 启用x-pack][elasticsearch6.3.1 _x-pack]

[安装Kibana6.3.2与Elasticsearch6.3.2集成][Kibana6.3.2_Elasticsearch6.3.2]

[x-pack6.3破解][x-pack6.3]

[filebeat 5.2.2详细配置说明][filebeat 5.2.2]

[logstash 启动缓慢问题-haveged安装][logstash _-haveged]

[elasticsearch start more than one nodes on the same data folder][]

[elasticsearch6.3.2集群安装][elasticsearch6.3.2]

### 一、elasticsearch ###

#### 1、安装 rz sz ####

sudo yum install lrzsz -y

#### 2、安装 jdk ####

/usr/local ------rpm -ivh jdk-8u152-linux-x64.rpm

#### 3、下载 elasticsearch ####

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.3.2.tar.gz
    tar -zxvf /usr/local/elasticsearch-6.3.2.tar.gz -C /usr/local/

#### 4、elasticsearch配置 ####

elasticsearch不能用root用户启动

改变elasticsearch文件夹所有者到当前用户

sudo chown -R noroot:noroot /usr/local/elasticsearch-6.3.2

设置配置文件elasticsearch.yml

$ grep -Ev "^$|^[#;]" elasticsearch.yml 
    cluster.name: es
    node.name: node-1
    node.master: true
    node.data: true
    path.data: /usr/local/elasticsearch-6.3.2/data
    path.logs: /usr/local/elasticsearch-6.3.2/logs
    network.host: 0.0.0.0
    http.port: 9200
    #xpack.security.enabled: true
    #xpack.security.transport.ssl.enabled: true
    #http.cors.enabled: true
    #http.cors.allow-origin: "*"
    #http.cors.allow-headers: Authorization,X-Requested-With,Content-Length,Content-Type

#### 5、启动 ####

nohup ./elasticsearch >/dev/null 2>&1 &

#### 6、 x-pack ####

##### 6.1、启用 x-pack #####

elasticsearch6.3.2 安装包中已经集成了 x-pack 插件，无需使用 elasticsearch-plugin install 来安装 x-pack ，我们需要做的是使 x-pack 工作

curl -H "Content-Type:application/json" -XPOST http://localhost:9200/_xpack/license/start_trial?acknowledge=true
    [2018-08-14T19:41:05,208][INFO ][o.e.l.LicenseService     ] [node-1] license [1fe2f382-c8cc-41d6-8d3a-d11d71ad2004] mode [basic] - valid
    [2018-08-14T19:41:05,220][INFO ][o.e.g.GatewayService     ] [node-1] recovered [0] indices into cluster_state
    [2018-08-14T19:43:24,005][INFO ][o.e.l.LicenseService     ] [node-1] license [37b55178-574f-47a8-aedf-dd20c6f710a6] mode [trial] - valid

可以看到elasticsearch控制台显示license 已变为trial

##### 6.2、设置用户名密码 #####

$ bin/elasticsearch-setup-passwords interactive
    Initiating the setup of passwords for reserved users elastic,kibana,logstash_system,beats_system.
    You will be prompted to enter passwords as the process progresses.
    Please confirm that you would like to continue [y/N]y
    
    
    Enter password for [elastic]:
    Reenter password for [elastic]:
    Enter password for [kibana]:
    Reenter password for [kibana]:
    Enter password for [logstash_system]:
    Reenter password for [logstash_system]:
    Enter password for [beats_system]:
    Reenter password for [beats_system]:
    Changed password for user [kibana]
    Changed password for user [logstash_system]
    Changed password for user [beats_system]
    Changed password for user [elastic]

##### 6.3、elasticsearch.yml 中开启安全验证配置 #####

此时访问 [http://localhost:9200][http_localhost_9200] 并不需要输入账号密码，需 elasticsearch.yml 中添加如下配置：

xpack.security.enabled: true
    #xpack.security.transport.ssl.enabled: true

重启 elasticsearch 之后 [http://localhost:9200][http_localhost_9200] 就需要使用账号验证了

###### 6.4、查看license ######

curl -u elastic:123456 "http://localhost:9200/_license"

##### 6.5、更新license #####

curl -XPUT -u elastic 'http://127.0.0.1:9200/_xpack/license?acknowledge=true' -H "Content-Type: application/json" -d @license.json

##### 6.6、x-pack 破解 #####

elk6.3版本之后，x-pack都是默认安装，无需install

分别创建两个文件

vim LicenseVerifier.java

package org.elasticsearch.license;
    import java.nio.*; import java.util.*;
    import java.security.*;
    import org.elasticsearch.common.xcontent.*;
    import org.apache.lucene.util.*;
    import org.elasticsearch.common.io.*;
    import java.io.*;
    
    public class LicenseVerifier {
        public static boolean verifyLicense(final License license, final byte[] encryptedPublicKeyData) {
            return true;
        }
    
        public static boolean verifyLicense(final License license)     {
            return true;
        }
    }

vim XPackBuild.java

package org.elasticsearch.xpack.core;
    import org.elasticsearch.common.io.*;
    import java.net.*;
    import org.elasticsearch.common.*;
    import java.nio.file.*;
    import java.io.*;
    import java.util.jar.*;
    public class XPackBuild {
        public static final XPackBuild CURRENT;
        private String shortHash;
        private String date;
        @SuppressForbidden(reason = "looks up path of xpack.jar directly") static Path getElasticsearchCodebase() {
            final URL url = XPackBuild.class.getProtectionDomain().getCodeSource().getLocation();
            try { return PathUtils.get(url.toURI()); }
            catch (URISyntaxException bogus) {
                throw new RuntimeException(bogus); }
            }
    
        XPackBuild(final String shortHash, final String date) {
                this.shortHash = shortHash;
                this.date = date;
                }
    
        public String shortHash() {
            return this.shortHash;
            }
        public String date(){
            return this.date;
            }
    
        static {
            final Path path = getElasticsearchCodebase();
            String shortHash = null;
            String date = null;
            Label_0157: { shortHash = "Unknown"; date = "Unknown";
        }
    
        CURRENT = new XPackBuild(shortHash, date);
        }
    }

将刚创建的两个java包打包成class文件，我们需要做的就是替换这两个class文件（因里面需要引用到其他的jar，故需要用到javac -cp命令）

#编译成class文件
    javac -cp "/usr/local/elasticsearch-6.3.2/lib/elasticsearch-6.3.2.jar:/usr/local/elasticsearch-6.3.2/lib/lucene-core-7.3.1.jar:/usr/local/elasticsearch-6.3.2/modules/x-pack/x-pack-core/x-pack-core-6.3.2.jar:/usr/local/elasticsearch-6.3.2/lib/elasticsearch-core-6.3.2.jar" XPackBuild.java
    
    #编译成class文件
    sudo javac -cp "/usr/local/elasticsearch-6.3.2/lib/elasticsearch-6.3.2.jar:/usr/local/elasticsearch-6.3.2/lib/lucene-core-7.3.1.jar:/usr/local/elasticsearch-6.3.2/modules/x-pack/x-pack-core/x-pack-core-6.3.2.jar:/usr/local/elasticsearch-6.3.2/lib/elasticsearch-core-6.3.2.jar" XPackBuild.java
    
    #创建解压目录
    sudo mkdir x-pack-core-6.3.2
    #解压到目标目录
    sudo unzip x-pack-core-6.3.2.jar -d x-pack-core-6.3.2
    #将生成的class文件 copy 到 解压后的文件夹
    sudo cp -a LicenseVerifier.class x-pack-core-6.3.2/org/elasticsearch/license/
    sudo cp -a XPackBuild.class x-pack-core-6.3.2/org/elasticsearch/xpack/core/
    #进入到解压目录 压缩成 jar文件
    sudo jar -cvf x-pack-core-6.3.2.jar *
    #旧jar包备份
    sudo cp /usr/local/elasticsearch-6.3.2/modules/x-pack/x-pack-core/x-pack-core-6.3.2.jar /usr/local/elasticsearch-6.3.2/modules/x-pack/x-pack-core/x-pack-core-6.3.2.jar.bak
    #新生成 jar 包替换
    sudo cp x-pack-core-6.3.2.jar /usr/local/elasticsearch-6.3.2/modules/x-pack/x-pack-core/x-pack-core-6.3.2.jar

破解后的文件下载地址 [x-pack-core-6.3.2.jar破解版][x-pack-core-6.3.2.jar]

启动elasticsearch

替换 license

官网注册 license [https://register.elastic.co/registration][https_register.elastic.co_registration]

替换其中的 type 为: platinum 白金版 和expiry\_date\_in\_millis 为 1565740799999 过期时间

{
      "license": {
        "uid": "e684c8fa-7281-40df-8764-8cf1fd00b406",
        "type": "basic",
        "issue_date_in_millis": 1534118400000,
        "expiry_date_in_millis": 1565740799999,
        "max_nodes": 100,
        "issued_to": "zhang kai (hc)",
        "issuer": "Web Form",
        "signature": "AAAAAwAAAA2TCeprkZ2aX7p+rKBcAAABmC9ZN0hjZDBGYnVyRXpCOW5Bb3FjZDAxOWpSbTVoMVZwUzRxVk1PSmkxaktJRVl5MUYvUWh3bHZVUTllbXNPbzBUemtnbWpBbmlWRmRZb25KNFlBR2x0TXc2K2p1Y1VtMG1UQU9TRGZVSGRwaEJGUjE3bXd3LzRqZ05iLzRteWFNekdxRGpIYlFwYkJiNUs0U1hTVlJKNVlXekMrSlVUdFIvV0FNeWdOYnlESDc3MWhlY3hSQmdKSjJ2ZTcvYlBFOHhPQlV3ZHdDQ0tHcG5uOElCaDJ4K1hob29xSG85N0kvTWV3THhlQk9NL01VMFRjNDZpZEVXeUtUMXIyMlIveFpJUkk2WUdveEZaME9XWitGUi9WNTZVQW1FMG1DenhZU0ZmeXlZakVEMjZFT2NvOWxpZGlqVmlHNC8rWVVUYzMwRGVySHpIdURzKzFiRDl4TmM1TUp2VTBOUlJZUlAyV0ZVL2kvVk10L0NsbXNFYVZwT3NSU082dFNNa2prQ0ZsclZ4NTltbU1CVE5lR09Bck93V2J1Y3c9PQAAAQANFdbpeTp95neGTKTT5R3ovic2yqJJaBHOGSIo3YvVDBCpXEU9C6Er1n5FX64GgpMZKivb/HCHvXY/c4ucArTwKi6pKVamjso1O7KGl0PioQCtUXZN4Dz22NT/2Rleql7zuRHcgU2LA6KOPIjcsmpkHZD3eGz0lQowwOrOPyurZFdAqi43423xtPgojfo3KfLo2Hhhl0aVzSyQ9n/pFHnAFucC9igAJaXm/bG9zIdFejKSAfJVckjAbr55rhmAtP4NwFWAB5xDQUdABKEQUeqZnYN3y7cBekvHwswznWOHawZkDz1rbMhbbvGAeNK2WUqyq/G+RO0MFmCCNSymTZxu",
        "start_date_in_millis": 1534118400000
      }
    }

更新 license

curl -XPUT -u elastic 'http://127.0.0.1:9200/_xpack/license?acknowledge=true' -H "Content-Type: application/json" -d @license.json

报错

{"error":{"root_cause":[{"type":"illegal_state_exception","reason":"Cannot install a [PLATINUM] license unless TLS is configured or security is disabled"}],"type":"illegal_state_exception","reason":"Cannot install a [PLATINUM] license unless TLS is configured or security is disabled"},"status":500}

在 elasticsearch.yml 文件中配置

xpack.security.transport.ssl.enabled: true

查看是否为白金会员  
　　  
curl -XGET -u elastic:123456 localhost:9200/\_license

也可以在 kibana 的 Management→ElasticSearch→License Management 看到过期时间

#### 7、集群配置 ####

集群ip : 192.168.99.101,192.168.99.102,192.168.99.103,192.168.99.104,192.168.99.105

拷贝 elasticsearch-6.3.2.tar.gz 到所有服务器，解压，替换破解之后的 x-pack-core-6.3.2.jar

elasticsearch.yml配置文件如下

cluster.name: es
    node.name: node-1
    node.master: true
    node.data: true
    node.max_local_storage_nodes: 256  #replace 256 with another num that greater than 1 
    path.data: /usr/local/elasticsearch-6.3.2/data
    path.logs: /usr/local/elasticsearch-6.3.2/logs
    network.host: 192.168.99.101
    http.port: 9200
    transport.tcp.port: 9300
    discovery.zen.ping.unicast.hosts: ["192.168.99.101","192.168.99.102","192.168.99.103","192.168.99.104","192.168.99.105"]
    discovery.zen.minimum_master_nodes: 2
    discovery.zen.ping_timeout: 6s

不同节点修改对应的 node.name 和 network.host 属性

集群启动，x-pack服务开启

更改 elasticsearch.yml 配置文件

xpack.security.enabled: true
    xpack.security.transport.ssl.enabled: true
    xpack.security.transport.ssl.verification_mode: certificate
    xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
    xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

[TLS/SSL][TLS_SSL]

生成认证文件 elastic-certificates.p12

bin/elasticsearch-certutil ca
    bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12

添加刚才输入的密码到 lasticsearch keystore ，会在 config 文件夹下生成 elasticseaerch.keystore 文件

bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password
    bin/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password

复制 elastic-certificates.p12 到 config 文件夹下

真正起作用的是 config 文件夹下的 elasticseaerch.keystore 文件

拷贝 elastic-certificates.p12 和 elasticseaerch.keystore 到所有节点机器

启动集群，更新 license

查看集群健康状态

http://192.168.99.101:9200/_cat/health?v

### 二、kibana ###

#### 1、下载官网kibana-6.3.2-linux-x86\_64.tar.gz文件 ####

#下载
    wget https://artifacts.elastic.co/downloads/kibana/kibana-6.3.2-linux-x86_64.tar.gz
    #解压
    tar xzvf kibana-6.3.2-linux-x86_64.tar.gz

#### 2、修改config/kibana.yml配置文件 ####

$ grep -Ev "^$|^[#;]" kibana.yml 
    #端口
    server.port: 5601
    #服务器IP
    server.host: "192.168.1.210"
    #elasticsearch服务器
    elasticsearch.url: "http://192.168.1.210:9200"
    elasticsearch.username: "kibana"
    elasticsearch.password: "123456"

注意：默认server.host=localhost，这样在自己本机启动可以，如果是想让网络上其他人使用，就需要改成本机IP地址。否则，访问时会提示：

dial tcp 192.168.1.210:5601: connectex: No connection could be made because the target machine actively refused it.

#### 3、启动 ####

改变kibana文件夹所有者到当前用户

sudo chown -R noroot:noroot /usr/local/kibana-6.3.2

启动

nohup ./kibana >/dev/null 2>&1 &

#### 4、停止 ####

查看端口占用命令

#查看运行的后台进程
    jobs -l  或者  ps -ef | grep node 或者 lsof -i :5601
    #杀掉进程
    kill -9 xxxx

#### 5、浏览器访问默认kibana端口5601 ####

使用 kibana/123456 可以进行登录访问

使用 elastic/123456 在菜单 Management→Security→Users、Roles可进行用户管理

#### 6、kibana 登陆用户管理 ####

创建角色

Management → Security → Roles → Create role

填写

name：app
    Run As Privileges ： 选择用户
    Index Privileges.Indices : *
    Index Privileges.Privileges : read,read_cross_cluster
    Index Privileges.Granted Fields : *

创建用户

Management → Security → Users → Create user

Username :
    Password :
    Password Again, Please :
    Full Name :
    Email :
    Roles :

之后该用户可以进行登陆，并可以查询 Discover 里的日志

### 三、filebeat ###

#### 1、下载 ####

#下载
    wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.3.2-linux-x86_64.tar.gz
    #解压
    tar xzvf filebeat-6.3.2-linux-x86_64.tar.gz

#### 2、修改配置文件 ####

可以配置多个配置文件，在启动时选择

直接输出日志到 elasticsearch

filebeat.yml

filebeat.inputs:
    - type: log
      enabled: true
      paths:
      #log文件路径
        - /opt/logs/app/api/*.log
      #多行日志合并
      multiline.pattern: ^\[
      multiline.negate: true
      multiline.match: after
      multiline.max_lines: 500
      multiline.timeout: 5s 
    filebeat.config.modules:
      path: ${path.config}/modules.d/*.yml
      reload.enabled: false
    #index修改后下面两个配置需要添加
    setup.template.name: "appname"
    setup.template.pattern: "appname-*"
    
    setup.template.settings:
      index.number_of_shards: 3
    setup.kibana:
      host: "192.168.99.100:5601"
      username: "kibana"  
      password: "123456"
    output.elasticsearch:
      hosts: ["192.168.99.11:9200"]
      index: appname
      protocol: "http"
      username: "elastic"
      password: "123456"

#### 3、启动 ####

nohup ./filebeat -e -c filebeat.yml >/dev/null 2>&1 &

#### 4、停止 filebeat ####

ps -ef|grep filebeat

### 四、logstash ###

#### 1、下载 ####

#下载
    wget https://artifacts.elastic.co/downloads/logstash/logstash-6.3.2.tar.gz
    #解压
    tar -xvf logstash-6.3.2

#### 2、配置logstash.conf ####

直接输出日志到 elasticsearch

可以配置多个logstash.conf 启动指定不同文件，端口和数据路径

input{
        file{
            path =>"/opt/logs/appname/api/main.log"
            start_position=>"beginning"
        }
        stdin {
            codec => multiline {
                pattern => "^\["
                negate => true
                what => "previous"
            }
        }
    }
    
    filter{
        grok{
            match=>{
               "message"=>"%{DATA:clientIp} - - \[%{HTTPDATE:accessTime}\] \"%{DATA:method} %{DATA:requestPath} %{DATA:httpversion}\" %{DATA:retcode} %{DATA:size} \"%{DATA:fromHtml}\" \"%{DATA:useragent}\""
            }
            remove_field=>"message"
       }
       date{
            match=>["accessTime","dd/MMM/yyyy:HH:mm:ss Z"]
       }
    
       if ([message] =~ "^debug") {    
          drop {}                   
       }
    }
    
    output {
        elasticsearch {
            hosts => "192.168.99.100"
            user => "elastic"
            password => "123456"
            index => "appname"
        }
        stdout { 
                codec => rubydebug 
        }
    }

#### 3、启动 ####

nohup ./logstash -f ../config/logstash.conf >/dev/null 2>&1 &

#### 4、停止 ####

ps -ef|grep logstash
    kill -9 pid

#### 5、logstash启动特别慢 ####

测试是否需要安装haveged

$ cat /proc/sys/kernel/random/entropy_avail
    186

如果值比较低(<1000),建议安装 haveged. 否则加密程序会等待系统有足够的熵。

安装haveged

加入epel源(实际就是添加yum源配置)

# yum install epel-release -y

安装haveged

#yum install haveged -y

启动haveged并开机启动,并查看haveged状态(对centos7不熟悉的童鞋需要补补哦\_)

# systemctl start haveged                    #启动
    # systemctl enable haveged                   #开机启动
    # systemctl status haveged                   #查看启动状态
    Created symlink from /etc/systemd/system/multi-user.target.wants/haveged.service to /usr/lib/systemd/system/haveged.service.
    # systemctl list-unit-files | grep haveged    #查看开机启动状态
    haveged.service                             enabled

装了还是很慢

### 五、安装中问题 ###

#### 1、elasticsearch start more than one nodes on the same data folder ####

aused by: java.lang.IllegalStateException: failed to obtain node locks, tried \[\[/elasticsearch-5.4.0/data/elasticsearch\]\] with lock id \[0\]; maybe these locations are not writable or multiple nodes were started without increasing \[node.max\_local\_storage\_nodes\] (was \[1\])?

当我在开启多个elasticsearch 实例时，遇到上述异常，解决方法是：在 config/elasticsearch.yml文件中新增一个配置变量

node.max_local_storage_nodes: 256  #replace 256 with another num that greater than 1

#### 2、javax.net.ssl.SSLException: Received fatal alert ####

没有配置 ssl 认证，参考集群 ssl 配置

[2018-08-16T20:03:05,110][WARN ][o.e.x.s.t.n.SecurityNetty4ServerTransport] [node-1] exception caught on transport layer [NettyTcpChannel{localAddress=0.0.0.0/0.0.0.0:47968, remoteAddress=/192.168.99.101:9300}], closing connection
    io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: Received fatal alert: handshake_failure
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152]
    Caused by: javax.net.ssl.SSLException: Received fatal alert: handshake_failure

#### 3、keystore password was incorrect ####

如果没有配置 config/elasticsearch.keystore , 集群 ssl 开启情况下会报如下错误，将第一台自己上生成的认证文件拷贝一份即可

[2018-08-17T16:21:50,332][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-5] uncaught exception in thread [main]
    org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:140) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.3.2.jar:6.3.2]
        at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.3.2.jar:6.3.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) ~[elasticsearch-6.3.2.jar:6.3.2]
    Caused by: java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]
        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:701) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:643) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:557) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:162) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.node.Node.<init>(Node.java:311) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.3.2.jar:6.3.2]
        ... 6 more
    Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_152]
        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:692) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:643) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:557) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:162) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.node.Node.<init>(Node.java:311) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.3.2.jar:6.3.2]
        ... 6 more
    Caused by: org.elasticsearch.ElasticsearchException: failed to initialize a TrustManagerFactory
        at org.elasticsearch.xpack.core.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:60) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:407) ~[?:?]
        at java.util.HashMap.computeIfAbsent(HashMap.java:1127) ~[?:1.8.0_152]
        at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:459) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:79) ~[?:?]
        at org.elasticsearch.xpack.core.XPackPlugin.<init>(XPackPlugin.java:134) ~[?:?]
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_152]
        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:692) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:643) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:557) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:162) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.node.Node.<init>(Node.java:311) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.3.2.jar:6.3.2]
        ... 6 more
    Caused by: java.io.IOException: keystore password was incorrect
        at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2059) ~[?:?]
        at java.security.KeyStore.load(KeyStore.java:1445) ~[?:1.8.0_152]
        at org.elasticsearch.xpack.core.ssl.CertUtils.readKeyStore(CertUtils.java:265) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.CertUtils.trustManager(CertUtils.java:256) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:58) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:407) ~[?:?]
        at java.util.HashMap.computeIfAbsent(HashMap.java:1127) ~[?:1.8.0_152]
        at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:459) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:79) ~[?:?]
        at org.elasticsearch.xpack.core.XPackPlugin.<init>(XPackPlugin.java:134) ~[?:?]
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_152]
        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:692) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:643) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:557) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:162) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.node.Node.<init>(Node.java:311) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.3.2.jar:6.3.2]
        ... 6 more
    Caused by: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded
        at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2059) ~[?:?]
        at java.security.KeyStore.load(KeyStore.java:1445) ~[?:1.8.0_152]
        at org.elasticsearch.xpack.core.ssl.CertUtils.readKeyStore(CertUtils.java:265) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.CertUtils.trustManager(CertUtils.java:256) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:58) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:407) ~[?:?]
        at java.util.HashMap.computeIfAbsent(HashMap.java:1127) ~[?:1.8.0_152]
        at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:459) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:79) ~[?:?]
        at org.elasticsearch.xpack.core.XPackPlugin.<init>(XPackPlugin.java:134) ~[?:?]
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_152]
        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:692) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:643) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:557) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:162) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.node.Node.<init>(Node.java:311) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.3.2.jar:6.3.2]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.3.2.jar:6.3.2]
        ... 6 more

[elasticsearch6.3.1 _x-pack]: https://blog.csdn.net/zou79189747/article/details/81164576
[Kibana6.3.2_Elasticsearch6.3.2]: https://blog.csdn.net/fishinhouse/article/details/81323959
[x-pack6.3]: https://www.cnblogs.com/jcici/p/9400369.html
[filebeat 5.2.2]: https://blog.csdn.net/weixin_39077573/article/details/73467712
[logstash _-haveged]: https://www.jianshu.com/p/8ffc521fc3ed
[elasticsearch start more than one nodes on the same data folder]: https://blog.csdn.net/the_conquer_zzy/article/details/72818896
[elasticsearch6.3.2]: https://blog.csdn.net/lk7688535/article/details/81452147
[http_localhost_9200]: http://localhost:9200
[x-pack-core-6.3.2.jar]: https://download.csdn.net/download/zk673820543/10605508
[https_register.elastic.co_registration]: https://register.elastic.co/registration
[TLS_SSL]: https://www.elastic.co/guide/en/elasticsearch/reference/6.3/configuring-tls.html#node-certificates