【Java】SpringBoot集成ELK(elasticsearch+logstash+kibana)及Elasticsearch-head插件

青旅半醒 2022-11-25 10:07 534阅读 0赞

### SpringBoot集成ELK及ES-Head插件 ###

*  前言
 *  开发环境
 *  Windows系统
 *   *  下载安装
     *   *  基础SpringBoot项目
         *  elasticsearch
         *  elasticsearch-head
         *  logstash
         *  kibana
         *  注意事项
     *  配置
     *   *  SpringBoot项目
         *   *  pom.xml
             *  application.yml
             *  需创建内容
             *   *  ElasticConfig
                 *  logback-spring.xml
                 *  \[可选\]
                 *   *  使用esclient手动保存数据
                     *  实体
         *  elasticsearch
         *  elasticsearch-head
         *  logstash
         *  kibana
         *  注意事项
     *  启动
     *   *  elasticsearch
         *  elasticsearch-head
         *  logstash
         *  kibana
         *  注意事项
         *  \[可选\]
         *   *  使用nssm将上面用控制台启动的安装为windows服务
     *  使用效果
 *  Linux系统
 *   *  下载安装
     *  配置
     *  启动
     *  注意事项
     *  其他
     *   *  重置密码

# 前言 #

> 本文横向展示Windows系统下SpringBoot集成ELK步骤及Linux系统Docker搭建ELK。，统一下载安装统一配置，必要先后顺序统一说明，统一介绍踩坑注意事项。

# 开发环境 #

**操作系统：** Windows 10 64位  
**IDE：** IntelliJ IDEA  
**展示ELK版本：** 7.8.1

# Windows系统 #

## 下载安装 ##

### 基础SpringBoot项目 ###

从[https://start.spring.io/][https_start.spring.io] 生成并下载基础项目。

### elasticsearch ###

从[https://www.elastic.co/cn/downloads/elasticsearch][https_www.elastic.co_cn_downloads_elasticsearch]下载。

### elasticsearch-head ###

从[https://github.com/mobz/elasticsearch-head][https_github.com_mobz_elasticsearch-head]下载。

### logstash ###

从[https://www.elastic.co/cn/downloads/logstash][https_www.elastic.co_cn_downloads_logstash]下载。

### kibana ###

从[https://www.elastic.co/cn/downloads/kibana][https_www.elastic.co_cn_downloads_kibana]下载。

### 注意事项 ###

E、L、K的版本需一致，注意不要下载成不同版本的，版本号完全一样。注意32/64位。JDK请使用11及以上。

![在这里插入图片描述][20200807191627707.png]

## 配置 ##

### SpringBoot项目 ###

#### pom.xml ####

<dependency>
    			<groupId>org.springframework.boot</groupId>
    			<artifactId>spring-boot-starter-data-elasticsearch</artifactId>
    			<version>2.3.2.RELEASE</version>
    		</dependency>
    		<dependency>
    			<groupId>org.elasticsearch.client</groupId>
    			<artifactId>elasticsearch-rest-high-level-client</artifactId>
    			<version>7.4.0</version>
    			<exclusions>
    				<exclusion>
    					<groupId>org.elasticsearch</groupId>
    					<artifactId>elasticsearch</artifactId>
    				</exclusion>
    				<exclusion>
    					<groupId>org.elasticsearch.client</groupId>
    					<artifactId>elasticsearch-rest-client</artifactId>
    				</exclusion>
    			</exclusions>
    		</dependency>
    		<dependency>
    			<groupId>org.elasticsearch.client</groupId>
    			<artifactId>elasticsearch-rest-client</artifactId>
    			<version>7.4.0</version>
    		</dependency>
    		<dependency>
    			<groupId>org.elasticsearch</groupId>
    			<artifactId>elasticsearch</artifactId>
    			<version>7.4.0</version>
    		</dependency>
    		<dependency>
    			<groupId>net.logstash.logback</groupId>
    			<artifactId>logstash-logback-encoder</artifactId>
    			<version>5.3</version>
    		</dependency>

#### application.yml ####

spring:
      servlet:
        multipart:
          enabled: false
      elasticsearch:
        rest:
          uris: http://127.0.0.1:9200  
    es:
      host: 127.0.0.1
      port: 9200
      scheme: http
    server:
      port: 22745

#### 需创建内容 ####

##### ElasticConfig #####

@Configuration
    public class ElasticConfig { 
    
        @Value("${es.host}")
        public String host;
        @Value("${es.port}")
        public int port;
        @Value("${es.scheme}")
        public String scheme;
    
        @Bean
        public RestHighLevelClient elasticsearchClient() { 
            ClientConfiguration.ClientConfigurationBuilderWithRequiredEndpoint builder = ClientConfiguration.builder();
            final ClientConfiguration clientConfiguration = ClientConfiguration.builder()
                    .connectedTo(host + ":" + port)
                    .build();
    
            return RestClients.create(clientConfiguration).rest();
        }
    }

##### logback-spring.xml #####

![在][20200807195019936.png]

<?xml version="1.0" encoding="UTF-8"?>
    <configuration>
        <include resource="org/springframework/boot/logging/logback/defaults.xml"/>
    
        <springProperty scope="context" name="springAppName" source="spring.application.name"/>
        <springProperty scope="context" name="serverPort" source="server.port"/>
        
        <property name="LOG_FILE" value="${BUILD_FOLDER:-build}/${springAppName}"/>
      
        <appender name="logstash" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
            <encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
                <providers>
                    <timestamp>
                        <timeZone>UTC</timeZone>
                    </timestamp>
                    <pattern>
                        <pattern>
                            {
                            "severity": "%level",
                            "service": "${springAppName:-}",
                            "port": "${serverPort:-}",
                            "trace": "%X{X-B3-TraceId:-}",
                            "span": "%X{X-B3-SpanId:-}",
                            "exportable": "%X{X-Span-Export:-}",
                            "pid": "${PID:-}",
                            "thread": "%thread",
                            "class": "%logger{40}",
                            "rest": "%message"
                            }
                        </pattern>
                    </pattern>
                </providers>
            </encoder>
        </appender>
        <root level="INFO">
            <appender-ref ref="console"/>
            <appender-ref ref="logstash"/>
        </root>
    </configuration>

##### \[可选\] #####

###### 使用esclient手动保存数据 ######

@Autowired
        private PageService pageService;
    	@GetMapping("/Set/{param}")
        public String Set(@PathVariable String param) { 
            String id = UUID.randomUUID().toString(); //ID数据结构与实体对应
            SimpleDateFormat df = new SimpleDateFormat("yyyyMMdd");
            IndexCoordinates indexOperations = IndexCoordinates.of("springelk-logs-"+df.format(new Date()));//必须小写
            Data d = new Data(); //实体
            d.setId(id);
            d.setObject(param);
            IndexQuery indexQuery = new IndexQueryBuilder()
                    .withId(id.toString())
                    .withObject(d)//此处接受参数Object但必须是实体类对象。直接用String会报错 如下图
                    .build();
            String documentId = elasticsearchTemplate.index(indexQuery, indexOperations);
            return documentId;
        }

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTQyOTQzMjU_size_16_color_FFFFFF_t_70]

###### 实体 ######

@Document(indexName = "log")
    public class Data { 
        @Id
        private String id;
    
        private Object object;
        public String getId() { 
            return id;
        }
    
        public void setId(String id) { 
            this.id = id;
        }
    
        public Object getObject() { 
            return object;
        }
    
        public void setObject(Object object) { 
            this.object = object;
        }
    }

### elasticsearch ###

修改\\config\\elasticsearch.yml配置  
添加

network.host:127.0.0.1      
    http.port: 9200       
    
    http.cors.enabled: true
    http.cors.allow-origin: "*"

### elasticsearch-head ###

修改\\Gruntfile.js配置

connect: { 
    			server: { 
    				options: { 
    					hostname:'*',
    					port: 9100,
    					base: '.',
    					keepalive: true
    				}
    			}
    		}

### logstash ###

在\\config下新建文件logstash.conf  
写入

input{ 
    	stdin{ 
    		path => "D:\DevelopSpace\project\logger.log"  #输入的日志文件 数据来源
    		type => "text_log" 
    		start_position => "beginning" 
    	}
    }
    
    output{ 
    	elasticsearch{   #转发到elasticsearch
    		hosts=>["127.0.0.1:9200"]  #配置和之前配elasticsearch的一样
    		index => "log-%{+YYYY.MM.dd}"
    	}
    	stdout{ 
    		codec => rubydebug
    	}
    }

### kibana ###

修改\\config\\kibana.yml配置

server.port: 5601   
    server.host: 127.0.0.1  
    elasticsearch.hosts: 127.0.0.1:9200

### 注意事项 ###

kibana配置不能用url ：elasticsearch.url: 127.0.0.1:9200 \#注意url在该版本已弃用

## 启动 ##

### elasticsearch ###

运行\\bin\\elasticsearch.bat启动ES  
浏览器访问：127.0.0.1:9200  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTQyOTQzMjU_size_16_color_FFFFFF_t_70 1]  
成功启动

### elasticsearch-head ###

在目录下打开命令行使用**npm install**安装包，**npm run start**启动  
![在这里插入图片描述][20200807193706220.png]

浏览器访问：127.0.0.1:9100  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTQyOTQzMjU_size_16_color_FFFFFF_t_70 2]

### logstash ###

运行\\bin\\logstash.bat启动logstash 使用logstash.conf配置  
命令行：logstash.bat -f …/config/logstash.conf

### kibana ###

运行\\bin\\kibana.bat启动kibana  
浏览器访问：127.0.0.1:5601  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTQyOTQzMjU_size_16_color_FFFFFF_t_70 3]

### 注意事项 ###

1.注意端口占用  
2.注意kibana32/64位  
3.若缺少npm及nodejs需下载安装  
4.启动顺序 E->L-> (ES-head)->K

### \[可选\] ###

#### 使用nssm将上面用控制台启动的安装为windows服务 ####

从[http://www.nssm.cc/download][http_www.nssm.cc_download]下载nssm  
使用命令行调用  
该工具可将各种用命令行启动的服务安装为Windows服务，方便使用，使用教程这里不赘述了。

## 使用效果 ##

Spring启动调用\[可选\] 函数 Set 传入参数qwe 记录Data实体 在kibana中查看  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTQyOTQzMjU_size_16_color_FFFFFF_t_70 4]

logstash转发 参数amyy 记录Person实体 在es-head中查看  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTQyOTQzMjU_size_16_color_FFFFFF_t_70 5]

# Linux系统 #

## 下载安装 ##

ELK在Docker上有成熟的镜像，从Git上下载

git clone https://github.com/deviantony/docker-elk.git

## 配置 ##

项目拉下来后可以直接使用，这里介绍一下几个配置要点  
docker-compose.yml

ELASTIC\_PASSWORD ES密码

environment:
          ES_JAVA_OPTS: "-Xmx256m -Xms256m"
          ELASTIC_PASSWORD: changeme  【ES密码】
          # Use single node discovery in order to disable production mode and avoid bootstrap checks
          # see https://www.elastic.co/guide/en/elasticsearch/reference/current/bootstrap-checks.html
          discovery.type: single-node

logstash.yml

## X-Pack security credentials
    #
    xpack.monitoring.enabled: true
    xpack.monitoring.elasticsearch.username: elastic
    xpack.monitoring.elasticsearch.password: changeme   【ES密码】

logstash.conf

output { 
    	elasticsearch { 
    		hosts => "elasticsearch:9200"
    		user => "elastic"
    		password => "changeme"【ES密码】
    	}
    }

kibana.yml

## X-Pack security credentials
    #
    elasticsearch.username: elastic
    elasticsearch.password: changeme 【ES密码】

## 启动 ##

在目录下使用命令行**docker-compose**启动，首次下载需要花费较长时间，之后就很快了。

## 注意事项 ##

如果构建失败，用**docker-compose --build**再试一次，一般只有首次启动时会出现该状况。

## 其他 ##

### 重置密码 ###

该项目默认设置的密码为changeme，是要修改的。  
在目录下使用命令行**docker-compose exec -T elasticsearch bin/elasticsearch-setup-passwords auto --batch**重置密码，会得到如下随机生成的密码。

Changed password for user apm_system
    PASSWORD apm_system = SVqOyQi7Jo8xueQPltJu
    
    Changed password for user kibana_system
    PASSWORD kibana_system = WMdXWC5e3f5jUI5Kazra
    
    Changed password for user kibana
    PASSWORD kibana = WMdXWC5e3f5jUI5Kazra
    
    Changed password for user logstash_system
    PASSWORD logstash_system = YeVCzX2R3Vbe1HgYxz54
    
    Changed password for user beats_system
    PASSWORD beats_system = TQqb8QqCg7rST9vxs4gB
    
    Changed password for user remote_monitoring_user
    PASSWORD remote_monitoring_user = LJlsrTz95Y8DgtSYq7CR
    
    Changed password for user elastic
    PASSWORD elastic = qnPqsPR9jamTr6voSa7D

[https_start.spring.io]: https://start.spring.io/
[https_www.elastic.co_cn_downloads_elasticsearch]: https://www.elastic.co/cn/downloads/elasticsearch
[https_github.com_mobz_elasticsearch-head]: https://github.com/mobz/elasticsearch-head
[https_www.elastic.co_cn_downloads_logstash]: https://www.elastic.co/cn/downloads/logstash
[https_www.elastic.co_cn_downloads_kibana]: https://www.elastic.co/cn/downloads/kibana
[20200807191627707.png]: /images/20221124/abda15fb546d4777939a752bc9e5903a.png
[20200807195019936.png]: /images/20221124/f16bb9ec618b4362b4dc6e12ebc86d28.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTQyOTQzMjU_size_16_color_FFFFFF_t_70]: /images/20221124/f00eaf6d3c6a4e099fff897b38d14106.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTQyOTQzMjU_size_16_color_FFFFFF_t_70 1]: /images/20221124/7d9fdbe1a4b742df8d00023c2d74439b.png
[20200807193706220.png]: /images/20221124/853a5988569743a38db1d8424fdf1367.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTQyOTQzMjU_size_16_color_FFFFFF_t_70 2]: /images/20221124/c52ee776444b4cd091b1b524f9aa2d7a.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTQyOTQzMjU_size_16_color_FFFFFF_t_70 3]: /images/20221124/eee7f98b98ae423e9259ba6e42a20a82.png
[http_www.nssm.cc_download]: http://www.nssm.cc/download
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTQyOTQzMjU_size_16_color_FFFFFF_t_70 4]: /images/20221124/217847ab34bd4b91b4d63f9fdd34f228.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTQyOTQzMjU_size_16_color_FFFFFF_t_70 5]: /images/20221124/c8d620968b254079bad13cf74f5b96ec.png