实战：ELK环境部署并采集springboot项目日志

秒速五厘米 2024-03-17 21:55 66阅读 0赞

#### 文章目录 ####

*   *  前言
     *  技术积累
     *   *  ELK组成及功能
         *  框架搭建基础
     *  EIK环境搭建
     *   *  elasticsearch配置相关
         *  kibana配置相关
         *  logstash配置相关
         *  elk目录下增加docker-compose文件
         *  查看elk目录文件树
         *  编排elk
     *  springboot集成logstash
     *   *  pom.xml
         *  logback-spring.xml
         *  启动项目logstash采集日志
     *  写在最后

### 前言 ###

相信作为一个资深的搬砖人，在处理问题的时候免不了查看应用系统日志，且可以根据这个日志日志精准、快速的解决实际的问题。一般情况下我们的系统日志都放置在包的运行目录下面，非常不便于查看和分类。那么。今天我们就引入ELK的日志处理架构来解决它。  
![在这里插入图片描述][b36face5559e4ce7a12feb97a5330a17.png]

### 技术积累 ###

#### ELK组成及功能 ####

> ELK是logstash、elasticsearch、kibana的简称，和其名字一样，elk架构就是将这三个中间件进行整合搭建一个日志系统。

> 首先我们应用系统集成logstash客户端并采集日志上传到logstash服务端进行过滤、转换，转换后的日志写入elasticsearch，es的强大功能提供数据存储，分词和倒排索引提升查询效率；最后的kibana直接是渲染日志数据的分析和可视化平台。

#### 框架搭建基础 ####

> 为方便我们架构的搭建，我们用docker-compose进行容器化编排，只要保存elk三个组件同网络下它们就能够根据服务名进行通讯。

> 当然，对于向外暴露的接口我们仅仅需要暴露logstash的进行数据上传，es的进行数据外部查询即可。每个应用服务都必须有自己的logstash配置，在配置中提供输入、输出路径和过滤参数，对于的端口我们也需要向外暴露以便于数据的上传。

### EIK环境搭建 ###

elk目录下文件树：  
./  
├── docker-compose.yml  
├── elasticsearch  
│ ├── config  
│ │ └── elasticsearch.yml  
│ ├── data  
│ └── logs  
├── kabana  
│ └── config  
│ └── kabana.yml  
└── logstash  
├── config  
│ ├── logstash.yml  
│ └── small-tools  
│ └── demo.config  
└── data

#### elasticsearch配置相关 ####

mkdie elk  
\#增加es目录  
cd elk  
mkdir -p ./elasticsearch/logs ./elasticsearch/data ./elasticsearch/config  
chmod 777 ./elasticsearch/data  
\#./elasticsearch/config 下增加es配置文件  
cd elasticsearch/config  
vim elasticsearch.yml

cluster.name: "docker-cluster"
    network.host: 0.0.0.0
    http.port: 9200
    # 开启es跨域
    http.cors.enabled: true
    http.cors.allow-origin: "*"
    http.cors.allow-headers: Authorization
    # 开启安全控制
    xpack.security.enabled: true
    xpack.security.transport.ssl.enabled: true

#### kibana配置相关 ####

cd elk  
mkdir -p ./kibana/config  
\#./kibana/config 下增加kibana配置文件  
cd kibana/config  
vim kibana.yml

server.name: kibana
    server.host: "0.0.0.0"
    server.publicBaseUrl: "http://kibana:5601"
    elasticsearch.hosts: [ "http://elasticsearch:9200" ] 
    xpack.monitoring.ui.container.elasticsearch.enabled: true
    elasticsearch.username: "elastic"
    elasticsearch.password: "123456"
    i18n.locale: zh-CN

#### logstash配置相关 ####

cd elk  
mkdir -p ./logstash/data ./logstash/config ./logstash/config/small-tools  
chmod 777 ./logstash/data  
\#./logstash/config 下增加logstash配置文件  
cd logstash/config  
vim logstash.yml

http.host: "0.0.0.0"
    xpack.monitoring.enabled: true
    xpack.monitoring.elasticsearch.hosts: [ "http://elasticsearch:9200" ]
    xpack.monitoring.elasticsearch.username: "elastic"
    xpack.monitoring.elasticsearch.password: "123456"

\#./logstash/config/small-tools 下增加demo项目监控配置文件  
cd small-tools  
vim demo.config

input { #输入
    
        tcp {
            mode => "server"
            host => "0.0.0.0"   # 允许任意主机发送日志
            type => "demo"      # 设定type以区分每个输入源
            port => 9999
            codec => json_lines # 数据格式
        }
    
    }
    
    
    filter {
        mutate {
            # 导入之过滤字段
            remove_field => ["LOG_MAX_HISTORY_DAY", "LOG_HOME", "APP_NAME"]
            remove_field => ["@version", "_score", "port", "level_value", "tags", "_type", "host"]
        }
    }
    
    
    output { #输出-控制台
        stdout{
            codec => rubydebug
        }
    }
    
    
    output { #输出-es
    
        if [type] == "demo" {
            elasticsearch {
                action => "index"                       # 输出时创建映射
                hosts  => "http://elasticsearch:9200"   # ES地址和端口
                user => "elastic"                       # ES用户名
                password => "123456"                    # ES密码
                index  => "demo-%{+YYYY.MM.dd}"         # 指定索引名-按天
                codec  => "json"
            }
        }
    
    }

#### elk目录下增加docker-compose文件 ####

docker-compose.yml

version: '3.3'
    networks:
      elk:
        driver: bridge
    services:
      elasticsearch:
        image: registry.cn-hangzhou.aliyuncs.com/zhengqing/elasticsearch:7.14.1
        container_name: elk_elasticsearch
        restart: unless-stopped
        volumes:
          - "./elasticsearch/data:/usr/share/elasticsearch/data"
          - "./elasticsearch/logs:/usr/share/elasticsearch/logs"
          - "./elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml"
        environment:
          TZ: Asia/Shanghai
          LANG: en_US.UTF-8
          TAKE_FILE_OWNERSHIP: "true"  # 权限
          discovery.type: single-node
          ES_JAVA_OPTS: "-Xmx512m -Xms512m"
          ELASTIC_PASSWORD: "123456" # elastic账号密码
        ports:
          - "9200:9200"
          - "9300:9300"
        networks:
          - elk
    
      kibana:
        image: registry.cn-hangzhou.aliyuncs.com/zhengqing/kibana:7.14.1
        container_name: elk_kibana
        restart: unless-stopped
        volumes:
          - "./kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml"
        ports:
          - "5601:5601"
        depends_on:
          - elasticsearch
        links:
          - elasticsearch
        networks:
          - elk
    
      logstash:
        image: registry.cn-hangzhou.aliyuncs.com/zhengqing/logstash:7.14.1
        container_name: elk_logstash
        restart: unless-stopped
        environment:
          LS_JAVA_OPTS: "-Xmx512m -Xms512m"
        volumes:
          - "./logstash/data:/usr/share/logstash/data"
          - "./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml"
          - "./logstash/config/small-tools:/usr/share/logstash/config/small-tools"
        command: logstash -f /usr/share/logstash/config/small-tools
        ports:
          - "9600:9600"
          - "9999:9999"
        depends_on:
          - elasticsearch
        networks:
          - elk

#### 查看elk目录文件树 ####

yum -y install tree  
\#查看当前目录下4层  
tree -L 4  
\#显示所有文件、文件夹  
tree -a  
\#显示大小  
tree -s

\[root@devops-01 elk\]\# pwd  
/home/test/demo/elk  
\[root@devops-01 elk\]\# tree ./  
./  
├── docker-compose.yml  
├── elasticsearch  
│ ├── config  
│ │ └── elasticsearch.yml  
│ ├── data  
│ └── logs  
├── kabana  
│ └── config  
│ └── kabana.yml  
└── logstash  
├── config  
│ ├── logstash.yml  
│ └── small-tools  
│ └── demo.config  
└── data

10 directories, 5 files

#### 编排elk ####

docker-compose up -d

编排成功查看容器是否成功启动

[root@devops-01 elk]# docker ps | grep elk

edcf6c1cecb3 registry.cn-hangzhou.aliyuncs.com/zhengqing/kibana:7.14.1 “/bin/tini – /usr/l…” 6 minutes ago Up 10 seconds 0.0.0.0:5601->5601/tcp, :::5601->5601/tcp elk\_kibana  
7c24b65d2a27 registry.cn-hangzhou.aliyuncs.com/zhengqing/logstash:7.14.1 “/usr/local/bin/dock…” 6 minutes ago Up 13 seconds 5044/tcp, 9600/tcp elk\_logstash  
b4be2f1c0a28 registry.cn-hangzhou.aliyuncs.com/zhengqing/elasticsearch:7.14.1 “/bin/tini – /usr/l…” 6 minutes ago Up 6 minutes 0.0.0.0:9800->9200/tcp, :::9800->9200/tcp, 0.0.0.0:9900->9300/tcp, :::9900->9300/tcp elk\_elasticsearch

编排成功访问kibana页面  
http://10.10.22.174:5601/app/home\#/  
![在这里插入图片描述][1e82ccb0a1d0486f92ad654e50b51fde.png]

### springboot集成logstash ###

#### pom.xml ####

<dependency>
        <groupId>net.logstash.logback</groupId>
        <artifactId>logstash-logback-encoder</artifactId>
        <version>6.6</version>
    </dependency>

#### logback-spring.xml ####

#### 启动项目logstash采集日志 ####

kibana配置查看日志  
http://10.10.22.174:5601/app/home\#/ 输入ES用户名和密码进入kibana控制台  
点击管理按钮进入管理界面  
![在这里插入图片描述][8a8937502c644fe88fe39df3217ea5d2.png]

点击索引模式进入–>创建索引模式  
![在这里插入图片描述][c00029fbbc914febab127e2a128ac1cf.png]

输入配置日志表达式–>点击下一步  
![在这里插入图片描述][8d472a15c21c411cbb7089a8bb1d1172.png]

选择timestamp -->创建索引模式  
![在这里插入图片描述][62c02dd4dcf84eae9b84718ee8c74cc1.png]

创建完成如下所示代表成功  
![在这里插入图片描述][1529099cdd434dd380cc22835e46436c.png]

查看日志  
菜单点击–>discover  
![在这里插入图片描述][bf80a46e9f914f918c6a21078b1e8f1e.png]  
![在这里插入图片描述][fe7348fd3ed144cfbbccb2f235bc0d9e.png]

### 写在最后 ###

ELK环境部署并采集springboot项目日志还是比较简单，我们只需要用docker容器化技术搭建起elk框架，然后在自己的项目中进行数据采集上传即可。当然对于elk组成元素的logstash、elasticsearch、kibana还是需要一些基础的了解，方便在实战的时候进行操作。

[b36face5559e4ce7a12feb97a5330a17.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/14/3a679783a35f42a7b33b22767d609b13.png
[1e82ccb0a1d0486f92ad654e50b51fde.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/14/7f9d9949135645579cfa0e7588eb1b19.png
[8a8937502c644fe88fe39df3217ea5d2.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/14/2be2ca6560b54cebb3fe1e9d4e9721af.png
[c00029fbbc914febab127e2a128ac1cf.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/14/64d95c4b9338413d87994fb924c733f1.png
[8d472a15c21c411cbb7089a8bb1d1172.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/14/ef9f9efc4f5b4976afbefc0cb1cccf9e.png
[62c02dd4dcf84eae9b84718ee8c74cc1.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/14/a7e7a483dd4748f6b937538d686baf62.png
[1529099cdd434dd380cc22835e46436c.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/14/faf066a165484ad9bfb7d4c625f09491.png
[bf80a46e9f914f918c6a21078b1e8f1e.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/14/7b49acbe61174c3ebab4abf4008b693d.png
[fe7348fd3ed144cfbbccb2f235bc0d9e.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/14/b40294a70b7e4d1d8c562121911f9d71.png