elasticsearch filebeat采集日志输出到elasticsearch

痛定思痛。 2023-06-29 08:59 67阅读 0赞

--------------------

### elasticsearch filebeat采集日志输出到elasticsearch ###

**使用filebeat采集日志文件，将日志输出到logstash，logstash输出到elasticsearch**

## ##

### \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* ###

**日志文件：my.log**

2020-01-12 10:06:14.549  INFO 9356 --- [main] com.example.demo.DemoApplication         : Starting DemoApplication on LAPTOP-D73GD8TE with PID 9356 (started by 28401 in E:\java\IdeaProjects\springboot hello-world)
    2020-01-12 10:06:14.555  INFO 9356 --- [main] com.example.demo.DemoApplication         : No active profile set, falling back to default profiles: default
    2020-01-12 10:06:16.775  INFO 9356 --- [main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8080 (http)
    2020-01-12 10:06:16.787  INFO 9356 --- [main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
    2020-01-12 10:06:16.787  INFO 9356 --- [main] org.apache.catalina.core.StandardEngine  : Starting Servlet engine: [Apache Tomcat/9.0.29]
    2020-01-12 10:06:16.967  INFO 9356 --- [main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
    2020-01-12 10:06:16.967  INFO 9356 --- [main] o.s.web.context.ContextLoader            : Root WebApplicationContext: initialization completed in 2314 ms
    2020-01-12 10:06:17.422  INFO 9356 --- [main] o.s.s.concurrent.ThreadPoolTaskExecutor  : Initializing ExecutorService 'applicationTaskExecutor'
    2020-01-12 10:06:17.710  INFO 9356 --- [main] o.s.b.a.e.web.EndpointLinksResolver      : Exposing 2 endpoint(s) beneath base path '/actuator'
    2020-01-12 10:06:17.781  INFO 9356 --- [main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8080 (http) with context path ''
    2020-01-12 10:06:17.787  INFO 9356 --- [main] com.example.demo.DemoApplication         : Started DemoApplication in 4.119 seconds (JVM running for 6.441)
    2020-01-12 10:06:19.332  INFO 9356 --- [RMI TCP Connection(5)-192.168.57.1] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring DispatcherServlet 'dispatcherServlet'
    2020-01-12 10:06:19.333  INFO 9356 --- [RMI TCP Connection(5)-192.168.57.1] o.s.web.servlet.DispatcherServlet        : Initializing Servlet 'dispatcherServlet'
    2020-01-12 10:06:19.342  INFO 9356 --- [RMI TCP Connection(5)-192.168.57.1] o.s.web.servlet.DispatcherServlet        : Completed initialization in 8 ms
    2020-01-12 10:09:21.353  INFO 3104 --- [main] com.example.demo.DemoApplicationTests    : Starting DemoApplicationTests on LAPTOP-D73GD8TE with PID 3104 (started by 28401 in E:\java\IdeaProjects\springboot hello-world)
    2020-01-12 10:09:21.355  INFO 3104 --- [main] com.example.demo.DemoApplicationTests    : No active profile set, falling back to default profiles: default
    2020-01-12 10:09:23.758  INFO 3104 --- [main] o.s.s.concurrent.ThreadPoolTaskExecutor  : Initializing ExecutorService 'applicationTaskExecutor'
    2020-01-12 10:09:24.435  INFO 3104 --- [main] o.s.b.a.e.web.EndpointLinksResolver      : Exposing 2 endpoint(s) beneath base path '/actuator'
    2020-01-12 10:09:24.516  INFO 3104 --- [main] com.example.demo.DemoApplicationTests    : Started DemoApplicationTests in 3.886 seconds (JVM running for 5.747)
    2020-01-12 10:09:25.677  INFO 3104 --- [SpringContextShutdownHook] o.s.s.concurrent.ThreadPoolTaskExecutor  : Shutting down ExecutorService 'applicationTaskExecutor'

## ##

### \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* ###

**相关配置**

**filebeat配置**

filebeat.inputs:
    - type: log
      enabled: true
      paths:
        - /var/log/*.log
    
    output.logstash:
      hosts: ["172.18.0.32:5044"]

**logstash管道配置**

input {
      beats {
        port => 5044
      }
    }
    
    output {
      elasticsearch { 
        hosts => ["172.18.0.33:9200"]
        index => "log-%{+yyyy.MM.dd}"
      }
    }

## ##

### \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* ###

**docker 创建容器**

docker run -it --net fixed --ip 172.18.0.31 \
    -v /usr/elasticsearch/filebeat/config/filebeat3.yml:/usr/share/filebeat/filebeat.yml \
    -v /usr/elasticsearch/filebeat/logs:/usr/share/filebeat/logs \
    --name filebeat docker.elastic.co/beats/filebeat:7.5.1
    
    docker run -it --net fixed --ip 172.18.0.32 -p 5044:5044 \
    -v /usr/elasticsearch/logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml \
    -v /usr/elasticsearch/logstash/config/logstash3.conf:/usr/share/logstash/pipeline/logstash.conf \
    --name logstash docker.elastic.co/logstash/logstash:7.5.1
    
    docker run -it --net fixed --ip 172.18.0.33 -p 9201:9200 -p 9301:9300  \
    -e ES_JAVA_OPTS="-Xms512m -Xmx512m"  \
    -v /usr/elasticsearch/single/conf/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml \
    --name es-single2 elasticsearch:7.5.1
    
    docker run -it --net fixed --ip 172.18.0.34 -p 5601:5601 \
    -e ELASTICSEARCH_HOSTS="http://172.18.0.33:9200" \
    --name kibana docker.elastic.co/kibana/kibana:7.5.1

### \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* ###

**kibana显示**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzMTYyNQ_size_16_color_FFFFFF_t_70][]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzMTYyNQ_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20200112134837264.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzMTYyNQ==,size_16,color_FFFFFF,t_70