Docker部署EFK+Kafka日志收集系统

落日映苍穹つ 2022-01-12 22:15 627阅读 0赞

日志收集流程为：fluentd - kafka - logstash - elasticsearch - kibana

使用镜像如下：

<table> 
 <tbody> 
  <tr> 
   <td>registry.cn-hangzhou.aliyuncs.com/xiao_bai/fluentd-kafka:v1&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;version:1.4</td> 
  </tr> 
  <tr> 
   <td>registry.cn-hangzhou.aliyuncs.com/xiao_bai/kafka:v1&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;version:2.1</td> 
  </tr> 
  <tr> 
   <td>docker.io/zookeeper:3.5</td> 
  </tr> 
  <tr> 
   <td>docker.elastic.co/elasticsearch/elasticsearch:6.5.4</td> 
  </tr> 
  <tr> 
   <td>docker.elastic.co/logstash/logstash:6.5.4</td> 
  </tr> 
  <tr> 
   <td>docker.elastic.co/kibana/kibana:6.5.4</td> 
  </tr> 
  <tr> 
   <td>docker.io/mobz/elasticsearch-head:5</td> 
  </tr> 
 </tbody> 
</table>

<table> 
 <tbody> 
  <tr> 
   <td>192.168.2.244</td> 
   <td>fluentd、zk、kafka、es、kibana</td> 
  </tr> 
  <tr> 
   <td>192.168.2.245</td> 
   <td>logstash、es-head、zk、kafka、es</td> 
  </tr> 
  <tr> 
   <td>192.168.2.246</td> 
   <td>es-balance、zk、kafka、es</td> 
  </tr> 
 </tbody> 
</table>

mkdir -p /efk/{es,es-balance,kafka,zk,fluentd,logstash}/conf && mkdir /efk/{es,kafka,zk}/data

*  系统环境配置

echo "vm.max_map_count=262144" >> /etc/sysctl.conf && sysctl -p
    echo -e "* soft nofile 65536\n* hard nofile 65536" >> /etc/security/limits.conf

*  zk集群安装

配置文件(三台机器使用相同配置)

cat > /efk/zk/conf/zoo.cfg << EOF
    tickTime=2000
    dataDir=/data/zookeeper/
    dataLogDir=/data/zookeeper/logs
    clientPort=2181
    initLimit=5
    syncLimit=2
    server.1=192.168.2.244:2888:3888
    server.2=192.168.2.245:2888:3888
    server.3=192.168.2.246:2888:3888
    EOF

为每台机器设置myid

echo 1 > /efk/zk/data/myid    #244     跟配置文件server.id设置一致
    echo 2 > /efk/zk/data/myid    #245
    echo 3 > /efk/zk/data/myid    #246

启动容器(三台机器执行相同命令)

docker run -d --restart=always --name zk --net=host -v /etc/localtime:/etc/localtime:ro -v /efk/zk/conf/zoo.cfg:/conf/zoo.cfg -v /efk/zk/data/:/data/zookeeper zookeeper:3.5

zk集群启动后查看各节点身份

docker exec -it zk zkServer.sh status

*  kafka集群安装

配置文件(每台机器的broker.id不能相同)

cat > /efk/kafka/conf/server.properties << EOF
    broker.id=1
    listeners=PLAINTEXT://192.168.2.246:9092
    port=9092
    logs.dirs=/tmp/kafka-logs
    message.max.byte=5242880
    default.replication.factor=2
    replica.fetch.max.bytes=5242880
    zookeeper.connect=192.168.2.244:2181,192.168.2.245:2181,192.168.2.246:2181
    EOF

启动容器(三台机器执行相同命令)

docker run -d --restart=always --name=kafka --net=host -v /etc/localtime:/etc/localtime:ro -v /efk/kafka/conf/server.properties:/usr/local/kafka_2.11-2.1.0/config/server.properties -v /efk/kafka/data:/tmp/kafka-logs registry.cn-hangzhou.aliyuncs.com/xiao_bai/kafka:v1

*  es集群安装

配置文件(修改node.name、network.publish\_host)

cat > /efk/es/conf/elasticsearch.yml << EOF
    cluster.name: es-cluster
    node.name: es-node1
    network.bind_host: 0.0.0.0
    network.publish_host: 192.168.2.244
    http.port: 9200
    transport.tcp.port: 9300
    http.cors.enabled: true
    http.cors.allow-origin: "*"
    node.master: true 
    node.data: true  
    discovery.zen.ping.unicast.hosts: ["192.168.2.244:9300","192.168.2.245:9300","192.168.2.246:9300"]
    discovery.zen.minimum_master_nodes: 2
    EOF

balance节点配置文件

cat > /efk/es-balance/conf/elasticsearch.yml << EOF
    cluster.name: es-cluster
    node.name: es-node4
    network.bind_host: 0.0.0.0
    network.publish_host: 192.168.2.246
    http.port: 9201
    transport.tcp.port: 9301
    http.cors.enabled: true
    http.cors.allow-origin: "*"
    node.master: false
    node.data: false
    node.ingest: false
    discovery.zen.ping.unicast.hosts: ["192.168.2.244:9300","192.168.2.245:9300","192.168.2.246:9300"]
    discovery.zen.minimum_master_nodes: 2
    EOF

启动容器(三台机器执行相同命令，数据节点)

docker run -d --restart=always --name=es --net=host -e ES_JAVA_OPTS="-Xms2g -Xmx2g" -v /etc/localtime:/etc/localtime:ro -v /efk/es/conf/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml -v /efk/es/data:/usr/share/elasticsearch/data docker.elastic.co/elasticsearch/elasticsearch:6.5.4

balance节点

docker run -d --restart=always --name=es-balance --net=host -e ES_JAVA_OPTS="-Xms512m -Xmx512m" -v /etc/localtime:/etc/localtime:ro -v /efk/es-balance/conf/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml docker.elastic.co/elasticsearch/elasticsearch:6.5.4

启动es-head插件

docker run -d --name=es-head -p 9100:9100 mobz/elasticsearch-head:5

*  kibana安装

启动容器

docker run -d --name kibana --restart=always -e ELASTICSEARCH_URL=http://192.168.2.246:9201 -v /etc/localtime:/etc/localtime:ro -p 5601:5601 docker.elastic.co/kibana/kibana:6.5.4

*  logstash安装

配置文件

consumer\_threads与kafka设置的分区数保持一致

touch /efk/logstash/conf/logstash.yml
    cat > /efk/logstash/conf/kafka-logstash-es.conf << EOF
    input {
        kafka {
    bootstrap_servers => "192.168.2.244:9092,192.168.2.245:9092,192.168.2.246:9092"
    group_id => "kafka-logstash"
    topics => ["docker_log","sys_log"]
    consumer_threads => 1
    codec => "json"
            }
        }
    output {
        elasticsearch {
    hosts => ["192.168.2.244:9200","192.168.2.245:9200","192.168.2.246:9200"]
    index => "logstash-%{+YYYY.MM.dd}"
    codec => "json"
        }
        stdout {
    codec => "rubydebug"
            }
        }
    EOF

启动logstash容器

docker run -d --name=logstash --net=host -v /etc/localtime:/etc/localtime:ro -v /efk/logstash/conf/logstash.yml:/usr/share/logstash/config/logstash.yml -v /efk/logstash/conf/kafka-logstash-es.conf:/usr/share/logstash/pipeline/kafka-logstash-es.conf docker.elastic.co/logstash/logstash:6.5.4

*  Fluentd安装

配置文件(收集容器日志以及系统日志)

最后一个match需要设置为\*\*，否则会出现warn信息

cat > /efk/fluentd/conf/fluent.conf << EOF
    <source>
      @type tail
      path /var/log/containers/*/*.log
      pos_file /var/log/containers/containers.log.pos
      tag docker.log 
      <parse>
        @type json
      </parse>
      read_from_head true
    </source>
    <source>
      @type syslog
      port 5140
      bind 0.0.0.0
      <parse>
        message_format rfc3164
      </parse>
      tag system.log
    </source>
    <filter **>
      @type record_transformer
      <record>
        hostname "192.168.2.244"
      </record>
    </filter>
    <match docker.*>
      @type kafka_buffered
      brokers 192.168.2.244:9092,192.168.2.245:9092,192.168.2.246:9092
      default_topic docker_log
      output_data_type json
    </match>
    <match **>
      @type kafka_buffered
      brokers 192.168.2.244:9092,192.168.2.245:9092,192.168.2.246:9092
      default_topic sys_log
      output_data_type json
    </match>
    EOF

收集系统日志需要开启rsyslog服务(根据服务器IP进行配置)

echo '*.* @192.168.2.244:5140' >> /etc/rsyslog.conf && systemctl restart rsyslog

启动fluentd容器

docker run -d --name=fluentd --net=host -v /etc/localtime:/etc/localtime:ro -v /efk/fluentd/conf/fluent.conf:/fluentd/etc/fluent.conf -v /var/lib/docker/containers/:/var/log/containers registry.cn-hangzhou.aliyuncs.com/xiao_bai/fluentd-kafka:v1

参考文档：

[https://www.elastic.co/guide][https_www.elastic.co_guide]

[https://github.com/mobz/elasticsearch-head][https_github.com_mobz_elasticsearch-head]

[https://docs.fluentd.org][https_docs.fluentd.org]

[http://kafka.apache.org/21/documentation.html][http_kafka.apache.org_21_documentation.html]

[http://zookeeper.apache.org/doc/r3.5.4-beta/zookeeperAdmin.html][http_zookeeper.apache.org_doc_r3.5.4-beta_zookeeperAdmin.html]

转载于:https://blog.51cto.com/13740724/2387263

[https_www.elastic.co_guide]: https://www.elastic.co/guide
[https_github.com_mobz_elasticsearch-head]: https://github.com/mobz/elasticsearch-head
[https_docs.fluentd.org]: https://docs.fluentd.org
[http_kafka.apache.org_21_documentation.html]: http://kafka.apache.org/21/documentation.html
[http_zookeeper.apache.org_doc_r3.5.4-beta_zookeeperAdmin.html]: http://zookeeper.apache.org/doc/r3.5.4-beta/zookeeperAdmin.html