大数据学习笔记第2课（续）通过filebeat收集nginx访问日志到kafka集群

￡神魔★判官ぃ 2022-12-28 04:22 148阅读 0赞

### 大数据学习笔记第2课（续） 通过filebeat收集nginx访问日志到kafka集群 ###

*  1、说明
 *  2、环境说明
 *  3、Nginx安装与配置
 *  4、启动一个Mini Http Server处理5000端口的请求
 *  5、filebeat下载、配置与安装
 *  6、启动filebeat开启日志收集
 *  7、通过java程序发送http请求至nginx，使nginx产生访问日志。
 *  8、通过java程序模拟kafka消息消费
 *  9、结尾

# 1、说明 #

本文是接着我的上一篇博文《[大数据学习笔记第2课 Zookeeper & Kafka集群搭建][2_ Zookeeper _ Kafka]》往下写的，请先阅读上一篇文章，不然会感到比较突兀。

# 2、环境说明 #

本次搭建的环境是在华为云上  
1、服务器：三台华为云ECS（都是4核8G）

<table> 
 <thead> 
  <tr> 
   <th>计算机名</th> 
   <th>内网IP</th> 
   <th>公网IP</th> 
   <th>别名</th> 
   <th>说明</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>ecs-ae8a-0001</td> 
   <td>192.168.0.177</td> 
   <td>x.x.x.x</td> 
   <td>hadoop01</td> 
   <td>zookeeper/broker01</td> 
  </tr> 
  <tr> 
   <td>ecs-ae8a-0002</td> 
   <td>192.168.0.56</td> 
   <td>x.x.x.x</td> 
   <td>hadoop02</td> 
   <td>broker02</td> 
  </tr> 
  <tr> 
   <td>ecs-ae8a-0003</td> 
   <td>192.168.0.59</td> 
   <td>x.x.x.x</td> 
   <td>hadoop03</td> 
   <td>broker03</td> 
  </tr> 
 </tbody> 
</table>

2、系统版本：CentOS Linux release 7.5.1804 (Core)  
3、zookeeper与kafka服务都已正常启动，且已创建了1个topic，值为qingdao

# 3、Nginx安装与配置 #

1、在CentOS上安装Nginx非常简单，我们再hadoop01服务器上执行以下命令：

yum install nginx

安装日志如下：

[root@ecs-ae8a-0001 ~]# yum install nginx
    Loaded plugins: fastestmirror
    Determining fastest mirrors
    epel/x86_64/metalink                                                                                                                                                                                                                                   | 8.4 kB  00:00:00     
     * base: mirrors.huaweicloud.com
     * epel: mirrors.bfsu.edu.cn
     * extras: mirrors.bfsu.edu.cn
     * updates: mirrors.huaweicloud.com
    base                                                                                                                                                                                                                                                   | 3.6 kB  00:00:00     
    epel                                                                                                                                                                                                                                                   | 4.7 kB  00:00:00     
    extras                                                                                                                                                                                                                                                 | 2.9 kB  00:00:00     
    updates                                                                                                                                                                                                                                                | 2.9 kB  00:00:00     
    (1/7): epel/x86_64/group_gz                                                                                                                                                                                                                            |  95 kB  00:00:00     
    (2/7): extras/7/x86_64/primary_db                                                                                                                                                                                                                      | 222 kB  00:00:00     
    (3/7): base/7/x86_64/primary_db                                                                                                                                                                                                                        | 6.1 MB  00:00:00     
    (4/7): epel/x86_64/updateinfo                                                                                                                                                                                                                          | 1.0 MB  00:00:00     
    (5/7): epel/x86_64/primary_db                                                                                                                                                                                                                          | 6.9 MB  00:00:00     
    (6/7): base/7/x86_64/group_gz                                                                                                                                                                                                                          | 153 kB  00:00:07     
    (7/7): updates/7/x86_64/primary_db                                                                                                                                                                                                                     | 3.7 MB  00:00:08     
    Resolving Dependencies
    --> Running transaction check
    ---> Package nginx.x86_64 1:1.16.1-3.el7 will be installed
    --> Processing Dependency: nginx-all-modules = 1:1.16.1-3.el7 for package: 1:nginx-1.16.1-3.el7.x86_64
    --> Processing Dependency: nginx-filesystem = 1:1.16.1-3.el7 for package: 1:nginx-1.16.1-3.el7.x86_64
    --> Processing Dependency: libcrypto.so.1.1(OPENSSL_1_1_0)(64bit) for package: 1:nginx-1.16.1-3.el7.x86_64
    --> Processing Dependency: libssl.so.1.1(OPENSSL_1_1_0)(64bit) for package: 1:nginx-1.16.1-3.el7.x86_64
    --> Processing Dependency: libssl.so.1.1(OPENSSL_1_1_1)(64bit) for package: 1:nginx-1.16.1-3.el7.x86_64
    --> Processing Dependency: nginx-filesystem for package: 1:nginx-1.16.1-3.el7.x86_64
    --> Processing Dependency: redhat-indexhtml for package: 1:nginx-1.16.1-3.el7.x86_64
    --> Processing Dependency: libcrypto.so.1.1()(64bit) for package: 1:nginx-1.16.1-3.el7.x86_64
    --> Processing Dependency: libprofiler.so.0()(64bit) for package: 1:nginx-1.16.1-3.el7.x86_64
    --> Processing Dependency: libssl.so.1.1()(64bit) for package: 1:nginx-1.16.1-3.el7.x86_64
    --> Running transaction check
    ---> Package centos-indexhtml.noarch 0:7-9.el7.centos will be installed
    ---> Package gperftools-libs.x86_64 0:2.6.1-1.el7 will be installed
    ---> Package nginx-all-modules.noarch 1:1.16.1-3.el7 will be installed
    --> Processing Dependency: nginx-mod-http-image-filter = 1:1.16.1-3.el7 for package: 1:nginx-all-modules-1.16.1-3.el7.noarch
    --> Processing Dependency: nginx-mod-http-perl = 1:1.16.1-3.el7 for package: 1:nginx-all-modules-1.16.1-3.el7.noarch
    --> Processing Dependency: nginx-mod-http-xslt-filter = 1:1.16.1-3.el7 for package: 1:nginx-all-modules-1.16.1-3.el7.noarch
    --> Processing Dependency: nginx-mod-mail = 1:1.16.1-3.el7 for package: 1:nginx-all-modules-1.16.1-3.el7.noarch
    --> Processing Dependency: nginx-mod-stream = 1:1.16.1-3.el7 for package: 1:nginx-all-modules-1.16.1-3.el7.noarch
    ---> Package nginx-filesystem.noarch 1:1.16.1-3.el7 will be installed
    ---> Package openssl11-libs.x86_64 1:1.1.1g-1.el7 will be installed
    --> Running transaction check
    ---> Package nginx-mod-http-image-filter.x86_64 1:1.16.1-3.el7 will be installed
    --> Processing Dependency: gd for package: 1:nginx-mod-http-image-filter-1.16.1-3.el7.x86_64
    --> Processing Dependency: libgd.so.2()(64bit) for package: 1:nginx-mod-http-image-filter-1.16.1-3.el7.x86_64
    ---> Package nginx-mod-http-perl.x86_64 1:1.16.1-3.el7 will be installed
    ---> Package nginx-mod-http-xslt-filter.x86_64 1:1.16.1-3.el7 will be installed
    --> Processing Dependency: libxslt.so.1(LIBXML2_1.0.11)(64bit) for package: 1:nginx-mod-http-xslt-filter-1.16.1-3.el7.x86_64
    --> Processing Dependency: libxslt.so.1(LIBXML2_1.0.18)(64bit) for package: 1:nginx-mod-http-xslt-filter-1.16.1-3.el7.x86_64
    --> Processing Dependency: libexslt.so.0()(64bit) for package: 1:nginx-mod-http-xslt-filter-1.16.1-3.el7.x86_64
    --> Processing Dependency: libxslt.so.1()(64bit) for package: 1:nginx-mod-http-xslt-filter-1.16.1-3.el7.x86_64
    ---> Package nginx-mod-mail.x86_64 1:1.16.1-3.el7 will be installed
    ---> Package nginx-mod-stream.x86_64 1:1.16.1-3.el7 will be installed
    --> Running transaction check
    ---> Package gd.x86_64 0:2.0.35-26.el7 will be installed
    --> Processing Dependency: libjpeg.so.62(LIBJPEG_6.2)(64bit) for package: gd-2.0.35-26.el7.x86_64
    --> Processing Dependency: libjpeg.so.62()(64bit) for package: gd-2.0.35-26.el7.x86_64
    --> Processing Dependency: libfontconfig.so.1()(64bit) for package: gd-2.0.35-26.el7.x86_64
    --> Processing Dependency: libXpm.so.4()(64bit) for package: gd-2.0.35-26.el7.x86_64
    --> Processing Dependency: libX11.so.6()(64bit) for package: gd-2.0.35-26.el7.x86_64
    ---> Package libxslt.x86_64 0:1.1.28-6.el7 will be installed
    --> Running transaction check
    ---> Package fontconfig.x86_64 0:2.13.0-4.3.el7 will be installed
    --> Processing Dependency: fontpackages-filesystem for package: fontconfig-2.13.0-4.3.el7.x86_64
    --> Processing Dependency: dejavu-sans-fonts for package: fontconfig-2.13.0-4.3.el7.x86_64
    ---> Package libX11.x86_64 0:1.6.7-3.el7_9 will be installed
    --> Processing Dependency: libX11-common >= 1.6.7-3.el7_9 for package: libX11-1.6.7-3.el7_9.x86_64
    --> Processing Dependency: libxcb.so.1()(64bit) for package: libX11-1.6.7-3.el7_9.x86_64
    ---> Package libXpm.x86_64 0:3.5.12-1.el7 will be installed
    ---> Package libjpeg-turbo.x86_64 0:1.2.90-8.el7 will be installed
    --> Running transaction check
    ---> Package dejavu-sans-fonts.noarch 0:2.33-6.el7 will be installed
    --> Processing Dependency: dejavu-fonts-common = 2.33-6.el7 for package: dejavu-sans-fonts-2.33-6.el7.noarch
    ---> Package fontpackages-filesystem.noarch 0:1.44-8.el7 will be installed
    ---> Package libX11-common.noarch 0:1.6.7-3.el7_9 will be installed
    ---> Package libxcb.x86_64 0:1.13-1.el7 will be installed
    --> Processing Dependency: libXau.so.6()(64bit) for package: libxcb-1.13-1.el7.x86_64
    --> Running transaction check
    ---> Package dejavu-fonts-common.noarch 0:2.33-6.el7 will be installed
    ---> Package libXau.x86_64 0:1.0.8-2.1.el7 will be installed
    --> Finished Dependency Resolution
    
    Dependencies Resolved
    
    ==============================================================================================================================================================================================================================================================================
     Package                                                                        Arch                                                      Version                                                            Repository                                                  Size
    ==============================================================================================================================================================================================================================================================================
    Installing:
     nginx                                                                          x86_64                                                    1:1.16.1-3.el7                                                     epel                                                       563 k
    Installing for dependencies:
     centos-indexhtml                                                               noarch                                                    7-9.el7.centos                                                     base                                                        92 k
     dejavu-fonts-common                                                            noarch                                                    2.33-6.el7                                                         base                                                        64 k
     dejavu-sans-fonts                                                              noarch                                                    2.33-6.el7                                                         base                                                       1.4 M
     fontconfig                                                                     x86_64                                                    2.13.0-4.3.el7                                                     base                                                       254 k
     fontpackages-filesystem                                                        noarch                                                    1.44-8.el7                                                         base                                                       9.9 k
     gd                                                                             x86_64                                                    2.0.35-26.el7                                                      base                                                       146 k
     gperftools-libs                                                                x86_64                                                    2.6.1-1.el7                                                        base                                                       272 k
     libX11                                                                         x86_64                                                    1.6.7-3.el7_9                                                      updates                                                    607 k
     libX11-common                                                                  noarch                                                    1.6.7-3.el7_9                                                      updates                                                    164 k
     libXau                                                                         x86_64                                                    1.0.8-2.1.el7                                                      base                                                        29 k
     libXpm                                                                         x86_64                                                    3.5.12-1.el7                                                       base                                                        55 k
     libjpeg-turbo                                                                  x86_64                                                    1.2.90-8.el7                                                       base                                                       135 k
     libxcb                                                                         x86_64                                                    1.13-1.el7                                                         base                                                       214 k
     libxslt                                                                        x86_64                                                    1.1.28-6.el7                                                       base                                                       242 k
     nginx-all-modules                                                              noarch                                                    1:1.16.1-3.el7                                                     epel                                                        20 k
     nginx-filesystem                                                               noarch                                                    1:1.16.1-3.el7                                                     epel                                                        21 k
     nginx-mod-http-image-filter                                                    x86_64                                                    1:1.16.1-3.el7                                                     epel                                                        30 k
     nginx-mod-http-perl                                                            x86_64                                                    1:1.16.1-3.el7                                                     epel                                                        39 k
     nginx-mod-http-xslt-filter                                                     x86_64                                                    1:1.16.1-3.el7                                                     epel                                                        29 k
     nginx-mod-mail                                                                 x86_64                                                    1:1.16.1-3.el7                                                     epel                                                        57 k
     nginx-mod-stream                                                               x86_64                                                    1:1.16.1-3.el7                                                     epel                                                        85 k
     openssl11-libs                                                                 x86_64                                                    1:1.1.1g-1.el7                                                     epel                                                       1.4 M
    
    Transaction Summary
    ==============================================================================================================================================================================================================================================================================
    Install  1 Package (+22 Dependent packages)
    
    Total download size: 5.9 M
    Installed size: 18 M
    Is this ok [y/d/N]: y
    Downloading packages:
    warning: /var/cache/yum/x86_64/7/base/packages/centos-indexhtml-7-9.el7.centos.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY
    Public key for centos-indexhtml-7-9.el7.centos.noarch.rpm is not installed
    (1/23): centos-indexhtml-7-9.el7.centos.noarch.rpm                                                                                                                                                                                                     |  92 kB  00:00:00     
    (2/23): dejavu-sans-fonts-2.33-6.el7.noarch.rpm                                                                                                                                                                                                        | 1.4 MB  00:00:00     
    (3/23): fontconfig-2.13.0-4.3.el7.x86_64.rpm                                                                                                                                                                                                           | 254 kB  00:00:00     
    (4/23): fontpackages-filesystem-1.44-8.el7.noarch.rpm                                                                                                                                                                                                  | 9.9 kB  00:00:00     
    (5/23): gd-2.0.35-26.el7.x86_64.rpm                                                                                                                                                                                                                    | 146 kB  00:00:00     
    (6/23): gperftools-libs-2.6.1-1.el7.x86_64.rpm                                                                                                                                                                                                         | 272 kB  00:00:00     
    (7/23): libXau-1.0.8-2.1.el7.x86_64.rpm                                                                                                                                                                                                                |  29 kB  00:00:00     
    (8/23): libjpeg-turbo-1.2.90-8.el7.x86_64.rpm                                                                                                                                                                                                          | 135 kB  00:00:00     
    (9/23): dejavu-fonts-common-2.33-6.el7.noarch.rpm                                                                                                                                                                                                      |  64 kB  00:00:00     
    (10/23): libxcb-1.13-1.el7.x86_64.rpm                                                                                                                                                                                                                  | 214 kB  00:00:00     
    warning: /var/cache/yum/x86_64/7/epel/packages/nginx-1.16.1-3.el7.x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID 352c64e5: NOKEY
    Public key for nginx-1.16.1-3.el7.x86_64.rpm is not installed
    (11/23): nginx-1.16.1-3.el7.x86_64.rpm                                                                                                                                                                                                                 | 563 kB  00:00:00     
    (12/23): nginx-all-modules-1.16.1-3.el7.noarch.rpm                                                                                                                                                                                                     |  20 kB  00:00:00     
    (13/23): nginx-filesystem-1.16.1-3.el7.noarch.rpm                                                                                                                                                                                                      |  21 kB  00:00:00     
    (14/23): nginx-mod-http-image-filter-1.16.1-3.el7.x86_64.rpm                                                                                                                                                                                           |  30 kB  00:00:00     
    (15/23): nginx-mod-http-perl-1.16.1-3.el7.x86_64.rpm                                                                                                                                                                                                   |  39 kB  00:00:00     
    (16/23): nginx-mod-http-xslt-filter-1.16.1-3.el7.x86_64.rpm                                                                                                                                                                                            |  29 kB  00:00:00     
    (17/23): nginx-mod-mail-1.16.1-3.el7.x86_64.rpm                                                                                                                                                                                                        |  57 kB  00:00:00     
    (18/23): nginx-mod-stream-1.16.1-3.el7.x86_64.rpm                                                                                                                                                                                                      |  85 kB  00:00:00     
    (19/23): libxslt-1.1.28-6.el7.x86_64.rpm                                                                                                                                                                                                               | 242 kB  00:00:00     
    (20/23): openssl11-libs-1.1.1g-1.el7.x86_64.rpm                                                                                                                                                                                                        | 1.4 MB  00:00:00     
    (21/23): libXpm-3.5.12-1.el7.x86_64.rpm                                                                                                                                                                                                                |  55 kB  00:00:00     
    Public key for libX11-common-1.6.7-3.el7_9.noarch.rpm is not installed
    (22/23): libX11-common-1.6.7-3.el7_9.noarch.rpm                                                                                                                                                                                                        | 164 kB  00:00:00     
    (23/23): libX11-1.6.7-3.el7_9.x86_64.rpm                                                                                                                                                                                                               | 607 kB  00:00:00     
    ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    Total                                                                                                                                                                                                                                          14 MB/s | 5.9 MB  00:00:00     
    Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
    Importing GPG key 0xF4A80EB5:
     Userid     : "CentOS-7 Key (CentOS 7 Official Signing Key) <security@centos.org>"
     Fingerprint: 6341 ab27 53d7 8a78 a7c2 7bb1 24c6 a8a7 f4a8 0eb5
     Package    : centos-release-7-5.1804.el7.centos.x86_64 (@anaconda)
     From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
    Is this ok [y/N]: y
    Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
    Importing GPG key 0x352C64E5:
     Userid     : "Fedora EPEL (7) <epel@fedoraproject.org>"
     Fingerprint: 91e9 7d7c 4a5e 96f1 7f3e 888f 6a2f aea2 352c 64e5
     Package    : epel-release-7-12.noarch (@epel)
     From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
    Is this ok [y/N]: y
    Running transaction check
    Running transaction test
    Transaction test succeeded
    Running transaction
    Warning: RPMDB altered outside of yum.
      Installing : fontpackages-filesystem-1.44-8.el7.noarch                                                                                                                                                                                                                 1/23 
      Installing : dejavu-fonts-common-2.33-6.el7.noarch                                                                                                                                                                                                                     2/23 
      Installing : dejavu-sans-fonts-2.33-6.el7.noarch                                                                                                                                                                                                                       3/23 
      Installing : fontconfig-2.13.0-4.3.el7.x86_64                                                                                                                                                                                                                          4/23 
      Installing : gperftools-libs-2.6.1-1.el7.x86_64                                                                                                                                                                                                                        5/23 
      Installing : 1:nginx-filesystem-1.16.1-3.el7.noarch                                                                                                                                                                                                                    6/23 
      Installing : libxslt-1.1.28-6.el7.x86_64                                                                                                                                                                                                                               7/23 
      Installing : libXau-1.0.8-2.1.el7.x86_64                                                                                                                                                                                                                               8/23 
      Installing : libxcb-1.13-1.el7.x86_64                                                                                                                                                                                                                                  9/23 
      Installing : 1:openssl11-libs-1.1.1g-1.el7.x86_64                                                                                                                                                                                                                     10/23 
      Installing : centos-indexhtml-7-9.el7.centos.noarch                                                                                                                                                                                                                   11/23 
      Installing : libX11-common-1.6.7-3.el7_9.noarch                                                                                                                                                                                                                       12/23 
      Installing : libX11-1.6.7-3.el7_9.x86_64                                                                                                                                                                                                                              13/23 
      Installing : libXpm-3.5.12-1.el7.x86_64                                                                                                                                                                                                                               14/23 
      Installing : libjpeg-turbo-1.2.90-8.el7.x86_64                                                                                                                                                                                                                        15/23 
      Installing : gd-2.0.35-26.el7.x86_64                                                                                                                                                                                                                                  16/23 
      Installing : 1:nginx-mod-mail-1.16.1-3.el7.x86_64                                                                                                                                                                                                                     17/23 
      Installing : 1:nginx-mod-http-perl-1.16.1-3.el7.x86_64                                                                                                                                                                                                                18/23 
      Installing : 1:nginx-mod-http-xslt-filter-1.16.1-3.el7.x86_64                                                                                                                                                                                                         19/23 
      Installing : 1:nginx-mod-stream-1.16.1-3.el7.x86_64                                                                                                                                                                                                                   20/23 
      Installing : 1:nginx-1.16.1-3.el7.x86_64                                                                                                                                                                                                                              21/23 
      Installing : 1:nginx-mod-http-image-filter-1.16.1-3.el7.x86_64                                                                                                                                                                                                        22/23 
      Installing : 1:nginx-all-modules-1.16.1-3.el7.noarch                                                                                                                                                                                                                  23/23 
      Verifying  : fontconfig-2.13.0-4.3.el7.x86_64                                                                                                                                                                                                                          1/23 
      Verifying  : 1:nginx-mod-mail-1.16.1-3.el7.x86_64                                                                                                                                                                                                                      2/23 
      Verifying  : 1:nginx-1.16.1-3.el7.x86_64                                                                                                                                                                                                                               3/23 
      Verifying  : fontpackages-filesystem-1.44-8.el7.noarch                                                                                                                                                                                                                 4/23 
      Verifying  : dejavu-fonts-common-2.33-6.el7.noarch                                                                                                                                                                                                                     5/23 
      Verifying  : 1:nginx-mod-http-perl-1.16.1-3.el7.x86_64                                                                                                                                                                                                                 6/23 
      Verifying  : libxcb-1.13-1.el7.x86_64                                                                                                                                                                                                                                  7/23 
      Verifying  : libXpm-3.5.12-1.el7.x86_64                                                                                                                                                                                                                                8/23 
      Verifying  : libjpeg-turbo-1.2.90-8.el7.x86_64                                                                                                                                                                                                                         9/23 
      Verifying  : libX11-common-1.6.7-3.el7_9.noarch                                                                                                                                                                                                                       10/23 
      Verifying  : libX11-1.6.7-3.el7_9.x86_64                                                                                                                                                                                                                              11/23 
      Verifying  : 1:nginx-mod-http-xslt-filter-1.16.1-3.el7.x86_64                                                                                                                                                                                                         12/23 
      Verifying  : dejavu-sans-fonts-2.33-6.el7.noarch                                                                                                                                                                                                                      13/23 
      Verifying  : gd-2.0.35-26.el7.x86_64                                                                                                                                                                                                                                  14/23 
      Verifying  : 1:nginx-mod-stream-1.16.1-3.el7.x86_64                                                                                                                                                                                                                   15/23 
      Verifying  : centos-indexhtml-7-9.el7.centos.noarch                                                                                                                                                                                                                   16/23 
      Verifying  : 1:openssl11-libs-1.1.1g-1.el7.x86_64                                                                                                                                                                                                                     17/23 
      Verifying  : libXau-1.0.8-2.1.el7.x86_64                                                                                                                                                                                                                              18/23 
      Verifying  : libxslt-1.1.28-6.el7.x86_64                                                                                                                                                                                                                              19/23 
      Verifying  : 1:nginx-filesystem-1.16.1-3.el7.noarch                                                                                                                                                                                                                   20/23 
      Verifying  : gperftools-libs-2.6.1-1.el7.x86_64                                                                                                                                                                                                                       21/23 
      Verifying  : 1:nginx-all-modules-1.16.1-3.el7.noarch                                                                                                                                                                                                                  22/23 
      Verifying  : 1:nginx-mod-http-image-filter-1.16.1-3.el7.x86_64                                                                                                                                                                                                        23/23 
    
    Installed:
      nginx.x86_64 1:1.16.1-3.el7                                                                                                                                                                                                                                                 
    
    Dependency Installed:
      centos-indexhtml.noarch 0:7-9.el7.centos          dejavu-fonts-common.noarch 0:2.33-6.el7  dejavu-sans-fonts.noarch 0:2.33-6.el7    fontconfig.x86_64 0:2.13.0-4.3.el7      fontpackages-filesystem.noarch 0:1.44-8.el7        gd.x86_64 0:2.0.35-26.el7                 
      gperftools-libs.x86_64 0:2.6.1-1.el7              libX11.x86_64 0:1.6.7-3.el7_9            libX11-common.noarch 0:1.6.7-3.el7_9     libXau.x86_64 0:1.0.8-2.1.el7           libXpm.x86_64 0:3.5.12-1.el7                       libjpeg-turbo.x86_64 0:1.2.90-8.el7       
      libxcb.x86_64 0:1.13-1.el7                        libxslt.x86_64 0:1.1.28-6.el7            nginx-all-modules.noarch 1:1.16.1-3.el7  nginx-filesystem.noarch 1:1.16.1-3.el7  nginx-mod-http-image-filter.x86_64 1:1.16.1-3.el7  nginx-mod-http-perl.x86_64 1:1.16.1-3.el7 
      nginx-mod-http-xslt-filter.x86_64 1:1.16.1-3.el7  nginx-mod-mail.x86_64 1:1.16.1-3.el7     nginx-mod-stream.x86_64 1:1.16.1-3.el7   openssl11-libs.x86_64 1:1.1.1g-1.el7   
    
    Complete!
    [root@ecs-ae8a-0001 ~]# which nginx
    /usr/sbin/nginx

中间有几步Is this OK?的提问的地方输入y就行。  
2、安装完成后  
nginx日志目录为：/var/log/nginx  
nginx配置文件目录为：/etc/nginx  
3、修改/etc/nginx/nginx.conf，内容如下：

# For more information on configuration, see:
    #   * Official English Documentation: http://nginx.org/en/docs/
    #   * Official Russian Documentation: http://nginx.org/ru/docs/
    
    user nginx;
    worker_processes auto;
    error_log /var/log/nginx/error.log;
    pid /run/nginx.pid;
    
    # Load dynamic modules. See /usr/share/doc/nginx/README.dynamic.
    include /usr/share/nginx/modules/*.conf;
    
    events {
        worker_connections 1024;
    }
    
    http {
        log_format  main  escape=none '$http_x_forwarded_for -[$time_local] "$request" '
    			'$status $body_bytes_sent '
    			'"#http_user_agent"';
    
        access_log  /var/log/nginx/access.log  main;
    
        sendfile            on;
        tcp_nopush          on;
        tcp_nodelay         on;
        keepalive_timeout   65;
        types_hash_max_size 2048;
    
        include             /etc/nginx/mime.types;
        default_type        application/octet-stream;
        
        include /etc/nginx/conf.d/*.conf;
    
        server {
            listen       80 default_server;
            listen       [::]:80 default_server;
            server_name  _;
            root         /usr/share/nginx/html;
    
            # Load configuration files for the default server block.
            include /etc/nginx/default.d/*.conf;
    
            location / {
    		proxy_pass http://127.0.0.1:5000;
            }
    
            error_page 404 /404.html;
            location = /404.html {
            }
    
            error_page 500 502 503 504 /50x.html;
            location = /50x.html {
            }
        }
    }

> 上面的配置主要修改了2个内容
> 
>  *  把80端口接收到的请求转发到5000端口下处理。
>  *  对访问日志的格式进行格式化处理。

4、启动nginx服务，在终端执行以下命令：

nginx

5、如果中间手动删除了/var/log/nginx目录下的日志文件，则需要让Nginx重新加载一下，否则不会自动生成新的日志文件，如下：

killall -s USR1 nginx

# 4、启动一个Mini Http Server处理5000端口的请求 #

1、为了能够处理Nginx转发至5000端口的请求，我们可以通过python开启一个httpserver，如下命令：

nohup phython -m SimpleHTTPServer 5000 &

# 5、filebeat下载、配置与安装 #

1、切换当前目录为/opt/soft

cd /opt/soft

2、执行以下命令下载filebeat，如下：

wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.2.4-linux-x86_64.tar.gz

3、解压filebeat

tar -xzvf filebeat-6.2.4-linux-x86_64.tar.gz

4、复制filebeat至/program/目录下：

cp -r filebeat-6.2.4-linux-x86_64 /program/

5、切换当前目录至/program/filebeat-6.2.4-linux-x86\_64/

cd /program/filebeat-6.2.4-linux-x86_64

6、创建一个空文件kafka.yml

touch kafka.yml

7、通过vim kafka.yml修改kafka.yml文件内容，内容如下：

filebeat:
      prospectors:
        -
          paths:
            - /var/log/nginx/access.log
          input_type: log
    output:
      kafka:
        hosts: ["hadoop01:9092","hadoop02:9092","hadoop03:9092"]
        topic: qingdao

# 6、启动filebeat开启日志收集 #

执行以下命令

nohup ./filebeat -e -c kafka.yml &

# 7、通过java程序发送http请求至nginx，使nginx产生访问日志。 #

1、运行test01的模拟java程序。  
2、在浏览器地址栏输入访问地址，访问格式如下：

> http:// 启 动 服 务 的 i p 及 端 口 / t e s t / m o c k d a t a / s t a r t ? u r l = h t t p : / / \{启动服务的ip及端口\}/test/mockdata/start?url=http:// 启动服务的ip及端口/test/mockdata/start?url=http://\{nginx的ip\}&thread=1

![在这里插入图片描述][20201214172949472.png]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ_size_16_color_FFFFFF_t_70]

# 8、通过java程序模拟kafka消息消费 #

1、运行kafka消费者程序（[完整代码][Link 1]），执行效果如下：

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ_size_16_color_FFFFFF_t_70 1]

# 9、结尾 #

至此使用filebeat收集日志数据至kafka集群的环境就搭建完成了，小伙伴们如果觉得有用就点个赞吧~

[2_ Zookeeper _ Kafka]: https://coding123.blog.csdn.net/article/details/110926717
[20201214172949472.png]: /images/20221120/a104b8f12279441996afe5d7dbd6063b.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ_size_16_color_FFFFFF_t_70]: /images/20221120/2d79274611154c948d9f055def7db11d.png
[Link 1]: https://coding123.blog.csdn.net/article/details/110949839
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20201214173351151.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ==,size_16,color_FFFFFF,t_70