nginx集成kafka

迷南。 2022-02-19 08:15 231阅读 0赞

nginx日志直接被kafka消费，省去flume组件。

# 安装nginx的c的库，编译，安装nginx集成kafka插件，编译 #

安装nginx的c的库  
1.安装git  
sudo yum install -y git  
2.切换到/usr/local/src目录，然后将kafka的c客户端源码clone到本地  
cd /usr/local/src  
sudo yum update -y nss curl libcurl  
sudo git clone [https://github.com/edenhill/librdkafka][https_github.com_edenhill_librdkafka]  
3.进入到librdkafka，然后进行编译  
cd librdkafka  
sudo yum install -y gcc gcc-c++ pcre-devel zlib-devel  
sudo ./configure  
sudo make  
sudo make install

4.安装nginx整合kafka的插件，进入到/usr/local/src，clone nginx整合kafka的源码  
cd /usr/local/src  
sudo git clone [https://github.com/brg-liuwei/ngx\_kafka\_module][https_github.com_brg-liuwei_ngx_kafka_module]

5.进入到nginx的源码包目录下 （编译nginx，然后将将插件同时编译）  
cd /~/nginx-1.12.2  
sudo ./configure --add-module=/usr/local/src/ngx\_kafka\_module/  
sudo make  
sudo make install

6.修改nginx的配置文件，详情请查看当前目录的nginx.conf

#user  nobody;
    worker_processes  1;
    
    #error_log  logs/error.log;
    #error_log  logs/error.log  notice;
    #error_log  logs/error.log  info;
    
    #pid        logs/nginx.pid;
    
    
    events {
        worker_connections  1024;
    }
    
    
    http {
        include       mime.types;
        default_type  application/octet-stream;
    
        #log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
        #                  '$status $body_bytes_sent "$http_referer" '
        #                  '"$http_user_agent" "$http_x_forwarded_for"';
        #access_log  logs/access.log  main;
        sendfile        on;
        #tcp_nopush     on;
        #keepalive_timeout  0;
        keepalive_timeout  65;
        #gzip  on;
        
        kafka;
        kafka_broker_list node-1.xiaoniu.com:9092 node-2.xiaoniu.com:9092 node-3.xiaoniu.com:9092; 	
        
        server {
            listen       80;
            server_name  node-6.xiaoniu.com;
            #charset koi8-r;
            #access_log  logs/host.access.log  main;
    
        	location = /kafka/track {
                    kafka_topic track;
            }
    
        	location = /kafka/user {
                    kafka_topic user;
            }
    
            #error_page  404              /404.html;
    
            # redirect server error pages to the static page /50x.html
            #
            error_page   500 502 503 504  /50x.html;
            location = /50x.html {
                root   html;
            }
    
        }
    
    }

8.启动nginx，报错，找不到kafka.so.1的文件  
error while loading shared libraries: librdkafka.so.1: cannot open shared object file: No such file or directory  
9.加载so库  
echo “/usr/local/lib” >> /etc/ld.so.conf  
sudo sh -c “echo ‘/usr/local/lib’ >> /etc/ld.so.conf”  
sudo ldconfig

[https_github.com_edenhill_librdkafka]: https://github.com/edenhill/librdkafka
[https_github.com_brg-liuwei_ngx_kafka_module]: https://github.com/brg-liuwei/ngx_kafka_module