【Logstash】安装配置及对接Kafka实战

谁借莪１个温暖的怀抱￠ 2022-12-20 12:53 230阅读 0赞

### 文章目录 ###

*  配置环境
 *   *  版本选择及安装
     *  前提环境
     *  初体验 hello
 *  对接kafka
 *   *  启动kafka
     *  logstash输入输出测试
     *  安全配置
     *  效果图

# 配置环境 #

## 版本选择及安装 ##

Logstash 6.6.2版本下载  
[https://www.elastic.co/cn/downloads/past-releases/logstash-6-6-2][https_www.elastic.co_cn_downloads_past-releases_logstash-6-6-2]  
官方文档  
[https://www.elastic.co/guide/en/logstash/6.6/first-event.html][https_www.elastic.co_guide_en_logstash_6.6_first-event.html]

上传文件后解压

[root@hadoop01 software]# tar -zxf logstash-6.6.2.tar.gz -C ../install/

## 前提环境 ##

java 1.8环境

java -version
    java version "1.8.0_261"

## 初体验 hello ##

启动：进入logstash/bin目录

logstash-6.6.2]# bin/logstash -e 'input { stdin { } } output { stdout {} }'

测试：输入文本  
Logstash将时间戳和IP地址信息添加到消息中  
![在这里插入图片描述][20201113085304187.png_pic_center]

停止：  
CTRL-D

# 对接kafka #

## 启动kafka ##

参考之前写过的kafka  
[https://bennyrhys.blog.csdn.net/article/details/109091286][https_bennyrhys.blog.csdn.net_article_details_109091286]  
启动zk  
启动kafka  
创建topic

kafka_2.13-2.6.0]# bin/kafka-topics.sh --create --zookeeper 192.168.137.121:2181 --replication-factor 1 --partitions 1 --topic logstash_k

查看主题

bin/kafka-topics.sh --zookeeper 192.168.137.121:2181 --describe
    // 可查看已创建的topic列表
    bin/kafka-topics.sh --zookeeper 192.168.137.121:2181 --list
    // 可具体指定topic
    bin/kafka-topics.sh --zookeeper 192.168.137.121:2181 --describe --topic logstash_k

测试kafka环境（写入消费正常？）  
生产：product向broker写入数据

kafka_2.13-2.6.0]# bin/kafka-console-producer.sh --broker-list 192.168.137.121:9092 --topic logstash_k

消费：consumer从topic的partition消费数据

bin/kafka-console-consumer.sh --bootstrap-server 192.168.137.121:9092 --topic logstash_k --from-beginning

## logstash输入输出测试 ##

创建配置文件

logstash-6.6.2]# touch logstash.conf
    [root@hadoop01 logstash-6.6.2]# vim logstash.conf

input{ 
        stdin{ }
    }
    output{ 
        kafka{ 
            topic_id => "logstash_k"
            bootstrap_servers => "192.168.137.121:9092"  # kafka的地址
            batch_size => 5
        }
        stdout{ 
            codec => rubydebug
        }
    }

**效果图**  
logstash数据写到kafka中  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzQ2OTY4MA_size_16_color_FFFFFF_t_70_pic_center]  
kafka消费到的数据  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzQ2OTY4MA_size_16_color_FFFFFF_t_70_pic_center 1]

## 安全配置 ##

input{ 
        stdin{ }
    }
    output{ 
        kafka{ 
            topic_id => "odeon_test_tymiao"
            bootstrap_servers => ","  # kafka的地址
            jaas_path => "/opt/install/logstash-XXX/odeon_jass.config"
            security_protocol => "SASL_PLAINTEXT"
            sasl_mechanism => "SCRAM-SHA-256"
            codec => "json"
        }
    }

## 效果图 ##

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzQ2OTY4MA_size_16_color_FFFFFF_t_70_pic_center 2]

[https_www.elastic.co_cn_downloads_past-releases_logstash-6-6-2]: https://www.elastic.co/cn/downloads/past-releases/logstash-6-6-2
[https_www.elastic.co_guide_en_logstash_6.6_first-event.html]: https://www.elastic.co/guide/en/logstash/6.6/first-event.html
[20201113085304187.png_pic_center]: /images/20221120/f49c745f967f42e2858d1f54e56fafe6.png
[https_bennyrhys.blog.csdn.net_article_details_109091286]: https://bennyrhys.blog.csdn.net/article/details/109091286
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzQ2OTY4MA_size_16_color_FFFFFF_t_70_pic_center]: /images/20221120/5ff004f4161542e49386493f7bd94339.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzQ2OTY4MA_size_16_color_FFFFFF_t_70_pic_center 1]: /images/20221120/80a9ee44e43d4898a5b017c584389906.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzQ2OTY4MA_size_16_color_FFFFFF_t_70_pic_center 2]: /images/20221120/2e22de3a132640ceacb702f5b67404a2.png