四、Kafka 监控、Flume对接

客官°小女子只卖身不卖艺 2023-06-03 15:55 12阅读 0赞

# 一、Kafka 监控Kafka Eagle配置 #

**1、修改 kafka 启动命令**

修改 kafka-server-start.sh 命令中

if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
    export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"
    fi

改为

if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
    export KAFKA_HEAP_OPTS="-server -Xms2G -Xmx2G -XX:PermSize=128m
    -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:ParallelGCThreads=8 -
    XX:ConcGCThreads=5 -XX:InitiatingHeapOccupancyPercent=70"
    export JMX_PORT="9999"
    #export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"
    fi

**注意：修改之后在启动 Kafka 之前要分发之其他节点**

**2.上传压缩包 kafka-eagle-bin-1.3.7.tar.gz 到集群/opt/software 目录**

**3.解压到本地**

tar -zxvf kafka-eagle-bin-1.3.7.tar.gz

**4.进入刚才解压的目录**

ll
    总用量 82932
    -rw-rw-r--. 1 atguigu atguigu 84920710 8 月 13 23:00 kafka-eagleweb-1.3.7-bin.tar.gz

**5.将 kafka-eagle-web-1.3.7-bin.tar.gz 解压至/opt/module**

tar -zxvf kafka-eagleweb-1.3.7-bin.tar.gz -C /opt/module/

**6.修改名称**

mv kafka-eagle-web-1.3.7/ eagle

**7.给启动文件执行权限**

cd bin/
    chmod 777 ke.sh

**8.修改配置文件**

######################################
    # multi zookeeper&kafka cluster list
    ######################################
    kafka.eagle.zk.cluster.alias=cluster1
    cluster1.zk.list=hadoop102:2181,hadoop103:2181,hadoop104:2181
    ######################################
    # kafka offset storage
    ######################################
    cluster1.kafka.eagle.offset.storage=kafka
    ######################################
    # enable kafka metrics
    ######################################
    kafka.eagle.metrics.charts=true
    kafka.eagle.sql.fix.error=false
    ######################################
    # kafka jdbc driver address
    ######################################
    kafka.eagle.driver=com.mysql.jdbc.Driver
    kafka.eagle.url=jdbc:mysql://hadoop102:3306/
            ke?useUnicode=true&characterEncoding=UTF-8&zeroDateTimeBehavior=convertToNull
    kafka.eagle.username=root
    kafka.eagle.password=000000

**9.添加环境变量**

export KE_HOME=/opt/module/eagle
    export PATH=$PATH:$KE_HOME/bin
    # 立即生效
    source /etc/profile

**10.启动**

bin/ke.sh start

**注意：启动之前需要先启动 ZK 以及 KAFKA**

**11.登录页面查看监控数据**

http://ip:8048/ke

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3MyOTQ4NzgzMDQ_size_16_color_FFFFFF_t_70][]

# 二、Flume 对接 Kafka #

**1）配置 flume(flume-kafka.conf)**

# define
    a1.sources = r1
    a1.sinks = k1
    a1.channels = c1
    # source
    a1.sources.r1.type = exec
    a1.sources.r1.command = tail -F -c +0 /opt/module/data/flume.log
    a1.sources.r1.shell = /bin/bash -c
    # sink
    a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
    a1.sinks.k1.kafka.bootstrap.servers =hadoop102:9092,hadoop103:9092,hadoop104:9092
    a1.sinks.k1.kafka.topic = first
    a1.sinks.k1.kafka.flumeBatchSize = 20
    a1.sinks.k1.kafka.producer.acks = 1
    a1.sinks.k1.kafka.producer.linger.ms = 1
    # channel
    a1.channels.c1.type = memory
    a1.channels.c1.capacity = 1000
    a1.channels.c1.transactionCapacity = 100
    # bind
    a1.sources.r1.channels = c1
    a1.sinks.k1.channel = c1

**2） 启动 kafkaIDEA 消费者**

**3） 进入 flume 根目录下，启动 flume**

bin/flume-ng agent -c conf/ -n a1 -f jobs/flume-kafka.conf

**4） 向 /opt/module/data/flume.log 里追加数据，查看 kafka 消费者消费情况**

echo hello >> /opt/module/data/flume.log

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3MyOTQ4NzgzMDQ_size_16_color_FFFFFF_t_70]: /images/20230601/64a23186034044d3bcb3679bd75794b1.png