NiFi 集群安装部署及使用

左手的ㄟ右手 2023-06-22 03:30 2阅读 0赞

1.  ##### NiFi 架构 #####
    
    ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA_size_16_color_FFFFFF_t_70][]
    
    NiFi在主机操作系统上的JVM内执行。JVM上NiFi的主要组件如下：
    
    1.  **Web Server**  
        Web服务器的目的是托管NiFi基于HTTP的命令和控制API。
    2.  **Flow Controller**  
        流量控制器是操作的大脑。它为扩展程序提供运行的线程，并管理扩展程序何时接收要执行的资源的计划。
    3.  **Extensions**  
        在其他文献中描述了各种类型的NiFi扩展。这里的关键点是扩展在JVM中运行和执行。
    4.  **FlowFile Repository**  
        FlowFile存储库是NiFi跟踪其对流中当前活动的给定FlowFile的了解状态的地方。存储库的实现是可插入的。默认方法是位于指定磁盘分区上的持久性预写日志。
    5.  **Content Repository**  
        内容存储库是给定FlowFile的实际内容字节。存储库的实现是可插入的。默认方法是一种相当简单的机制，它将数据块存储在文件系统中。可以指定多个文件系统存储位置，以便获得不同的物理分区以减少任何单个卷上的争用。
    6.  **Provenance Repository**  
        Provenance Repository是存储所有出处事件数据的地方。存储库构造是可插入的，默认实现是使用一个或多个物理磁盘卷。在每个位置内，事件数据被索引并可搜索。
2.  ##### 下载 NiFi 安装包并解压 #####
    
    下载地址: http://nifi.apache.org/download.html  
    ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA_size_16_color_FFFFFF_t_70 1][]
3.  ##### 单机部署 #####
    
    1.  ###### 修改 con/nifi.properties 配置文件(可不修改) ######
        
            # HTTP 主机地址。默认为空。
            nifi.web.http.host=DSJ-TVM001
            # HTTP 端口。默认为8080
            nifi.web.http.port=8080
    2.  ###### 启动 NiFi ######
        
        在 $\{NIFI\_HOME\}/bin 目录中，执行命令./nifi.sh .
        
        <table> 
         <thead> 
          <tr> 
           <th>command</th> 
           <th>描述</th> 
          </tr> 
         </thead> 
         <tbody> 
          <tr> 
           <td>start</td> 
           <td>后台启动 NiFi</td> 
          </tr> 
          <tr> 
           <td>stop</td> 
           <td>停止后台运行的 NiFi</td> 
          </tr> 
          <tr> 
           <td>status</td> 
           <td>查询 NiFi 的当前状态</td> 
          </tr> 
          <tr> 
           <td>run</td> 
           <td>在前台启动 NiFi，Ctrl+C 关闭 NiFi</td> 
          </tr> 
          <tr> 
           <td>install</td> 
           <td>将 NiFi 安装为服务。使用 service nifi {start|stop|run|restart|status}</td> 
          </tr> 
         </tbody> 
        </table>
        
        访问地址: http://localhost:8080
4.  ##### 集群部署 #####
    
    ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA_size_16_color_FFFFFF_t_70 2][]  
    **使用 NiFi 内嵌 ZooKeeper**  
    集群的默认状态提供程序为 ZooKeeperStateProvider。这意味着 NiFi 依赖ZooKeeper才能表现为群集。在许多部署 NiFi 的环境中，可能没有 集成 ZooKeeper 集群。为避免必须单独维护的 ZooKeeper 集群，NiFi 提供了启动嵌入式 ZooKeeper 集群。通常，建议在3个或5个节点上运行ZooKeeper。  
    参考: http://nifi.apache.org/docs/nifi-docs/html/administration-guide.html\#clustering
    
    1.  ###### 修改配置文件 nifi.properties ######
        
            # 当前 NiFi 实例是否运行嵌入式ZooKeeper服务器
            nifi.state.management.embedded.zookeeper.start=true
            # ZooKeeper 属性文件
            nifi.state.management.embedded.zookeeper.properties=./conf/zookeeper.properties
            # 当前 nifi 节点主机名
            nifi.web.http.host=cdh01
            #nifi.web.http.host=cdh02
            #nifi.web.http.host=cdh03
            # 当前 nifi 节点端口
            nifi.web.http.port=8990
            # 是否是集群中的节点，默认值为false。
            nifi.cluster.is.node=true
            # 设置为当前节点的主机名
            nifi.cluster.node.address=cdh01
            #nifi.cluster.node.address=cdh02
            #nifi.cluster.node.address=cdh03
            # 此端口设置为高于1024的开放端口(任何较低的端口都需要root)。
            nifi.cluster.node.protocol.port=8888
            # 用于与集群中其他节点通信的线程数。此属性默认为10。
            nifi.cluster.node.protocol.threads=10
            #与集群中其他节点进行通信的最大线程数。此属性默认为50。
            nifi.cluster.node.protocol.max.threads=50
            # 指定在选择流作为“正确”流之前要等待的时间
            nifi.cluster.flow.election.max.wait.time=5 mins
            # 指定集群中导致流的早期选择所需的节点数
            nifi.cluster.flow.election.max.candidates=1
            # cluster 负载均衡配置 #
            nifi.cluster.load.balance.host=cdh01
            nifi.cluster.load.balance.port=6342
            nifi.cluster.load.balance.connections.per.node=4
            nifi.cluster.load.balance.max.thread.count=8
            nifi.cluster.load.balance.comms.timeout=30 sec
            
            # ZooKeeper 列表 例如: cdh01:2181,cdh02:2181,cdh03:2181。
            nifi.zookeeper.connect.string=cdh01:2181,cdh02:2181,cdh03:2181
            nifi.zookeeper.connect.timeout=3 secs
            nifi.zookeeper.session.timeout=3 secs
            # ZooKeeper 存储数据根目录。默认值为 /nifi
            nifi.zookeeper.root.node=/nifi
    2.  ###### 修改 conf/state-management.xml 配置文件 ######
        
            <cluster-provider>
                <id>zk-provider</id>
                <class>org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider</class>
                <property name="Connect String">cdh01:2181,cdh02:2181,cdh03:2181</property>
                <property name="Root Node">/nifi</property>
                <property name="Session Timeout">10 seconds</property>
                <property name="Access Control">Open</property>
            </cluster-provider>
    3.  ###### 修改 conf/zookeeper.properties 配置文件 ######
        
            initLimit=10
            autopurge.purgeInterval=24
            syncLimit=5
            tickTime=2000
            dataDir=./state/zookeeper
            autopurge.snapRetainCount=30
            server.1=cdh01:2888:3888;2181
            server.2=cdh02:2888:3888;2181
            server.3=cdh03:2888:3888;2181
    4.  ###### 创建 ZooKeeper 节点ID ######
        
            #第一个节点
            cd $NIFI_HOME
            mkdir state
            mkdir state/zookeeper
            echo 1 > state/zookeeper/myid
            #第二个节点
            cd $NIFI_HOME
            mkdir state
            mkdir state/zookeeper
            echo 2 > state/zookeeper/myid
            #第三个节点
            cd $NIFI_HOME
            mkdir state
            mkdir state/zookeeper
            echo 3 > state/zookeeper/myid
        
        参考: http://nifi.apache.org/docs/nifi-docs/html/administration-guide.html\#embedded\_zookeeper
    5.  ###### 各个上节点启动 NiFi ######
        
        NiFi 集群中的任一节点均可访问  
        ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA_size_16_color_FFFFFF_t_70 3][]
5.  ##### NiFi 示例 #####
    
    1.  ###### NiFi 首页 ######
        
        ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA_size_16_color_FFFFFF_t_70 4][]
    2.  ###### 添加处理器 ######
        
        ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA_size_16_color_FFFFFF_t_70 5][]
        
        ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA_size_16_color_FFFFFF_t_70 6][]
    3.  ###### 配置处理器(GetFile 和 PutFile) ######
        
        **GetFile**  
        ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA_size_16_color_FFFFFF_t_70 7][]
        
        ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA_size_16_color_FFFFFF_t_70 8][]
        
        ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA_size_16_color_FFFFFF_t_70 9][]
        
        配置说明:  
        ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA_size_16_color_FFFFFF_t_70 10][]
        
        参考: http://nifi.apache.org/docs.html
        
        **PutFile**
        
        ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA_size_16_color_FFFFFF_t_70 11][]
        
        ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA_size_16_color_FFFFFF_t_70 12][]
    4.  ###### 连接处理器 ######
        
        ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA_size_16_color_FFFFFF_t_70 13][]
        
        ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA_size_16_color_FFFFFF_t_70 14][]  
        ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA_size_16_color_FFFFFF_t_70 15][]  
        ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA_size_16_color_FFFFFF_t_70 16][]
    5.  ###### 启动和停止处理器 ######
        
        **启动:**  
        ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA_size_16_color_FFFFFF_t_70 17][]  
        **停止:**  
        ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA_size_16_color_FFFFFF_t_70 18][]

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA_size_16_color_FFFFFF_t_70 19]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20191213190108492.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20191213172102168.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/20191213185352472.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA_size_16_color_FFFFFF_t_70 3]: https://img-blog.csdnimg.cn/20191213190323836.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA_size_16_color_FFFFFF_t_70 4]: https://img-blog.csdnimg.cn/20191216103125357.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA_size_16_color_FFFFFF_t_70 5]: https://img-blog.csdnimg.cn/20191216103327183.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA_size_16_color_FFFFFF_t_70 6]: https://img-blog.csdnimg.cn/2019121610435435.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA_size_16_color_FFFFFF_t_70 7]: https://img-blog.csdnimg.cn/20191216104426194.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA_size_16_color_FFFFFF_t_70 8]: https://img-blog.csdnimg.cn/2019121610504799.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA_size_16_color_FFFFFF_t_70 9]: https://img-blog.csdnimg.cn/20191216111715248.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA_size_16_color_FFFFFF_t_70 10]: https://img-blog.csdnimg.cn/20191216110120822.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA_size_16_color_FFFFFF_t_70 11]: https://img-blog.csdnimg.cn/20191216112141721.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA_size_16_color_FFFFFF_t_70 12]: https://img-blog.csdnimg.cn/20191216112507898.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA_size_16_color_FFFFFF_t_70 13]: https://img-blog.csdnimg.cn/20191216113826655.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA_size_16_color_FFFFFF_t_70 14]: https://img-blog.csdnimg.cn/20191216113844342.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA_size_16_color_FFFFFF_t_70 15]: https://img-blog.csdnimg.cn/20191216113953245.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA_size_16_color_FFFFFF_t_70 16]: https://img-blog.csdnimg.cn/2019121611412057.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA_size_16_color_FFFFFF_t_70 17]: https://img-blog.csdnimg.cn/20191216114331598.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA_size_16_color_FFFFFF_t_70 18]: https://img-blog.csdnimg.cn/20191216114501222.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA_size_16_color_FFFFFF_t_70 19]: https://img-blog.csdnimg.cn/2019121613001265.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA==,size_16,color_FFFFFF,t_70