FlinkX 实现原理(全局总览)

末蓝、 2023-02-12 05:43 15阅读 0赞

## FlinkX 实现原理 ##

从上一节的代码结构，我们知道FlinkX的结构是`Core+Plugin`，我们就简单分析一下`Core`.

## Core 代码架构 ##

![image-20200517173901628][]

### 基础支持 ###

*  classloader 自定义类加载器
 *  Exception 统一异常
 *  enums 统一枚举
 *  constants 常量
 *  options 入参选项配置
 *  util 工具类
 *  latch 同步类支持

### 核心逻辑 ###

*  config 配置文件格式schema
 *  reader Reader插件
 *  writer Writer插件
 *  inputformat
    
    ![image-20200517174241116][]
 *  outputformat
    
    ![image-20200517174303556][]

### 安全与核心功能 ###

*  authenticate 安全
 *  restore **断点续传** **实时采集**
 *  streaming 流支持

### 监控与日志 ###

*  log 日志
 *  metrics 指标监控

## 流程 (Main.java) ##

![image-20200517181309368][]

FlinkX 本质就是创建了一个`Flink`执行环境，然后将**Source数据源**经由`Reader`的`readData`方法创建`InputFormat`读取数据并转化为`DataStream`，然后将数据经由`Writer`的`writeData`方法创建的`OutputFormat`导出到**Target 数据源**。

BaseDataReader dataReader = DataReaderFactory.getDataReader(config, env);
            DataStream<Row> dataStream = dataReader.readData();
            if(speedConfig.getReaderChannel() > 0){ 
                dataStream = ((DataStreamSource<Row>) dataStream).setParallelism(speedConfig.getReaderChannel());
            }
    
            if (speedConfig.isRebalance()) { 
                dataStream = dataStream.rebalance();
            }
    
            BaseDataWriter dataWriter = DataWriterFactory.getDataWriter(config);
            DataStreamSink<?> dataStreamSink = dataWriter.writeData(dataStream);
            if(speedConfig.getWriterChannel() > 0){ 
                dataStreamSink.setParallelism(speedConfig.getWriterChannel());
            }

### InputFormat ###

![image-20200517175025686][]

### OutputFormat ###

![image-20200517175014210][]

[image-20200517173901628]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly93dWh1bGFsYS5vc3MtY24tc2hhbmdoYWkuYWxpeXVuY3MuY29tL2Jsb2cvaW1hZ2UtMjAyMDA1MTcxNzM5MDE2MjgucG5n?x-oss-process=image/format,png
[image-20200517174241116]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly93dWh1bGFsYS5vc3MtY24tc2hhbmdoYWkuYWxpeXVuY3MuY29tL2Jsb2cvaW1hZ2UtMjAyMDA1MTcxNzQyNDExMTYucG5n?x-oss-process=image/format,png
[image-20200517174303556]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly93dWh1bGFsYS5vc3MtY24tc2hhbmdoYWkuYWxpeXVuY3MuY29tL2Jsb2cvaW1hZ2UtMjAyMDA1MTcxNzQzMDM1NTYucG5n?x-oss-process=image/format,png
[image-20200517181309368]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly93dWh1bGFsYS5vc3MtY24tc2hhbmdoYWkuYWxpeXVuY3MuY29tL2Jsb2cvaW1hZ2UtMjAyMDA1MTcxODEzMDkzNjgucG5n?x-oss-process=image/format,png
[image-20200517175025686]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly93dWh1bGFsYS5vc3MtY24tc2hhbmdoYWkuYWxpeXVuY3MuY29tL2Jsb2cvaW1hZ2UtMjAyMDA1MTcxNzUwMjU2ODYucG5n?x-oss-process=image/format,png
[image-20200517175014210]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly93dWh1bGFsYS5vc3MtY24tc2hhbmdoYWkuYWxpeXVuY3MuY29tL2Jsb2cvaW1hZ2UtMjAyMDA1MTcxNzUwMTQyMTAucG5n?x-oss-process=image/format,png