Flink 1.12.2 源码浅析 : 本地DEBUG环境准备

灰太狼 2022-11-07 05:44 17阅读 0赞

### . ###

*  一 .前言
 *  二 .环境准备
 *   *  2.1. Hdfs & Yarn 环境.
     *  2.2. Flink
     *   *  2.2.1. 文件下载
     *  2.3. 安装
     *  2.4. 配置环境变量
     *  2.5. 验证

# 一 .前言 #

准备开始看Flink的源码,为了方便,所以要准备一套可以DEBUG的环境.

环境清单 :

<table> 
 <thead> 
  <tr> 
   <th>组件</th> 
   <th>版本</th> 
   <th>备注</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>JDK</td> 
   <td>1.8</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>SCALA</td> 
   <td>2.11.x</td> 
   <td>…</td> 
  </tr> 
  <tr> 
   <td>HDFS</td> 
   <td>3.2.1</td> 
   <td>…</td> 
  </tr> 
  <tr> 
   <td>YARN</td> 
   <td>3.2.1</td> 
   <td>…</td> 
  </tr> 
  <tr> 
   <td>Flink</td> 
   <td>1.12.2</td> 
   <td>…</td> 
  </tr> 
  <tr> 
   <td>…</td> 
   <td>…</td> 
   <td>…</td> 
  </tr> 
  <tr> 
   <td>…</td> 
   <td>…</td> 
   <td>…</td> 
  </tr> 
 </tbody> 
</table>

# 二 .环境准备 #

## 2.1. Hdfs & Yarn 环境. ##

hadoop这里我用的是3.2.1版本. 是提前准备好的. 所以不再做说明. 直接启动服务就好.

*  **HDFS** : http://127.0.0.1:9870/dfshealth.html\#tab-overview  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3poYW5nbG9uZ180NDQ0_size_16_color_FFFFFF_t_70]

注意 hdfs不要处于安全模式.

> hdfs dfsadmin -safemode leave

*  **YARN** : http://127.0.0.1:8088/cluster  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3poYW5nbG9uZ180NDQ0_size_16_color_FFFFFF_t_70 1]

## 2.2. Flink ##

### 2.2.1. 文件下载 ###

<table> 
 <thead> 
  <tr> 
   <th>安装包</th> 
   <th>下载地址</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>flink-1.12.2-bin-scala_2.11</td> 
   <td>https://www.apache.org/dyn/closer.lua/flink/flink-1.12.2/flink-1.12.2-bin-scala_2.11.tgz</td> 
  </tr> 
  <tr> 
   <td>hadoop 依赖包</td> 
   <td>https://mvnrepository.com/artifact/org.apache.flink/flink-shaded-hadoop-3-uber/3.1.1.7.1.1.0-565-9.0</td> 
  </tr> 
 </tbody> 
</table>

> 我没找到flink关于3.2.1版本的hadoop依赖所以先拿3.1.1版本的代替…

## 2.3. 安装 ##

1.  直接将flink-1.12.2-bin-scala\_2.11.tgz安装包解压到/opt/tools目录.
2.  将hadoop依赖包放到 flink-shaded-hadoop-3-uber-3.1.1.7.1.1.0-565-9.0.jar 目录下.

## 2.4. 配置环境变量 ##

> vi /etc/profile

export HADOOP_HOME=/opt/tools/hadoop-3.2.1
    export PATH=$PATH:$HADOOP_HOME/bin
    
    export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
    export PATH=$PATH:$HADOOP_CONF_DIR
    
    export FLINK_HOME=/opt/tools/flink-1.12.2
    export PATH=$PATH:$FLINK_HOME/bin

## 2.5. 验证 ##

执行命令:

> cd $\{FLINK\_HOME\}
> 
> ./bin/flink run -m yarn-cluster ./examples/batch/WordCount.jar

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3poYW5nbG9uZ180NDQ0_size_16_color_FFFFFF_t_70 2]

BoYi-Pro:flink-1.12.2 sysadmin$ flink run -m yarn-cluster ./examples/batch/WordCount.jar
    Executing WordCount example with default input data set.
    Use --input to specify file input.
    Printing result to stdout. Use --output to specify output path.
    2021-03-11 15:25:43,885 WARN  org.apache.flink.yarn.configuration.YarnLogConfigUtil        [] - The configuration directory ('/opt/tools/flink-1.12.2/conf') already contains a LOG4J config file.If you want to use logback, then please delete or rename the log configuration file.
    2021-03-11 15:25:43,941 INFO  org.apache.hadoop.yarn.client.RMProxy                        [] - Connecting to ResourceManager at /0.0.0.0:8032
    2021-03-11 15:25:44,214 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
    2021-03-11 15:25:44,548 INFO  org.apache.hadoop.conf.Configuration                         [] - resource-types.xml not found
    2021-03-11 15:25:44,549 INFO  org.apache.hadoop.yarn.util.resource.ResourceUtils           [] - Unable to find 'resource-types.xml'.
    2021-03-11 15:25:44,667 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - The configured JobManager memory is 1600 MB. YARN will allocate 2048 MB to make up an integer multiple of its minimum allocation memory (1024 MB, configured via 'yarn.scheduler.minimum-allocation-mb'). The extra 448 MB may not be used by Flink.
    2021-03-11 15:25:44,668 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - The configured TaskManager memory is 1728 MB. YARN will allocate 2048 MB to make up an integer multiple of its minimum allocation memory (1024 MB, configured via 'yarn.scheduler.minimum-allocation-mb'). The extra 320 MB may not be used by Flink.
    2021-03-11 15:25:44,668 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Cluster specification: ClusterSpecification{ masterMemoryMB=1600, taskManagerMemoryMB=1728, slotsPerTaskManager=1}
    2021-03-11 15:25:50,839 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Submitting application master application_1615446205104_0003
    2021-03-11 15:25:51,314 INFO  org.apache.hadoop.yarn.client.api.impl.YarnClientImpl        [] - Submitted application application_1615446205104_0003
    2021-03-11 15:25:51,314 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Waiting for the cluster to be allocated
    2021-03-11 15:25:51,319 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Deploying cluster, current state ACCEPTED
    2021-03-11 15:26:11,497 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - YARN application has been deployed successfully.
    2021-03-11 15:26:11,502 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Found Web Interface 192.168.8.188:59203 of application 'application_1615446205104_0003'.
    Job has been submitted with JobID 94f4cb52086349a4ee2a8f4bac6e4974
    Program execution finished
    Job with JobID 94f4cb52086349a4ee2a8f4bac6e4974 has finished.
    Job Runtime: 21409 ms
    Accumulator Results:
    - aa803f083e41e6b8aa48e3e33a5f5212 (java.util.ArrayList) [170 elements]
    (a,5)
    (action,1)
    (nobler,1)
    (not,2)
    (now,1)
    (nymph,1)
    .......

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3poYW5nbG9uZ180NDQ0_size_16_color_FFFFFF_t_70]: /images/20221023/3cfbbf7dff0a4500acd02f26896f474f.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3poYW5nbG9uZ180NDQ0_size_16_color_FFFFFF_t_70 1]: /images/20221023/83d34edeaa394038a3bb80ed9ff6c9b0.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3poYW5nbG9uZ180NDQ0_size_16_color_FFFFFF_t_70 2]: /images/20221023/64b33253d9c5478bb0013bc97740db47.png