工作流调度器azkaban

我会带着你远行 2022-12-18 02:56 194阅读 0赞

# 一、Azkaban概述 #

`Azkaban`是由`Linkedin`公司推出的一个批量工作流任务调度器，主要用于在一个工作流内以一个特定的顺序运行一组工作和流程，它的配置是通过简单的`key:value`对的方式，通过配置中的`dependencies`来设置依赖关系。`Azkaban`使用`job`配置文件建立任务之间的依赖关系，并提供一个易于使用的`web`用户界面维护和跟踪你的工作流。

## 1.1 为什么需要工作流调度系统 ##

*  一个完整的数据分析系统通常都是由大量任务单元组成：`shell`脚本程序，`java`程序，`MapReduce`程序、`hive`脚本等；
 *  各任务单元之间存在时间先后及前后依赖关系；
 *  为了很好地组织起这样的复杂执行计划，需要一个工作流调度系统来调度执行。

## 1.2 Azkaban特点 ##

1.  兼容任何版本的`Hadoop`
2.  易于使用的`Web`用户界面
3.  简单的工作流的上传
4.  方便设置任务之间的关系
5.  调度工作流
6.  模块化和可插拔的插件机制
7.  认证/授权(权限的工作)
8.  能够杀死并重新启动工作流
9.  有关失败和成功的电子邮件提醒

## 1.3 常见工作流调度系统 ##

*  简单的任务调度：直接使用`crontab`实现；
 *  复杂的任务调度：开发调度平台或使用现成的开源调度系统，比如`ooize`、`azkaban`等。

## 1.4 Azkaban的架构 ##

![在这里插入图片描述][20200703173002896.png]  
`Azkaban`由三个关键组件构成：

*  `AzkabanWebServer`：`AzkabanWebServer`是整个`Azkaban`工作流系统的主要管理者，它用户登录认证、负责`project`管理、定时执行工作流、跟踪工作流执行进度等一系列任务。
 *  `AzkabanExecutorServer`：负责具体的工作流的提交、执行，它们通过mysql数据库来协调任务的执行。
 *  关系型数据库（`MySQL`）：存储大部分执行流状态，`AzkabanWebServer`和`AzkabanExecutorServer`都需要访问数据库。

## 1.5 在3.0后，Azkaban提供了两种部署模式： ##

### 1.5.1、solo-server模式 ###

DB使用的是一个内嵌的H2，Web Server和Executor Server运行在同一个进程里。这种模式包含Azkaban的所有特性，但一般用来学习和测试，也可以用于小的应用。

### 1.5.2、distributed multiple-executor模式 ###

DB使用的是MySQL，MySQL最好使用master-slave架构，Web Server和Executor Server运行在不同机器上，且有多个Executor Server；这为Azkaban提供了很强的扩展性。

## 1.6 启动Azkaban ##

## 编译安装 ##

### 1、下载源码 ###

从github上下载azkaban官方源码，目前最新release为3.90.0

下载链接：[https://github.com/azkaban/azkaban/releases][https_github.com_azkaban_azkaban_releases]

### 2、编译 ###

进入到下载的azkaban目录中进行编译：（提前安装好git，gradle，jdk8）

./gradlew build -x test

BUILD SUCCESSFUL in 32m 7s  
73 actionable tasks: 73 executed

###  3、解压 ###

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70][]

/mine/azkaban-3.90.0/azkaban-web-server/build/distributions
    /mine/azkaban-3.90.0/azkaban-exec-server/build/distributions
    /mine/azkaban-3.90.0/azkaban-solo-server/build/distributions
    /mine/azkaban-3.90.0/azkaban-db/build/distributions
    azkaban-exec-server-0.1.0-SNAPSHOT.tar.gz
    azkaban-web-server-0.1.0-SNAPSHOT.tar.gz
    azkaban-solo-server-0.1.0-SNAPSHOT.tar.gz
    azkaban-db-0.1.0-SNAPSHOT.tar.gz

###  4、**azkaban web服务器安装** ###

解压azkaban-web-server-0.1.0-SNAPSHOT.tar.gz、azkaban-exec-server-0.1.0-SNAPSHOT.tar.gz、azkaban-db-0.1.0-SNAPSHOT.tar.gz

tar –zxvf azkaban-web-server-0.1.0-SNAPSHOT.tar.gz
    tar -zxvf azkaban-exec-server-0.1.0-SNAPSHOT.tar.gz
    tar -zxvf azkaban-db-0.1.0-SNAPSHOT.tar.gz
    tar -zxvf azkaban-solo-server-0.1.0-SNAPSHOT.tar.gz
    mv ../azkaban-db-0.1.0-SNAPSHOT sql

### 5、solo-server模式 ###

编译完成后，

cd /mine/azkaban-3.90.0/azkaban-solo-server/build/distributions

**5.1、修改时区**

conf/azkaban.properties：

#default.timezone.id=America/Los_Angeles 
    default.timezone.id=Asia/Shanghai

.default.servlet.path=/index
    web.resource.dir=/mine/azkaban/solo/web/
    default.timezone.id=Asia/Shanghai
    # Azkaban UserManager class
    user.manager.class=azkaban.user.XmlUserManager
    user.manager.xml.file=/mine/azkaban/solo/conf/azkaban-users.xml
    # Loader for projects
    executor.global.properties=/mine/azkaban/solo/conf/global.properties
    azkaban.project.dir=projects
    database.type=h2
    h2.path=./h2
    h2.create.tables=true
    # Velocity dev mode
    velocity.dev.mode=false
    # Azkaban Jetty server properties.
    jetty.use.ssl=false
    jetty.maxThreads=25
    jetty.port=8081
    # Azkaban Executor settings
    executor.port=12321
    # mail settings
    mail.sender=
    mail.host=
    # User facing web server configurations used to construct the user facing server URLs. They are useful when there is a reverse proxy between Azkaban web servers and users.
    # enduser -> myazkabanhost:443 -> proxy -> localhost:8081
    # when this parameters set then these parameters are used to generate email links.
    # if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used.
    # azkaban.webserver.external_hostname=myazkabanhost.com
    # azkaban.webserver.external_ssl_port=443
    # azkaban.webserver.external_port=8081
    job.failure.email=
    job.success.email=
    lockdown.create.projects=false
    cache.directory=cache
    # JMX stats
    jetty.connector.stats=true
    executor.connector.stats=true
    # Azkaban plugin settings
    azkaban.jobtype.plugin.dir=plugins/jobtypes
    # Number of executions to be displayed
    azkaban.display.execution_page_size=16
    azkaban.use.multiple.executors=true
    # Azkaban Ramp Feature Configuration
    #Ramp Feature Related
    azkaban.ramp.enabled=true
    azkaban.ramp.status.polling.enabled=true
    azkaban.ramp.status.polling.interval.min=30
    azkaban.ramp.status.push.interval.threshold=15
    azkaban.ramp.status.pull.interval.threshold=100

vim conf/azkaban-users.xml

<azkaban-users>
      <user groups="azkaban" password="azkaban" roles="admin" username="azkaban"/>
      <user password="metrics" roles="metrics" username="metrics"/>
    <user username="admin" password="admin" roles="admin,metrics" />
      <role name="admin" permissions="ADMIN"/>
      <role name="metrics" permissions="METRICS"/>
    </azkaban-users>

###  6、distributed multiple-executor模式 ###

假设Mysql已经安装完成，这里只考虑Web Server和Executor Server；源代码的编译在**192.168.60.101**上，目录为/mine/azkaban-3.90.0。

<table> 
 <tbody> 
  <tr> 
   <td>192.168.60.101</td> 
   <td>Web Server</td> 
  </tr> 
  <tr> 
   <td>192.168.60.102</td> 
   <td>Executor Server</td> 
  </tr> 
  <tr> 
   <td>192.168.60.103</td> 
   <td>Executor Server</td> 
  </tr> 
 </tbody> 
</table>

**6.1、创建数据库**

# ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 1][] #

# ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 2][] #

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 3][]

**6.2 部署Executor Server**

从192.168.60.101上拷贝部署包到192.168.60.102上

![20201029200607111.png][]

scp azkaban-exec-server-0.1.0-SNAPSHOT.tar.gz hadoop02:/mine

**192.168.60.102上部署Executor Server：**

修改conf/azkaban.properties文件：

# Azkaban Personalization Settings
    azkaban.name=Test
    azkaban.label=My Local Azkaban
    azkaban.color=#FF3601
    azkaban.default.servlet.path=/index
    web.resource.dir=web/
    default.timezone.id=Asia/Shanghai
    # Azkaban UserManager class
    user.manager.class=azkaban.user.XmlUserManager
    user.manager.xml.file=conf/azkaban-users.xml
    # Loader for projects
    executor.global.properties=conf/global.properties
    azkaban.project.dir=projects
    # Velocity dev mode
    velocity.dev.mode=false
    # Azkaban Jetty server properties.
    jetty.use.ssl=false
    jetty.maxThreads=25
    jetty.port=8081
    # Where the Azkaban web server is located
    azkaban.webserver.url=http://hadoop01:8081
    # mail settings
    mail.sender=
    mail.host=
    # User facing web server configurations used to construct the user facing server URLs. They are useful when there is a reverse proxy between Azkaban web servers and users.
    # enduser -> myazkabanhost:443 -> proxy -> localhost:8081
    # when this parameters set then these parameters are used to generate email links.
    # if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used.
    # azkaban.webserver.external_hostname=myazkabanhost.com
    # azkaban.webserver.external_ssl_port=443
    # azkaban.webserver.external_port=8081
    job.failure.email=
    job.success.email=
    lockdown.create.projects=false
    cache.directory=cache
    # JMX stats
    jetty.connector.stats=true
    executor.connector.stats=true
    # Azkaban plugin settings
    azkaban.jobtype.plugin.dir=plugins/jobtypes
    # Azkaban mysql settings by default. Users should configure their own username and password.
    database.type=mysql
    mysql.port=3306
    mysql.host=ljxwtl.cn
    mysql.database=azkaban
    mysql.user=root
    mysql.password=wtl199201180271
    mysql.numconnections=100
    # Azkaban Executor settings
    executor.maxThreads=50
    executor.flow.threads=30
    executor.port=6666

启动：

bin/start-exec.sh

** 问题：**

**如果出现连接MySQL 有问题，而且报错里有caused by:java.lang.NullPointerException**

是因为我用的mysql是8.\*版本的，然而azkaban自带的mysql的jdbc包是mysql-connector-java-5\*.jar，不支持，需要删掉原来的jdbc的驱动包，用mysql-connector-java-8\*.jar。

![20201029202617363.png][]

rm -rf mysql-connector-java-5.1.28.jar

![20201029203406753.png][]

启动完之后，还需要激活：

curl -G "localhost:12321/executor?action=activate" && echo

![20201029211730240.png][]

**192.168.60.103上部署Executor Server（同192.168.60.102（hadoop02））的操作一致！！！**

**6.3、部署Web Server**

可以直接使用192.168.600.101（hadoop01）上编译好的部署目录/mine/azkaban-web-server-0.1.0-SNAPSHOT

![20201029204348432.png][]

** 问题：**

**如果出现连接MySQL 有问题，而且报错里有caused by:java.lang.NullPointerException**

是因为我用的mysql是8.\*版本的，然而azkaban自带的mysql的jdbc包是mysql-connector-java-5\*.jar，不支持，需要删掉原来的jdbc的驱动包，用mysql-connector-java-8\*.jar。

![20201029202617363.png][]

rm -rf mysql-connector-java-5.1.28.jar

![20201029203406753.png][]

修改conf/azkaban.properties文件：

# Azkaban Personalization Settings
    azkaban.name=Test
    azkaban.label=My Local Azkaban
    azkaban.color=#FF3601
    azkaban.default.servlet.path=/index
    web.resource.dir=/mine/azkaban-web-server-0.1.0-SNAPSHOT/web/
    default.timezone.id=Asia/Shanghai
    # Azkaban UserManager class
    user.manager.class=azkaban.user.XmlUserManager
    user.manager.xml.file=/mine/azkaban-web-server-0.1.0-SNAPSHOT/conf/azkaban-users.xml
    # Loader for projects
    executor.global.properties=/mine/azkaban-web-server-0.1.0-SNAPSHOT/conf/global.properties
    azkaban.project.dir=projects
    # Velocity dev mode
    velocity.dev.mode=false
    # Azkaban Jetty server properties.
    jetty.use.ssl=false
    jetty.maxThreads=25
    jetty.port=8081
    # Azkaban Executor settings
    # mail settings
    mail.sender=
    mail.host=
    # User facing web server configurations used to construct the user facing server URLs. They are useful when there is a reverse proxy between Azkaban web servers and users.
    # enduser -> myazkabanhost:443 -> proxy -> localhost:8081
    # when this parameters set then these parameters are used to generate email links.
    # if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used.
    # azkaban.webserver.external_hostname=myazkabanhost.com
    # azkaban.webserver.external_ssl_port=443
    # azkaban.webserver.external_port=8081
    job.failure.email=
    job.success.email=
    lockdown.create.projects=false
    cache.directory=cache
    # JMX stats
    jetty.connector.stats=true
    executor.connector.stats=true
    # Azkaban mysql settings by default. Users should configure their own username and password.
    database.type=mysql
    mysql.port=3306
    mysql.host=ljxwtl.cn
    mysql.database=azkaban
    mysql.user=root
    mysql.password=wtl199201180271
    mysql.numconnections=100
    #Multiple Executor
    azkaban.use.multiple.executors=true
    azkaban.executorselector.filters=StaticRemainingFlowSize,MinimumFreeMemory,CpuStatus
    azkaban.executorselector.comparator.NumberOfAssignedFlowComparator=1
    azkaban.executorselector.comparator.Memory=1
    azkaban.executorselector.comparator.LastDispatched=1
    azkaban.executorselector.comparator.CpuUsage=1

vim azkaban-users.xml

问题：

2020/10/29 09:27:49.902 -0400  INFO [ExecutorManager] [AzkabanWebServer-QueueProcessor-Thread] [Azkaban] Using dispatcher for execution id :20
    2020/10/29 09:27:49.907 -0400  INFO [ExecutorManager] [AzkabanWebServer-QueueProcessor-Thread] [Azkaban] Reached handleNoExecutorSelectedCase stage for exec 20 with error count 0
    2020/10/29 09:27:49.907 -0400  INFO [ExecutorManager] [AzkabanWebServer-QueueProcessor-Thread] [Azkaban] Using dispatcher for execution id :20
    2020/10/29 09:27:49.908 -0400  INFO [ExecutorManager] [AzkabanWebServer-QueueProcessor-Thread] [Azkaban] Reached handleNoExecutorSelectedCase stage for exec 20 with error count 0

![20201029212834356.png][]

解决方案：

可能原因：

MinimumFreeMemory过滤器会检查executor主机空余内存是否会大于6G，如果不足6G，则web-server不会将任务交由该主机执行。

处理方法：

修改conf/azkaban.properties文件，去除MinimumFreeMemory过滤器

#azkaban.executorselector.filters=StaticRemainingFlowSize,MinimumFreeMemory,CpuStatus
    azkaban.executorselector.filters=StaticRemainingFlowSize,CpuStatus

![20201029213119846.png][]

问题：

![2020103107243283.png][]

原因：

azkaban要求执行主机可用内存必须大于3G才能满足执行任务的条件。

处理方法：

Executor Server部署目录下plugins/jobtypes/commonprivate.properties文件增加：

memCheck.enabled=false

# 二、Azkaban实战 #

**Azkaba内置的任务类型支持command、java**

## 1、Command类型单一job示例 ##

### （1）创建job描述文件 ###

vim command.job

### (2) 将job资源文件打包成zip文件 ###

#command.job
    type=command
    command=echo 'hello world'

zip command.job.zip command.job

###  （3）通过azkaban的web管理平台创建project并上传job压缩包 ###

首先创建project

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 4][]

上传zip包

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 5][]

**问题：**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 6][]

**原因：**

azkaban要求执行主机可用内存必须大于3G才能满足执行任务的条件。

plugins/jobtypes/commonprivate.properties文件增加：

memCheck.enabled=false

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 7][]

## 2、**Command类型多job工作流flow** ##

### （1）创建有依赖关系的多个job描述 ###

**第一个job：foo.job**

# foo.job
    type=command
    command=echo foo

** 第二个job：bar.job依赖foo.job**

# bar.job
    type=command
    dependencies=foo
    command=echo bar

** 将所有job资源文件打到一个zip包中**

![20201026212116675.png][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 8][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 9][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 10][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 11][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 12][]

## 3、**HDFS操作任务** ##

**（1）创建job描述文件**

# fs.job
    type=command
    command=hadoop fs -mkdir /azkaban

**（2）将job资源文件打包成zip文件 **

![20201031073423764.png][]

**（3）通过azkaban的web管理平台创建project并上传job压缩包  
（4）启动执行该job**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 13][]

## 4、**MAPREDUCE任务** ##

**Mr任务依然可以使用command的job类型来执行**

**（1）创建job描述文件，及mr程序jar包（示例中直接使用hadoop自带的example jar）**

# mrwc.job
    type=command
    command=hadoop jar /mine/hadoop-3.2.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.0.jar wordcount /wordcount/input /wordcount/azout

**（2）将所有job资源文件打到一个zip包中**

![20201031074542332.png][]

**（3）在azkaban的web管理界面创建工程并上传zip包**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 14][]  
**（4）启动job**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 15][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 16][]

## 4、**HIVE脚本任务** ##

**（1）创建job描述文件和hive脚本**

Hive脚本： test.sql

use default;
    drop table aztest;
    create table azkaban_student(id string,name string,sex string,age int,address string)
    row format delimited fields terminated by ','
    stored as textfile;
    load data local inpath '/mine/students.txt' into table azkaban_student;
    create table azres as select * from azkaban_student;
    insert overwrite directory '/aztest/hiveoutput' select count(1) from azkaban_student;

Job描述文件：hivef.job

# hivef.job
    type=command
    command=hive -f 'test.sql'

**(2) 将所有job资源文件打到一个zip包中**

![20201031105057657.png][]

**(3) 在azkaban的web管理界面创建工程并上传zip包**

**(4) 启动job**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 17][]

**报错原因：**

**在hadoop02里没有hive的程序，所以报错！！！**

**解决方案：**

**在每台executor机器里都安装hive。**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 18][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 19][]

[20200703173002896.png]: /images/20221122/95468d6471694d7681664a9d3e553130.png
[https_github.com_azkaban_azkaban_releases]: https://github.com/azkaban/azkaban/releases
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70]: /images/20221122/841b41cefe084b958f0410d4bd88416b.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 1]: /images/20221122/90b2debb1386418b992d460495b5b184.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 2]: /images/20221122/c870cdb39bc84ae3b1381cc086e2e708.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 3]: /images/20221122/2a359ebbe5164c65b8e896ee282c95c4.png
[20201029200607111.png]: /images/20221122/57455fe8328040f4b647188e965e3453.png
[20201029202617363.png]: /images/20221122/802eda139cc147959392ad6cd806b588.png
[20201029203406753.png]: /images/20221122/69c1399cde4344cdb0d44f60bde2684e.png
[20201029211730240.png]: /images/20221122/c893b7b8bf9b44ffae67844f878cc6a9.png
[20201029204348432.png]: /images/20221122/568bacaa0e204be394564751cb4ec519.png
[20201029212834356.png]: /images/20221122/7c0ea5a2799d495987c4f0e0bc6bfb68.png
[20201029213119846.png]: /images/20221122/21ecfaf94bbe42a7b262d2a2cbacd968.png
[2020103107243283.png]: /images/20221122/14e7a83e91c64840970f753a65b73303.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 4]: /images/20221122/923f733baa24454b87a17ce26ca2f045.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 5]: /images/20221122/a62a6651f52440cc8eee636be840f714.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 6]: /images/20221122/07415e7d015b4f61bcb4b71ce22489c1.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 7]: /images/20221122/50ae7afaced0437c875a96b1015bff31.png
[20201026212116675.png]: /images/20221122/d364514616104fedae6d652fcef93b67.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 8]: /images/20221122/5d5f90d04d7641f28945517b427dbf8c.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 9]: /images/20221122/a180daca26264a35a299115c5b31b7b6.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 10]: /images/20221122/111667bcd3c1480c94f4aa01327b6d77.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 11]: /images/20221122/4dfd8bcbd3c04561a517a402445d7153.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 12]: /images/20221122/e19e7caf9718471f8e3a4fb96a5a949e.png
[20201031073423764.png]: /images/20221122/3880f2abf67b45e1ad6d7d6ab873de6f.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 13]: /images/20221122/32761abc33b6460395b30d1ee921d2db.png
[20201031074542332.png]: /images/20221122/8cd31a47f4b042a2976d4be48510f013.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 14]: /images/20221122/b665b3b49618464c862a861ebfe84e18.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 15]: /images/20221122/c7cfe72909e845698655f820673e44cb.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 16]: /images/20221122/1907f5819a384901a0c8c17c065e63ec.png
[20201031105057657.png]: /images/20221122/7e14e8f4a3364caf85e7c5d69c7b8de9.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 17]: /images/20221122/1b2a9564be01440d91d3828c52eea85d.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 18]: /images/20221122/df6c22945d114e118df4945d50177fbb.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 19]: /images/20221122/0f947e6ddfbe4cd9a3f6ab644c1afaac.png