万字整理，全网最详细Fate集群配置总结

小咪咪 2022-08-30 05:18 322阅读 0赞

# **一、准备** #

## **1、硬件** ##

准备一台至少12核CPU、24G内存、1T硬盘（最好SSD）和一个不低于10MB能上互联网的网络。（最好能满足上面的条件，要不然即使安装的没有问题，也可能跑不起来）。

## **2    软件** ##

Xshell 、VMware Workstation 16 Pro，这两款软件的下载地址： [https://pan.baidu.com/s/1j6j4n4TMlxxvtSFb-r\_G5A][https_pan.baidu.com_s_1j6j4n4TMlxxvtSFb-r_G5A]  提取码：52bv

Centos，下载地址：

[http://vault.centos.org/7.5.1804/isos/x86\_64/CentOS-7-x86\_64-DVD-1804.iso][http_vault.centos.org_7.5.1804_isos_x86_64_CentOS-7-x86_64-DVD-1804.iso]

# **二、安装 VMWare虚拟机安装** #

从我给的网盘地址将VMWare软件下载下来安装即可，自行激活，如果有问题问度娘。

安装完成后，点击“编辑→虚拟网络编辑器”，如下图：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70][]

查找网关

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 1][]

选择VMnet8，然后点击NAT设置。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 2][]

在这里找到网关：192.168.65.2 记住这个网关，后面配置网络的时候要用到。

# **三、CentOS安装** #

打开vwmare软件，点击创建新的虚拟机

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 3][]

选择“典型（推荐）“，然后选择”下一步“。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 4][]

选择“下一步“

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 5][]

如下图，选择“Linux“、”Centos7 64位“、然后选择”下一步“

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 6][]

给虚拟机命名，第一台命名为“fate01”,第二台虚拟机命名为“fate02”。

然后选择安装路径，安装在空间较大的盘符。然后选择“下一步”

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 7][]

最大磁盘大小设置为“500G”，选择“将磁盘拆分成多个文件”，然后选择“下一步”.

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 8][]

选择“自定义硬件”，如下图：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 9][]

然后设置内存为8G，处理器数量设置为1，每个处理器的内核数量设置为4，在“新CD/DCD(IDE)”选项中设置“CentOS-7-x86\_64-DVD-1804.iso”文件的路径，完成上面的设置后选择关闭。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 10][]![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 11][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 12][]

然后我们就能看到虚拟机的配置，如果没有问题，则选择”完成”，等待虚拟机安装结束。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 13][]

出现下图则说明安装完成，选择“开启此虚拟机”，你就可以看到Centos7的安装页面。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 14][]

选择“Install CentOS 7”，然后按“Enter”键。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 15][]

出下面如下画面，选择“中文”，在右侧选择“简体中文（中国）”，然后选择“继续”。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 16][]

选择“键盘”，点击“+”，选择“英语（英国）”，然后点击“添加”，再选择“完成”。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 17][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 18][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 19][]

接下来，点击“软件选择”。在左侧选择“最小安装”，在右侧选择“调试工具”和”系统管理工具”。然后选择“完成”。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 20][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 21][]

设置“安装位置”，然后选择“我要配置分区”，选择“完成”。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 22][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 23][]

然后出现如下画面，选择“点这里自动创建他们”

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 24][]

选择“home”，设置期望容量为100GiB，然后点击“+”

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 25][]

添加新的挂载点“/data”,容量为“341.12GiB”，然后点击“添加挂载点”。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 26][]

然后点击“完成”，选择“接受更改”

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 27][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 28][]

点击“网路和主机名”

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 29][]

以太网，选择“打开”，设置主机名为“fate01”，点击“应用”，然后选择“配置”。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 30][]

点击“IPv4设置”，方法设置为“手动”，点击“Add”，增加IP地址，如下图，ip地址为“192.168.65.161”，另一台的ip地址设置为192.168.65.162，子网掩码和网关相同都，分别设置为“24”和“192.168.65.2”（这个网关我们在前面提到过）。DNS服务器设置为“192.168.65.2”，勾选“需要IPv4地址完成这个连接”。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 31][]

点击”IPv6设置”，方法选择“忽略”。然后选择“保存”。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 32][]

然后观察IP设置是否正确，如果没有问题则选择“完成”。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 33][]

完成上面的设置后，选择“开始安装”。然后设置root密码。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 34][]

将密码设置为“123456”，然后点击“完成”，等待安装完成。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 35][]

安装完成后，点击“重启”。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 36][]

重启之后，出现如下画面，输入“root”，密码“123456”登录root账户。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 37][]

另一台虚拟机的安装步骤和第一台一样只是名字是“fate02”，IP地址为“192.168.65.162”。

安装完成后，安装Xshell，用Xshell登录。个人认为用xshell执行命令方便一些。

# **四、Fate AllinOne部署集群** #

## 1.集群规划 ##

##  ##

<table> 
 <thead> 
 <tr> 
 <td style="background-color:#f8f8f8;border-color:#dfe2e5;"> party </td> 
 <td style="background-color:#f8f8f8;"> 主机名 </td> 
 <td style="background-color:#f8f8f8;"> IP地址 </td> 
 <td style="background-color:#f8f8f8;"> 操作系统 </td> 
 <td style="background-color:#f8f8f8;"> 安装软件 </td> 
 <td style="background-color:#f8f8f8;"> 服务 </td> 
 </tr> 
 </thead> 
 <tbody> 
 <tr> 
 <td style="background-color:#ffffff;border-color:#dfe2e5;"> PartyA </td> 
 <td style="background-color:#ffffff;border-color:#dfe2e5;"> VM_0_1_centos </td> 
 <td style="background-color:#ffffff;border-color:#dfe2e5;"> 192.168.65.161 </td> 
 <td style="background-color:#ffffff;border-color:#dfe2e5;"> CentOS 7.2/ </td> 
 <td style="background-color:#ffffff;border-color:#dfe2e5;"> fate,eggroll,mysql </td> 
 <td style="background-color:#ffffff;border-color:#dfe2e5;"> fate_flow，fateboard，clustermanager，nodemanager，rollsite，mysql </td> 
 </tr> 
 <tr> 
 <td style="background-color:#f8f8f8;border-color:#dfe2e5;"> PartyB </td> 
 <td style="background-color:#f8f8f8;"> VM_0_2_centos </td> 
 <td style="background-color:#f8f8f8;"> 192.168.65.162 </td> 
 <td style="background-color:#f8f8f8;"> CentOS 7.2 </td> 
 <td style="background-color:#f8f8f8;"> fate,eggroll,mysql </td> 
 <td style="background-color:#f8f8f8;"> fate_flow，fateboard，clustermanager，nodemanager，rollsite，m </td> 
 </tr> 
 </tbody> 
</table>

## 2.架构图 ##

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 38][]

## 3.组件说明 ##

<table> 
 <thead> 
 <tr> 
 <td style="background-color:#f8f8f8;border-color:#dfe2e5;"> 软件产品 </td> 
 <td style="background-color:#f8f8f8;"> 组件 </td> 
 <td style="background-color:#f8f8f8;"> 端口 </td> 
 <td style="background-color:#f8f8f8;"> 说明 </td> 
 </tr> 
 </thead> 
 <tbody> 
 <tr> 
 <td style="border-color:#dfe2e5;"> fate </td> 
 <td style="border-color:#dfe2e5;"> fate_flow </td> 
 <td style="border-color:#dfe2e5;"> 9360;9380 </td> 
 <td style="border-color:#dfe2e5;"> 联合学习任务流水线管理模块 </td> 
 </tr> 
 <tr> 
 <td style="background-color:#f8f8f8;border-color:#dfe2e5;"> fate </td> 
 <td style="background-color:#f8f8f8;"> fateboard </td> 
 <td style="background-color:#f8f8f8;"> 8080 </td> 
 <td style="background-color:#f8f8f8;"> 联合学习过程可视化模块 </td> 
 </tr> 
 <tr> 
 <td style="border-color:#dfe2e5;"> eggroll </td> 
 <td> clustermanager </td> 
 <td> 4670 </td> 
 <td> cluster manager管理集群 </td> 
 </tr> 
 <tr> 
 <td style="background-color:#f8f8f8;border-color:#dfe2e5;"> eggroll </td> 
 <td style="background-color:#f8f8f8;"> nodemanager </td> 
 <td style="background-color:#f8f8f8;"> 4671 </td> 
 <td style="background-color:#f8f8f8;"> node manager管理每台机器资源 </td> 
 </tr> 
 <tr> 
 <td style="border-color:#dfe2e5;"> eggroll </td> 
 <td> rollsite </td> 
 <td> 9370 </td> 
 <td> 跨站点或者说跨party通讯组件，相当于以前版本的proxy+federation </td> 
 </tr> 
 <tr> 
 <td style="background-color:#f8f8f8;border-color:#dfe2e5;"> mysql </td> 
 <td style="background-color:#f8f8f8;"> mysql </td> 
 <td style="background-color:#f8f8f8;"> 3306 </td> 
 <td style="background-color:#f8f8f8;"> 数据存储，clustermanager和fateflow依赖 </td> 
 </tr> 
 </tbody> 
</table>

## 4、环境配置 ##

### 4.1 hostname配置 ###

** 1）修改主机名**

在192.168.65.161 root用户下执行：

hostnamectl set-hostname VM\_0\_1\_centos

在192.168.65.162 root用户下执行：

hostnamectl set-hostname VM\_0\_2\_centos

**  2）加入主机映射**

在目标服务器（192.168.65.161 192.168.65.162）root用户下执行：

vi /etc/hosts

添加内容

192.168.0.1 VM_0_1_centos
    192.168.0.2 VM_0_2_centos

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 39][]

添加完成后，执行"su root"命令重新登录。

### 4.2 关闭selinux ###

**在目标服务器（192.168.65.161 192.168.65.162）root用户下执行：**

确认是否已安装selinux

centos系统执行：rpm -qa | grep selinux

ubuntu系统执行：apt list --installed | grep selinux

如果已安装了selinux就执行：setenforce 0

![20210719142605421.png][]

### 4.3 配置yum仓库 ###

**在目标服务器（192.168.65.161 192.168.65.162）root用户下执行：**

**1）备份**CentOS-Base.repo文件

mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.bak

![20210719142900432.png][]

**2）修改yum仓库配置**

vi /etc/yum.repos.d/CentOS-Base.repo

添加内容：

# CentOS-Base.repo
    #
    # The mirror system uses the connecting IP address of the client and the
    # update status of each mirror to pick mirrors that are updated to and
    # geographically close to the client.  You should use this for CentOS updates
    # unless you are manually picking other mirrors.
    #
    # If the mirrorlist= does not work for you, as a fall back you can try the
    # remarked out baseurl= line instead.
    #
    #
    
    [base]
    name=CentOS-$releasever - Base
    baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos/$releasever/os/$basearch/
    #mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=os
    gpgcheck=1
    gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
    
    #released updates
    [updates]
    name=CentOS-$releasever - Updates
    baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos/$releasever/updates/$basearch/
    #mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=updates
    gpgcheck=1
    gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
    
    #additional packages that may be useful
    [extras]
    name=CentOS-$releasever - Extras
    baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos/$releasever/extras/$basearch/
    #mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=extras
    gpgcheck=1
    gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
    
    #additional packages that extend functionality of existing packages
    [centosplus]
    name=CentOS-$releasever - Plus
    baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos/$releasever/centosplus/$basearch/
    #mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=centosplus
    gpgcheck=1
    enabled=0
    gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

**3）使新仓库配置生效**

yum clean all
    yum makecache

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 40][]

### 4.4 修改Linux系统参数 ###

在目标服务器（192.168.65.161 192.168.65.162）root用户下执行：

**1）清理20-nproc.conf文件**

cd /etc/security/limits.d
    ls -lrt 20-nproc.conf

存在则：

mv 20-nproc.conf 20-nproc.conf_bak

![20210719145709158.png][]

**2）修改limits.conf文件**

vi /etc/security/limits.conf

添加下面的信息：

* soft nofile 65535
    * hard nofile 65535
    * soft nproc 65535
    * hard nproc 65535

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 41][]

重新登陆，ulimit -a查看是否生效

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 42][]

###  4.5 **修改系统mysql配置** ###

mv /etc/my.cnf /etc/my.cnf\_bak

![20210719150459550.png][]

### 4.6 关闭防火墙(可选) ###

**在目标服务器（192.168.65.161 192.168.65.162）root用户下执行**

systemctl disable firewalld.service
    systemctl stop firewalld.service
    systemctl status firewalld.service

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 43][]

### 4.7 创建APP用户 ###

**在目标服务器（192.168.65.161 192.168.65.162）root用户下执行**

groupadd -g 6000 apps
    useradd -s /bin/bash -g apps -d /home/app app
    passwd app

app初识密码设置为123456

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 44][]

### **4.8  给APP用户配置sudo** ###

**在目标服务器（192.168.65.161 192.168.65.162）root用户下执行**

vi /etc/sudoers.d/app

添加内容：

app ALL=(ALL) ALL
    app ALL=(ALL) NOPASSWD: ALL
    Defaults !env_reset

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 45][]

给文件sudoers文件增加写入权限

chmod u+w /etc/sudoers

编辑 sudoers文件

vi /etc/sudoers

在root    ALL=(ALL)       ALL下面追加：

root    ALL=(ALL)       NOPASSWD: ALL
    app     ALL=(ALL)       ALL
    app     ALL=(ALL)       NOPASSWD: ALL

给文件sudoers文件删除写入权限

chmod u-w /etc/sudoers

![20210719152604690.png][]

### 4.9 增加虚拟内存 ###

**在目标服务器（192.168.65.161 192.168.65.162）root用户下执行**

生产环境使用时，因内存计算需要增加128G虚拟内存，执行前需检查存储空间是否足够。

手工创建，root用户执行：

cd /data
    dd if=/dev/zero of=/data/swapfile128G bs=1024 count=134217728
    mkswap /data/swapfile128G
    swapon /data/swapfile128G
    cat /proc/swaps
    echo '/data/swapfile128G swap swap defaults 0 0' >> /etc/fstab

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 46][]

### **4.10  设置projects目录** ###

**在目标服务器（192.168.65.161 192.168.65.162）root用户下执行**

cd /data
    mkdir projects
    chown app:apps projects
    ls -l
    chown -R app /data

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 47][]

### 4.11 **配置ssh无密登录** ###

**1）配置自身免密**

**在目标服务器（192.168.65.161 192.168.65.162）app用户下执行**

在虚拟机VM\_0\_1\_centos 上以app用户身份在home目录下执行

su app
    ssh-keygen -t rsa
    cat ~/.ssh/id_rsa.pub >>/home/app/.ssh/authorized_keys
    chmod 600 ~/.ssh/authorized_keys

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 48][]

使用ssh 192.168.65.161命令测试，第一次需要确认yes ,之后应可以不需密码登录为成功。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 49][]

在VM\_0\_2\_centos 上执行相同步骤配置VM\_0\_2\_centos 的自身免密登录。

**2）配置VM\_0\_1\_centos 免密登录VM\_0\_2\_centos**

在**VM\_0\_1\_centos**  app用户下执行

scp ~/.ssh/authorized_keys app@192.168.65.162:/home/app/.ssh

输入**VM\_0\_2\_centos**的app密码

在**VM\_0\_2\_centos **app用户下执行

cat ~/.ssh/id_rsa.pub >> /home/app/.ssh/authorized_keys
    
    scp ~/.ssh/authorized_keys app@192.168.65.161:/home/app/.ssh

输入**VM\_0\_1\_centos **的app密码完成文件传输。

在**VM\_0\_1\_centos **app用户下执行

ssh 192.168.65.162

应不需要密码即可登录fate02机。

到这里我们已经完成环境的配置了。

# 五.项目部署 #

## 5.1 获取项目 ##

在目标服务器（192.168.65.161 具备外网环境）app用户下执行

进入执行节点的/data/projects/目录，执行：

cd /data/projects/
    wget https://webank-ai-1251170195.cos.ap-guangzhou.myqcloud.com/fate_cluster_install_1.6.0_release-c7-u18.tar.gz
    tar xzf fate_cluster_install_1.6.0_release-c7-u18.tar.gz

## 5.2 部署前检查 ##

**在目标服务器（192.168.65.161 192.168.65.162 ）app用户下执行**

把检查脚本fate-cluster-install/tools-install/check.sh从192.168.65.161拷贝到192.168.65.162

#在192.168.0.1和192.168.0.2服务器上分别执行检查脚本
    sh ./check.sh
    
    #确认app用户已配置sudo
    #虚拟内存，size不低于128G，如不满足需参考4.6章节重新设置
    #文件句柄数，不低于65535，如不满足需参考4.3章节重新设置
    #用户进程数，不低于64000，如不满足需参考4.3章节重新设置
    #确认部署前没有fate进程和端口冲突
    #确认/etc/my.cnf是否存在，存在需要mv；确认是否存在/data/projects/fate目录，存在需把fate目录mv备份。

## 5.3 配置文件修改和示例 ##

**在目标服务器（192.168.65.161）app用户下执行**

修改配置文件fate-cluster-install/allInone/conf/setup.conf.

vi fate-cluster-install/allInone/conf/setup.conf

配置文件setup.conf说明：

<table> 
 <thead> 
 <tr> 
 <th>配置项</th> 
 <th>配置项值</th> 
 <th>说明</th> 
 </tr> 
 </thead> 
 <tbody> 
 <tr> 
 <td>roles</td> 
 <td>默认："host" "guest"</td> 
 <td>部署的角色，有HOST端、GUEST端</td> 
 </tr> 
 <tr> 
 <td>version</td> 
 <td>默认：1.5.1</td> 
 <td>Fate 版本号</td> 
 </tr> 
 <tr> 
 <td>pbase</td> 
 <td>默认： /data/projects</td> 
 <td>项目根目录</td> 
 </tr> 
 <tr> 
 <td>lbase</td> 
 <td>默认：/data/logs</td> 
 <td>保持默认不要修改</td> 
 </tr> 
 <tr> 
 <td>ssh_user</td> 
 <td>默认：app</td> 
 <td>ssh连接目标机器的用户，也是部署后文件的属主</td> 
 </tr> 
 <tr> 
 <td>ssh_group</td> 
 <td>默认：apps</td> 
 <td>ssh连接目标的用户的属组，也是部署后文件的属组</td> 
 </tr> 
 <tr> 
 <td>ssh_port</td> 
 <td>默认：22,根据实际情况修改</td> 
 <td>ssh连接端口，部署前确认好端口，不然会报连接错误</td> 
 </tr> 
 <tr> 
 <td>eggroll_dbname</td> 
 <td>默认：eggroll_meta</td> 
 <td>eggroll连接的DB名字</td> 
 </tr> 
 <tr> 
 <td>fate_flow_dbname</td> 
 <td>默认：fate_flow</td> 
 <td>fate_flow、fateboard等连接的DB名字</td> 
 </tr> 
 <tr> 
 <td>mysql_admin_pass</td> 
 <td>默认</td> 
 <td>mysql的管理员（root）密码</td> 
 </tr> 
 <tr> 
 <td>redis_pass</td> 
 <td>默认</td> 
 <td>redis密码，暂未使用</td> 
 </tr> 
 <tr> 
 <td>mysql_user</td> 
 <td>默认：fate</td> 
 <td>msyql的应用连接账号</td> 
 </tr> 
 <tr> 
 <td>mysql_port</td> 
 <td>默认：3306，根据实际情况修改</td> 
 <td>msql服务监听的端口</td> 
 </tr> 
 <tr> 
 <td>host_id</td> 
 <td>默认 : 10000，根据实施规划修改</td> 
 <td>HOST端的party id。</td> 
 </tr> 
 <tr> 
 <td>host_ip</td> 
 <td>192.168.0.1</td> 
 <td>HOST端的ip</td> 
 </tr> 
 <tr> 
 <td>host_mysql_ip</td> 
 <td>默认和host_ip保持一致</td> 
 <td>HOST端mysql的ip</td> 
 </tr> 
 <tr> 
 <td>host_mysql_pass</td> 
 <td>默认</td> 
 <td>HOST端msyql的应用连接账号</td> 
 </tr> 
 <tr> 
 <td>guest_id</td> 
 <td>默认 : 9999，根据实施规划修改</td> 
 <td>GUEST端的party id</td> 
 </tr> 
 <tr> 
 <td>guest_ip</td> 
 <td>192.168.0.2</td> 
 <td>GUEST端的ip</td> 
 </tr> 
 <tr> 
 <td>guest_mysql_ip</td> 
 <td>默认和guest_ip保持一致</td> 
 <td>GUEST端mysql的ip</td> 
 </tr> 
 <tr> 
 <td>guest_mysql_pass</td> 
 <td>默认</td> 
 <td>GUEST端msyql的应用连接账号</td> 
 </tr> 
 <tr> 
 <td>dbmodules</td> 
 <td>默认："mysql"</td> 
 <td>DB组件的部署模块列表，如mysql</td> 
 </tr> 
 <tr> 
 <td>basemodules</td> 
 <td>默认："tools" "base" "java" "python" "eggroll" "fate"</td> 
 <td>非DB组件的部署模块列表，如 "tools" "base"、 "java"、 "python" 、"eggroll" 、"fate"</td> 
 </tr> 
 <tr> 
 <td>fateflow_grpc_port</td> 
 <td>默认：9360</td> 
 <td>fateflow grpc服务端口</td> 
 </tr> 
 <tr> 
 <td>fateflow_http_port</td> 
 <td>默认：9380</td> 
 <td>fateflow http服务端口</td> 
 </tr> 
 <tr> 
 <td>fateboard_port</td> 
 <td>默认：8080</td> 
 <td>fateboard服务端口</td> 
 </tr> 
 <tr> 
 <td>rollsite_port</td> 
 <td>默认：9370</td> 
 <td>rollsite服务端口</td> 
 </tr> 
 <tr> 
 <td>clustermanager_port</td> 
 <td>默认：4670</td> 
 <td>clustermanager服务端口</td> 
 </tr> 
 <tr> 
 <td>nodemanager_port</td> 
 <td>默认：4671</td> 
 <td>nodemanager服务端口</td> 
 </tr> 
 </tbody> 
</table>

**1）两台主机partyA+partyB同时部署**

#to install role
    roles=( "host" "guest" )
    
    version="1.6.0"
    #project base
    pbase="/data/projects"
    #log directory
    lbase="/data/logs"
    
    #user who connects dest machine by ssh
    ssh_user="app"
    ssh_group="apps"
    #ssh port
    ssh_port=22
    
    #eggroll_db name
    eggroll_dbname="eggroll_meta"
    #fate_flow_db name
    fate_flow_dbname="fate_flow"
    
    #mysql init root password
    mysql_admin_pass="fate_dev"
    
    #redis passwd
    redis_pass=""
    
    #mysql user
    mysql_user="fate"
    #mysql port
    mysql_port="3306"
    
    #host party id
    host_id="10000"
    #host ip
    host_ip="192.168.65.161"
    #host mysql ip
    host_mysql_ip="${host_ip}"
    host_mysql_pass="fate_deV2999"
    
    #guest party id
    guest_id="9999"
    #guest ip
    guest_ip="192.168.65.162"
    #guest mysql ip
    guest_mysql_ip="${guest_ip}"
    guest_mysql_pass="fate_deV2999"
    
    #db module lists
    dbmodules=( "mysql" )
    
    #base module lists
    basemodules=( "tools" "base" "java" "python" "eggroll" "fate" )
    
    fateflow_grpc_port=9360
    fateflow_http_port=9380
    fateboard_port=8080
    
    rollsite_port=9370
    clustermanager_port=4670
    nodemanager_port=4671

**2）只部署一个party**

#to install role
    roles=( "host" )
    
    version="1.6.0"
    #project base
    pbase="/data/projects"
    #log directory
    lbase="/data/logs"
    
    #user who connects dest machine by ssh
    ssh_user="app"
    ssh_group="apps"
    #ssh port
    ssh_port=22
    
    #eggroll_db name
    eggroll_dbname="eggroll_meta"
    #fate_flow_db name
    fate_flow_dbname="fate_flow"
    
    #mysql init root password
    mysql_admin_pass="fate_dev"
    
    #redis passwd
    redis_pass=""
    
    #mysql user
    mysql_user="fate"
    #mysql port
    mysql_port="3306"
    
    #host party id
    host_id="10000"
    #host ip
    host_ip="192.168.65.161"
    #host mysql ip
    host_mysql_ip="${host_ip}"
    host_mysql_pass="fate_deV2999"
    
    #guest party id
    guest_id=""
    #guest ip
    guest_ip=""
    #guest mysql ip
    guest_mysql_ip="${guest_ip}"
    guest_mysql_pass=""
    
    #db module lists
    dbmodules=( "mysql" )
    
    #base module lists
    basemodules=( "tools" "base" "java" "python" "eggroll" "fate" )
    
    fateflow_grpc_port=9360
    fateflow_http_port=9380
    fateboard_port=8080
    
    rollsite_port=9370
    clustermanager_port=4670
    nodemanager_port=4671

## 5.4 部署 ##

按照上述配置含义修改setup.conf文件对应的配置项后，然后在fate-cluster-install/allInone目录下执行部署脚本：

cd fate-cluster-install/allInone
    nohup sh ./deploy.sh > logs/boot.log 2>&1 &

部署日志输出在fate-cluster-install/allInone/logs目录下,实时查看是否有报错：

tail -f ./logs/deploy.log （部署结束，查看一下即可）
    tail -f ./logs/deploy-guest.log （实时打印GUEST端的部署情况）
    tail -f ./logs/deploy-mysql-guest.log  （实时打印GUEST端mysql的部署情况）
    tail -f ./logs/deploy-host.log    （实时打印HOST端的部署情况）
    tail -f ./logs/deploy-mysql-host.log    （实时打印HOST端mysql的部署情况）

deploy.log出现以下内容说明安装完成

init over  
deploy guest mysql  172.16.132.190  over  
deploy guest 172.16.132.190  over  
deploy host mysql 172.16.132.189  over  
deploy host 172.16.132.189  over

## 5.5 问题定位 ##

1）eggroll日志

/data/projects/fate/eggroll/logs/eggroll/bootstrap.clustermanager.err

/data/projects/fate/eggroll/logs/eggroll/clustermanager.jvm.err.log

/data/projects/fate/eggroll/logs/eggroll/nodemanager.jvm.err.log

/data/projects/fate/eggroll/logs/eggroll/bootstrap.nodemanager.err

/data/projects/fate/eggroll/logs/eggroll/bootstrap.rollsite.err

/data/projects/fate/eggroll/logs/eggroll/rollsite.jvm.err.log

2）fateflow日志

/data/projects/fate/python/logs/fate\_flow/

3）fateboard日志

/data/projects/fate/fateboard/logs

# 6.测试 #

## 6.1 Toy\_example部署验证 ##

此测试您需要设置3个参数：guest\_partyid，host\_partyid，work\_mode。

### 6.1.1 单边测试 ###

1）192.168.65.161上执行，guest\_partyid和host\_partyid都设为10000：

source /data/projects/fate/bin/init_env.sh
    cd /data/projects/fate/examples/toy_example/
    python run_toy_example.py 10000 10000 1

注：执行source /data/projects/fate/bin/init\_env.sh命令时，会进入一个虚拟环境，但是很多人不知道怎么退出，退出的命令是：

deactivate

类似如下结果表示成功：

"2020-04-28 18:26:20,789 - secure_add_guest.py[line:126] - INFO: success to calculate secure_sum, it is 1999.9999999999998"

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 50][]

提示：如出现max cores per job is 1, please modify job parameters报错提示，需要修改当前目录下文件toy\_example\_conf.json中参数task\_cores为1.

2）192.168.65.162上执行，guest\_partyid和host\_partyid都设为9999：

source /data/projects/fate/bin/init_env.sh
    cd /data/projects/fate/examples/toy_example/
    python run_toy_example.py 9999 9999 1

类似如下结果表示成功：

"2020-04-28 18:26:20,789 - secure\_add\_guest.py\[line:126\] - INFO: success to calculate secure\_sum, it is 1999.9999999999998"

### 6.1.2 双边测试 ###

选定9999为guest方，在192.168.65.162上执行：

source /data/projects/fate/bin/init_env.sh
    cd /data/projects/fate/examples/toy_example/
    python run_toy_example.py 9999 10000 1

类似如下结果表示成功：

"2020-04-28 18:26:20,789 - secure\_add\_guest.py\[line:126\] - INFO: success to calculate secure\_sum, it is 1999.9999999999998"

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 51][]

## 6.2 最小化测试 ##

### **6.2.1 上传预设数据：** ###

分别在192.168.65.161和192.168.65.162上执行：

source /data/projects/fate/bin/init_env.sh
    cd /data/projects/fate/examples/scripts/
    python upload_default_data.py -m 1

### **6.2.2 快速模式：** ###

请确保guest和host两方均已分别通过给定脚本上传了预设数据。

快速模式下，最小化测试脚本将使用一个相对较小的数据集，即包含了569条数据的breast数据集。

选定9999为guest方，在192.168.65.162上执行：

source /data/projects/fate/bin/init_env.sh
    cd /data/projects/fate/examples/min_test_task/
    #单边测试
    python run_task.py -m 1 -gid 9999 -hid 9999 -aid 9999 -f fast
    #双边测试
    python run_task.py -m 1 -gid 9999 -hid 10000 -aid 10000 -f fast

其他一些可能有用的参数包括：

1.  \-f: 使用的文件类型. "fast" 代表 breast数据集, "normal" 代表 default credit 数据集.
2.  \--add\_sbt: 如果被设置为1, 将在运行完lr以后，启动secureboost任务，设置为0则不启动secureboost任务，不设置此参数系统默认为1。

若数分钟后在结果中显示了“success”字样则表明该操作已经运行成功了。若出现“FAILED”或者程序卡住，则意味着测试失败。

### **6.2.3 正常模式**： ###

只需在命令中将“fast”替换为“normal”，其余部分与快速模式相同。

## 6.3 Fateboard testing ##

Fateboard是一项Web服务。如果成功启动了fateboard服务，则可以通过访问 [http://192.168.65.161:8080][http_192.168.65.161_8080] 和 [http://192.168.65.162:8080][http_192.168.65.162_8080] 来查看任务信息，如果有防火墙需开通。

![2021071916312217.png][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 52][]

# 7.系统运维 #

## 7.1 服务管理 ##

**在目标服务器（192.168.65.161 192.168.65.162）app用户下执行**

### 7.1.1 Eggroll服务管理 ###

source /data/projects/fate/bin/init_env.sh
    cd /data/projects/fate/eggroll

启动/关闭/查看/重启所有：

sh ./bin/eggroll.sh all start/stop/status/restart

启动/关闭/查看/重启单个模块(可选：clustermanager，nodemanager，rollsite)：

sh ./bin/eggroll.sh clustermanager start/stop/status/restart

### 7.1.2 Mysql服务管理 ###

启动/关闭/查看/重启mysql服务

cd /data/projects/fate/common/mysql/mysql-8.0.13
    sh ./service.sh start|stop|status|restart

### 7.1.3 Fate服务管理 ###

1) 启动/关闭/查看/重启fate\_flow服务

source /data/projects/fate/bin/init_env.sh
    cd /data/projects/fate/python/fate_flow
    sh service.sh start|stop|status|restart

如果逐个模块启动，需要先启动eggroll和mysql再启动fateflow，fateflow依赖eggroll的启动。

2) 启动/关闭/重启fateboard服务

cd /data/projects/fate/fateboard
    sh service.sh start|stop|status|restart

## 7.2 查看进程和端口 ##

**在目标服务器（192.168.65.161 192.168.65.162 ）app用户下执行**

### 7.2.1 查看进程 ###

### 7.2.2 查看进程端口 ###

## 7.3 服务日志 ##

<table> 
 <thead> 
 <tr> 
 <th>服务</th> 
 <th>日志路径</th> 
 </tr> 
 </thead> 
 <tbody> 
 <tr> 
 <td>eggroll</td> 
 <td>/data/projects/fate/eggroll/logs</td> 
 </tr> 
 <tr> 
 <td>fate_flow&amp;任务日志</td> 
 <td>/data/projects/fate/python/logs</td> 
 </tr> 
 <tr> 
 <td>fateboard</td> 
 <td>/data/projects/fate/fateboard/logs</td> 
 </tr> 
 <tr> 
 <td>mysql</td> 
 <td>/data/logs/mysql/</td> 
 </tr> 
 </tbody> 
</table>

# 8. 附录 #

[https_pan.baidu.com_s_1j6j4n4TMlxxvtSFb-r_G5A]: https://pan.baidu.com/s/1j6j4n4TMlxxvtSFb-r_G5A
[http_vault.centos.org_7.5.1804_isos_x86_64_CentOS-7-x86_64-DVD-1804.iso]: http://vault.centos.org/7.5.1804/isos/x86_64/CentOS-7-x86_64-DVD-1804.iso
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70]: /images/20220829/abbf67e874b542a8b09c728962d109d8.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 1]: /images/20220829/951fc779a5ff4c40bbcb832f220dea30.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 2]: /images/20220829/1cf49d2cc6a94a10adb27ca8ccf3e136.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 3]: /images/20220829/e7caa3bee9994a3aacf2e2479c4a2d01.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 4]: /images/20220829/d0ecd0198df746c4bc15073e42db94e4.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 5]: /images/20220829/eddd275d415f4661890628c2c687c630.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 6]: /images/20220829/ad9fee5e4ed3472db8ca02073274ba90.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 7]: /images/20220829/4883dd93c1d04f92aec41b87b78380c8.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 8]: /images/20220829/c297729cd01a44cdb960c5beeae76b5c.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 9]: /images/20220829/d62c6c58aec44730a4f314729848d087.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 10]: https://img-blog.csdnimg.cn/20210719140543630.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 11]: /images/20220829/458f56f6c0f3481b9530eec40a1a020a.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 12]: /images/20220829/be90de2d79d94294ae4a6198e4b8e49e.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 13]: /images/20220829/1854b9bea1b648ce9d92c1a54d85324e.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 14]: /images/20220829/71b9e19c20af485da3695d8f174830cd.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 15]: /images/20220829/2e380006008d4433b387e12c85042902.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 16]: /images/20220829/9b1914705840477a8052e9781b2e8109.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 17]: /images/20220829/b9ea7acb9c074dd8bb5e744855c09725.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 18]: /images/20220829/f65d91b93bbf4d918bac4c87f7e69449.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 19]: /images/20220829/1ff9233844d44aee8f5314560d9adb46.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 20]: /images/20220829/cf63867c90ad4ebaaa671b9dc0644803.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 21]: /images/20220829/ec25537dbc1b4d6db9f04b81f68a6d89.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 22]: /images/20220829/ea1e972630344e62ad3d70ce05d38c75.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 23]: /images/20220829/da5fd1c8726c4392981f08d35840734c.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 24]: /images/20220829/637d022d3c3e478681547e739c422180.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 25]: /images/20220829/4d4cbf043c7349ce914087f2db5db863.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 26]: /images/20220829/b25901c4aa9d44028b98c1bf3b7edc49.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 27]: /images/20220829/0fba95378c654f3ca462e36f2d5b0c83.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 28]: /images/20220829/dc5fe577727c4888bbf4f1a5df6316e2.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 29]: /images/20220829/4df0f0a3849d405d9fcf0bb3de3e9d1f.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 30]: /images/20220829/7a40f0d41b3d4772b813739fcd6c6d71.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 31]: /images/20220829/fe698787cc0642d19b4e297b33e74ba8.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 32]: /images/20220829/59d36bc757c74523abfc0b625bfd6e7a.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 33]: /images/20220829/e04cf32d95104c01a58dd3befd4f8df6.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 34]: /images/20220829/24ac6e3e5a5e4dabafb000957b6e297f.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 35]: /images/20220829/f810a4c1001543e6acae687bad6df979.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 36]: /images/20220829/376f11502f144e2788782ac15d3069fd.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 37]: /images/20220829/56d0196f605740bbb8abec478f4917ff.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 38]: /images/20220829/be0a9b08cfa54de2a9d036ca1faa80c4.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 39]: /images/20220829/89c9e799345f480d8354778eec17e67d.png
[20210719142605421.png]: /images/20220829/6d28f038e49241fbbaf4ee15a52755aa.png
[20210719142900432.png]: /images/20220829/5d75efe3c9f24abab37bc060155c7ba3.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 40]: /images/20220829/b4417ff2e89649beaca1b76d65d04875.png
[20210719145709158.png]: /images/20220829/a212098c66794f1daf430fcbaeaead67.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 41]: /images/20220829/b3833800abc84fd0833338116a60619a.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 42]: /images/20220829/3207371a3bf840988f50058389fe0755.png
[20210719150459550.png]: /images/20220829/e56e667de8464f94be696690bd1f88fd.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 43]: /images/20220829/d75ddbcd03cb4a34bb9ec075312959bd.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 44]: /images/20220829/79fc9c547f7c43debd65244162c31429.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 45]: /images/20220829/76c77b26096d4f9297a1bee316b13c17.png
[20210719152604690.png]: /images/20220829/fd632807333c4206a12aedba9e3aefaf.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 46]: /images/20220829/e872c12c781146d6b37e454b3a188d3e.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 47]: /images/20220829/c6bf56f1b8e446f3bb8a9b0e4c728711.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 48]: /images/20220829/922d1e3da2bb405989d2ac52e0bcbfbb.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 49]: /images/20220829/a316a59c1c5e4dfeb4596bf7e2a8cf23.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 50]: /images/20220829/ae56f6d137a94fa8961bfb0862409478.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 51]: /images/20220829/6f2228f0f39545aa9c42a6ac036e698a.png
[http_192.168.65.161_8080]: http://192.168.0.1:8080
[http_192.168.65.162_8080]: http://192.168.0.2:8080
[2021071916312217.png]: /images/20220829/bc38b50a913443f5ba99ecdf816aa812.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 52]: /images/20220829/29f184f2597d4399af1803c98571e622.png