docker与k8s部署nacos

た入场券 2023-02-14 10:11 10阅读 0赞

**1 mariadb安装**  
参考[docker安装MariaDB][docker_MariaDB]

docker pull mariadb
    sudo mkdir -p /data/mariadb/data
    # 设置容器的3307端口映射到主机3306端口
    docker run --name mariadb -p 3307:3306 -e MYSQL_ROOT_PASSWORD=000000 -v /data/mariadb/data:/var/lib/mysql -d mariadb
    # 7481d6bdd2de为mariadb的容器id,如下图所示
    docker container update --restart=always 7481d6bdd2de
    # 进入容器
    docker exec -it 7481d6bdd2de bash
    # 容器内登录数据库
    mysql -uroot -p000000

![1][]  
![1][1 1]  
**2 nacos 安装**  
[HashiCorp 企业版本中国禁止使用了][HashiCorp]，  
[Eureka 2.0 (Discontinued)][Eureka 2.0 _Discontinued]，结合现在的环境，还是选择国产开源软件nacos吧。  
**2.1 单机版**  
参考[使用Docker部署Nacos-Server(单机版)][Docker_Nacos-Server],nacos并不会自动创建数据库，故需要从[alibaba/nacos][alibaba_nacos]中下载sql脚本  
如果一开始不指定数据库的字符集，因为脚本里面说明了字符集

```sh
    # 包含prometheus/grafana等监控组件
    git clone --depth 1 https://github.com/nacos-group/nacos-docker.git
    cd nacos-docker
    docker-compose -f example/standalone-derby.yaml up -d 
    # 挂载目录
      #新建logs目录
    mkdir -p /home/test/nacos/logs/ 
    mkdir -p /home/test/nacos/init.d/
     #修改配置文件,这个custom.properties是nacos的公共配置,集群的方式跟这个不一样
    vim /home/test/nacos/init.d/custom.properties

单机版本`custom.properties`如下所示：

server.contextPath=/nacos
    server.servlet.contextPath=/nacos
    server.port=8848
    
    spring.datasource.platform=mysql
    
    db.num=1
    db.url.0=jdbc:mysql://10.101.5.147:3307/nacos_config?characterEncoding=utf8&connectTimeout=1000&socketTimeout=3000&autoReconnect=true
    db.user=root
    db.password=000000
    
    
    nacos.cmdb.dumpTaskInterval=3600
    nacos.cmdb.eventTaskInterval=10
    nacos.cmdb.labelTaskInterval=300
    nacos.cmdb.loadDataAtStart=false
    
    management.metrics.export.elastic.enabled=false
    
    management.metrics.export.influx.enabled=false
    
    
    server.tomcat.accesslog.enabled=true
    server.tomcat.accesslog.pattern=%h %l %u %t "%r" %s %b %D %{User-Agent}i
    
    
    nacos.security.ignore.urls=/,/**/*.css,/**/*.js,/**/*.html,/**/*.map,/**/*.svg,/**/*.png,/**/*.ico,/console-fe/public/**,/v1/auth/login,/v1/console/health/**,/v1/cs/**,/v1/ns/**,/v1/cmdb/**,/actuator/**,/v1/console/server/**
    nacos.naming.distro.taskDispatchThreadCount=1
    nacos.naming.distro.taskDispatchPeriod=200
    nacos.naming.distro.batchSyncKeyCount=1000
    nacos.naming.distro.initDataRatio=0.9
    nacos.naming.distro.syncRetryDelay=5000
    nacos.naming.data.warmup=true
    nacos.naming.expireInstance=true

启动nacos

docker  run \
    --name nacos -d \
    -p 8848:8848 \
    --privileged=true \
    --restart=always \
    -e JVM_XMS=256m \
    -e JVM_XMX=256m \
    -e MODE=standalone \
    -e PREFER_HOST_MODE=hostname \
    -v /home/dengjun/nacos/logs:/home/dengjun/nacos/logs \
    -v /home/dengjun/nacos/init.d/custom.properties:/home/dengjun/nacos/init.d/custom.properties \
    nacos/nacos-server

在执行`docker-compose -f example/standalone-derby.yaml up -d`镜像有时候下载不下来，这个时候考虑更换为国内镜像源

# 没有则创建一个
    sudo vi /etc/docker/daemon.json
    # 添加下面的内容
    {"registry-mirrors": [
            "http://hub-mirror.c.163.com",
            "https://docker.mirrors.ustc.edu.cn",
            "https://pee6w651.mirror.aliyuncs.com",
            "https://registry.docker-cn.com"
    ],
    "dns": ["8.8.8.8","8.8.4.4"]
    }
    # 生效
    sudo systemctl daemon-reload
    sudo systemctl restart docker

systemctl status docker.service
    ~/Downloads/nacos-docker$ sudo service docker restart
    Job for docker.service failed because the control process exited with error code.
    See "systemctl status docker.service" and "journalctl -xe" for details.
    dengjun@bw147:~/Downloads/nacos-docker$ systemctl status docker.service
    ● docker.service - Docker Application Container Engine
       Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
       Active: failed (Result: exit-code) since Tue 2020-06-02 10:55:40 CST; 3s ago
         Docs: https://docs.docker.com
      Process: 14384 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock (code=e
     Main PID: 14384 (code=exited, status=1/FAILURE)
    
    6月 02 10:55:40 bw147 systemd[1]: docker.service: Service hold-off time over, scheduling restart.
    6月 02 10:55:40 bw147 systemd[1]: docker.service: Scheduled restart job, restart counter is at 3.
    6月 02 10:55:40 bw147 systemd[1]: Stopped Docker Application Container Engine.
    6月 02 10:55:40 bw147 systemd[1]: docker.service: Start request repeated too quickly.
    6月 02 10:55:40 bw147 systemd[1]: docker.service: Failed with result 'exit-code'.
    6月 02 10:55:40 bw147 systemd[1]: Failed to start Docker Application Container Engine

按照[解决docker无法启动的问题][docker],执行`cat /etc/default/docker`,最后一行已经是注释掉的了。再仔细检查`/etc/docker/daemon.json`，原来里面的内容错了，更改过来就可以了  
![1][1 2]  
执行`SELECT version()`查看mysql版本号，执行nacos中的脚本，会出现问题，nacos的官方脚本没有问题，为什么会有这个错误呢。因为设置utf8字符集

CREATE TABLE permissions (
        role varchar(50) NOT NULL,
        resource varchar(512) NOT NULL,
        action varchar(8) NOT NULL,
        constraint uk_role_permission UNIQUE (role,resource,action)
    );
    [Err] 1071 - Specified key was too long; max key length is 767 bytes

注意在mysql中执行，貌似没什么用

mysql> set global innodb_file_format = BARRACUDA;
    Query OK, 0 rows affected
    
    mysql> set global innodb_large_prefix = ON;
    Query OK, 0 rows affected

![1][1 3]  
[DOCKER安装NACOS以及配置数据库][DOCKER_NACOS]

docker pull nacos/nacos-server
    docker run --env MODE=standalone --name nacos -d -p 8848:8848 nacos/nacos-server

**2.2 nacos持久化**  
使用nacos帐号登录进入后，新增一条数据，但是进入到`nacos_config`数据库中却找不到记录，这是为什么呢，难道他将数据缓存在什么地方了吗？  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dhcnJhaA_size_16_color_FFFFFF_t_70]  
在数据库直接添加数据，前台页面同样无法显示。  
![1][1 4]  
调研分析可以知道nacos默认是把数据保存在本地磁盘文件读取，那么什么时候往数据库中写呢？  
使用2.1中的命令是数据是写不进数据库的，这里做一下调整，注意ubuntu要执行`sudo ufw disable`关闭防火墙

docker run --name nacos -p 8847:8848 --privileged=true --restart=always --env-file=/home/dengjun/nacos/init.d/env.list -v /home/dengjun/nacos/logs:/home/nacos/logs -v /home/dengjun/nacos/init.d/custom.properties:/home/nacos/init.d/custom.properties -d nacos/nacos-server

这里`env.list`

PREFER_HOST_MODE=hostname
    MODE=standalone
    SPRING_DATASOURCE_PLATFORM=mysql
    MYSQL_SERVICE_HOST=10.101.5.147
    MYSQL_SERVICE_PORT=3307
    MYSQL_SERVICE_USER=root
    MYSQL_SERVICE_PASSWORD=000000
    MYSQL_SERVICE_DB_NAME=nacos_config
    
    JVM_XMS=512m
    JVM_XMX=512m
    JVM_XMN=256m
    JVM_MS=32m
    JVM_MMS=80m

另外`custom.properties`中就不需要配置数据库了

server.contextPath=/nacos
    server.servlet.contextPath=/nacos
    server.port=8848
    
    nacos.cmdb.dumpTaskInterval=3600
    nacos.cmdb.eventTaskInterval=10
    nacos.cmdb.labelTaskInterval=300
    nacos.cmdb.loadDataAtStart=false
    
    management.metrics.export.elastic.enabled=false
    
    management.metrics.export.influx.enabled=false
    
    server.tomcat.accesslog.enabled=true
    server.tomcat.accesslog.pattern=%h %l %u %t "%r" %s %b %D %{User-Agent}i
    
    nacos.security.ignore.urls=/,/**/*.css,/**/*.js,/**/*.html,/**/*.map,/**/*.svg,/**/*.png,/**/*.ico,/console-fe/public/**,/v1/auth/login,/v1/console/health/**,/v1/cs/**,/v1/ns/**,/v1/cmdb/**,/actuator/**,/v1/console/server/**
    nacos.naming.distro.taskDispatchThreadCount=1
    nacos.naming.distro.taskDispatchPeriod=200
    nacos.naming.distro.batchSyncKeyCount=1000
    nacos.naming.distro.initDataRatio=0.9
    nacos.naming.distro.syncRetryDelay=5000
    nacos.naming.data.warmup=true
    nacos.naming.expireInstance=true
    
    nacos.core.auth.enabled=true

**2.2 nacos集群**  
[集群部署说明][Link 1],官网上面对docker、k8s的集群方案都有描述，但怎么组合还是要看自己。  
[Kubernetes Nacos][]中操作

git clone https://github.com/nacos-group/nacos-k8s.git
    cd nacos-k8s
    chmod +x quick-startup.sh
    ./quick-startup.sh

执行完就后悔了，因为再执行`kubectl create -f deploy/mysql/mysql-nfs.yaml`就报错了

[root@worker1 nacos-k8s-master]# kubectl create -f deploy/mysql/mysql-nfs.yaml
    Error from server (AlreadyExists): error when creating "deploy/mysql/mysql-nfs.yaml": replicationcontrollers "mysql" already exists
    Error from server (AlreadyExists): error when creating "deploy/mysql/mysql-nfs.yaml": services "mysql" already exists

执行`kubectl get pod --all-namespaces`,发现里面有一个mysql  
![1][1 5]  
查看`quick-startup.sh`就明白了，执行`kubectl delete -f ./deploy/mysql/mysql-local.yaml`就可以了。

[root@worker1 nacos-k8s-master]# cat quick-startup.sh 
    #!/usr/bin/env bash
    
    echo "mysql mysql startup"
    kubectl create -f ./deploy/mysql/mysql-local.yaml
    
    
    echo "nacos quick startup"
    kubectl create -f ./deploy/nacos/nacos-quick-start.yaml
    [root@worker1 nacos-k8s-master]#

执行`kubectl create -f deployment.yaml`,这个时候执行`kubectl api-versions`,更改为：`apiVersion: apps/v1`

[root@worker1 nfs]# kubectl create -f deployment.yaml 
    serviceaccount/nfs-client-provisioner created
    error: unable to recognize "deployment.yaml": no matches for kind "Deployment" in version "extensions/v1beta1"

如果出现下面的错误，则需要增加下图红色区域的配置。解决方案参考了[Kubernetes基础：Deployments从beta版本到v1的变化对应方法][Kubernetes_Deployments_beta_v1]

kubectl create -f deploy/nfs/deployment.yaml[root@worker1 nacos-k8s-master]# kubectl create -f deploy/nfs/deployment.yaml
    serviceaccount/nfs-client-provisioner created
    error: error validating "deploy/nfs/deployment.yaml": error validating data: ValidationError(Deployment.spec): missing required field "selector" in io.k8s.api.apps.v1.DeploymentSpec; if you choose to ignore these errors, turn validation off with --validate=false

![1][1 6]

kubectl create -f deploy/nfs/rbac.yaml
    kubectl create -f deploy/nfs/deployment.yaml
    kubectl create -f deploy/nfs/class.yaml
    kubectl get pod -l app=nfs-client-provisioner
    kubectl create -f deploy/mysql/mysql-nfs.yaml
    kubectl create -f deploy/nacos/nacos-pvc-nfs.yaml
    # 需要等个十几秒钟
    kubectl get pod -l app=nacos

在Kuboard中可以看到有两个  
![1][1 7]  
按照官方步骤部署完毕，nacos启动不能成功，原因是找不到数据库。  
通过下面可以查看到mysql的日志, 发现全是`2020-06-04T08:29:20.273263Z 6323 [Note] Access denied for user 'nacos'@'%' to database 'nacos_devtest'`,不得不说阿里做的东西确实不错，但是写帮助手册的说明思路太差了。  
![1][1 8]  
进入之后，点击`sh`  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dhcnJhaA_size_16_color_FFFFFF_t_70 1]  
接着在这里里面执行mysql的命令,我自己调整了`mysql-nfs.yaml`,但这个是我最开始的版本啊。  
![1][1 9]  
从下图可以看到自动创建数据的脚本对应的数据的名称不正确  
![1][1 10]  
在容器内部执行发布配置测试  
![1][1 11]  
在数据库中也可以查看到刷新进去的配置，但是我是通过容器组IP写进去的，并不是主机ip，那么为什么外部如何访问的到呢？  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dhcnJhaA_size_16_color_FFFFFF_t_70 2]  
执行`kubectl describe svc`查看端口映射情况  
![1][1 12]  
[k8s部署nacos][k8s_nacos]建议将`nacos-pvc-nfs.yaml`中更改为`type: NodePort`，这里照做后，还是有问题  
![1][1 13]  
执行`kubectl logs nacos-1`错误日志内容为

2020/06/05 16:01:53 Have not found myself in list yet.
    My Hostname: nacos-0.nacos-headless.default.svc.cluster.local

有人建议干脆不要使用nacos的k8s集群，可是为什么不去探究一下为什么呢？这里得研究一下[kubernetes之StatefulSet详解][kubernetes_StatefulSet]，因为nacos是有状态的设计，故在k8s中选择的是`StatefulSet`,而有状态的服务之间的访问，在k8s中使用的是`Headless Service`进行代理.`headless service`没有 `Cluster IP`  
创建`Headless Service`在k8s中`<pod-name>.<svc-name>.<namespace>.svc.cluster.local`这个地址是唯一的“可解析身份”，使用`kubectl get pod`查看`pod-name`，执行`kubectl get svc`获取`svc-name`  
参考[StatefulSet（一）：拓扑状态][StatefulSet]，执行

kubectl run -i --tty --image busybox:1.28.4 dns-test --restart=Never --rm /bin/sh
    # 接着执行
    / # nslookup nacos-1.nacos-headless
    Server:    179.10.0.2
    Address 1: 179.10.0.2 coredns.kube-system.svc.cluster.local
    
    Name:      nacos-1.nacos-headless
    Address 1: 179.20.3.198 nacos-1.nacos-headless.default.svc.cluster.local
    / # nslookup nacos-0.nacos-headless
    Server:    179.10.0.2
    Address 1: 179.10.0.2 coredns.kube-system.svc.cluster.local
    
    Name:      nacos-0.nacos-headless
    Address 1: 179.20.4.53 nacos-0.nacos-headless.default.svc.cluster.local

k8s内部可以识别这些DNS，那么如果想外部也能访问，该如何处理呢？[StatefulSet][StatefulSet 1]工作原理讲的比较清楚，  
在统一个namespace容器内部，去ping另外一个容器，是可以ping通的，如果不在一个namespace则需要通过`ping nacos-1.nacos-headless.default`,其中`default`可以更改为自己的命名空间  
![1][1 14]  
这个时候使用在其他容器中通过此域名,访问是可以的`curl -X GET "http://nacos-0.nacos-headless.default:8848/nacos/v1/cs/configs?dataId=nacos.cfg.dataId&group=test"`  
![1][1 15]  
如此在测试环境、生产环境的解决方案就出来了，但是开发环境，我将nacos部署到k8s中，而我的应用要连上去该怎么办呢？[有状态服务-StatefulSet使用最佳实践][-StatefulSet]这些文档虽然很好，但只是讲了部分内容，实地开发就有些无从下手了。  
通过[KubeOperator][]可以选择traefix作为插件进行安装,如果traefix来代理这些nacos是否可行呢?  
部署traefix之后，执行配置`kubectl create -f nacos-ingress.yaml`,其中`hs.nacos.com`自己设置，本地host文件配置即可

---
    apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
      name: nacos
      annotations:
        traefik.frontend.priority: "1"
    spec:
      rules:
      - host: hs.nacos.com
        http:
          paths:
          - path: /
            backend:
              serviceName: nacos-headless
              servicePort: 8848

为了外部能够访问，我反复的执行了很多遍，突然k8s不稳定了，提示下面的错误，导致nacos提示`502 Bad Gateway`.

Warning
    FailedMount
    2020-06-08 14:41:36
    5 分钟
    kubelet, worker2.bw-cluster.shahu.local
    Unable to attach or mount volumes: unmounted volumes=[plugindir nfs-client-provisioner-token-2266k datadir logdir], unattached volumes=[plugindir nfs-client-provisioner-token-2266k datadir logdir]: timed out waiting for the condition
    Warning
    Failed
    2020-06-08 14:39:40
    7 分钟
    kubelet, worker2.bw-cluster.shahu.local
    Error: cannot find volume "plugindir" to mount into container "nacos"

从下图可以按到，安装nacos会下载`nacos/nacos-peer-finder-plugin:1.0`镜像，有可能是网络问题导致下载失败，docker有重试机制，他又会自动下载  
![1][1 16]  
检查发现有一台机器已经无法分配Pod，因为k8s默认Pod数量为110，那么怎么更改呢？Kubernetes的node默认最大pod数量为110个，所有node都达到110个时无法再调度

0/6 nodes are available: 1 Insufficient pods, 2 node(s) had taints that the pod didn't tolerate, 3 node(s) were unschedulable.

![1][1 17]

vi /etc/systemd/system/kubelet.service
    # 将
    --max-pods调大一些
    # 重启
    systemctl daemon-reload
    systemctl restart kubelet
    systemctl status kubelet

调整之后，仍然有一个节点是挂掉的

[root@worker3 ~]# kubectl get pods|grep nacos
    nacos-0                                 1/1     Running       0          21h
    nacos-1                                 1/1     Running       0          21h
    nacos-2                                 0/1     Terminating   0          5m12s

查看nacos-2日志，有两个错误:

Error: cannot find volume "plugindir" to mount into container "nacos"
    Unable to attach or mount volumes: unmounted volumes=[datadir logdir plugindir nfs-client-provisioner-token-5mxhq], unattached volumes=[datadir logdir plugindir nfs-client-provisioner-token-5mxhq]: timed out waiting for the condition

通过执行`journalctl -f -u kubelet` 发现异常信息,执行`kubectl create clusterrolebinding system-node-role-bound --clusterrole=system:node --group=system:nodes`,  
下面这个问题是权限造成的，应该nacos deployment的配置有问题

Failed to list *v1.Secret: secrets is forbidden: User "system:node:worker3.bw-cluster.shahu.local" cannot list resource "secrets" in API group

[docker_MariaDB]: https://www.cnblogs.com/WaterGe/p/11067387.html
[1]: https://img-blog.csdnimg.cn/20200602093202544.png
[1 1]: https://img-blog.csdnimg.cn/20200602093412119.png
[HashiCorp]: https://mp.weixin.qq.com/s/3AJiNkN3EuIxBAQedva8Vw
[Eureka 2.0 _Discontinued]: https://github.com/Netflix/eureka/wiki
[Docker_Nacos-Server]: https://www.jianshu.com/p/3d3e17bc629f
[alibaba_nacos]: https://github.com/alibaba/nacos
[docker]: https://blog.csdn.net/u012922706/article/details/80094509
[1 2]: https://img-blog.csdnimg.cn/20200602110031840.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dhcnJhaA==,size_16,color_FFFFFF,t_70
[1 3]: https://img-blog.csdnimg.cn/20200602142736520.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dhcnJhaA==,size_16,color_FFFFFF,t_70
[DOCKER_NACOS]: https://www.cnblogs.com/zyb2016/p/14747544.html
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dhcnJhaA_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20200602143104320.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dhcnJhaA==,size_16,color_FFFFFF,t_70
[1 4]: https://img-blog.csdnimg.cn/20200602144343148.png
[Link 1]: https://nacos.io/zh-cn/docs/cluster-mode-quick-start.html
[Kubernetes Nacos]: https://nacos.io/zh-cn/docs/use-nacos-with-kubernetes.html
[1 5]: https://img-blog.csdnimg.cn/20200604140738432.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dhcnJhaA==,size_16,color_FFFFFF,t_70
[Kubernetes_Deployments_beta_v1]: https://blog.csdn.net/liumiaocn/article/details/104058302/
[1 6]: https://img-blog.csdnimg.cn/20200604145830322.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dhcnJhaA==,size_16,color_FFFFFF,t_70
[1 7]: https://img-blog.csdnimg.cn/2020060415101777.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dhcnJhaA==,size_16,color_FFFFFF,t_70
[1 8]: https://img-blog.csdnimg.cn/20200604163012835.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dhcnJhaA==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dhcnJhaA_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20200604163511383.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dhcnJhaA==,size_16,color_FFFFFF,t_70
[1 9]: https://img-blog.csdnimg.cn/20200604163636967.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dhcnJhaA==,size_16,color_FFFFFF,t_70
[1 10]: https://img-blog.csdnimg.cn/20200604164335171.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dhcnJhaA==,size_16,color_FFFFFF,t_70
[1 11]: https://img-blog.csdnimg.cn/20200605091129337.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dhcnJhaA==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dhcnJhaA_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/20200605092206951.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dhcnJhaA==,size_16,color_FFFFFF,t_70
[1 12]: https://img-blog.csdnimg.cn/20200605094153410.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dhcnJhaA==,size_16,color_FFFFFF,t_70
[k8s_nacos]: https://blog.csdn.net/weixin_30622107/article/details/97528632
[1 13]: https://img-blog.csdnimg.cn/20200605160633125.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dhcnJhaA==,size_16,color_FFFFFF,t_70
[kubernetes_StatefulSet]: https://blog.51cto.com/newfly/2140004
[StatefulSet]: https://www.cnblogs.com/crazymagic/p/10565531.html
[StatefulSet 1]: https://www.jianshu.com/p/dec455ae0455
[1 14]: https://img-blog.csdnimg.cn/20200608094935347.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dhcnJhaA==,size_16,color_FFFFFF,t_70
[1 15]: https://img-blog.csdnimg.cn/20200608100301540.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dhcnJhaA==,size_16,color_FFFFFF,t_70
[-StatefulSet]: https://www.alibabacloud.com/help/zh/doc-detail/100013.htm
[KubeOperator]: https://docs.kubeoperator.io/KubeOperator-v2.6/introduction/
[1 16]: https://img-blog.csdnimg.cn/20200609161720270.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dhcnJhaA==,size_16,color_FFFFFF,t_70
[1 17]: https://img-blog.csdnimg.cn/2020060815135030.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dhcnJhaA==,size_16,color_FFFFFF,t_70