Kubernetes和Jenkins——基于Kubernetes构建Jenkins持续集成平台

布满荆棘的人生 2023-01-20 11:47 51阅读 0赞

### 基于Kubernetes/K8S构建Jenkins持续集成平台 ###

*  Kubernetes+Docker+Jenkins持续集成架构图
 *   *  Kubernetes+Docker+Jenkins持续集成方案好处
 *  Kubeadm安装Kubernetes
 *   *  Kubernetes的架构
     *  安装环境准备
     *  安装Docker、kubelet、kubeadm、kubectl
     *   *  安装Docker
         *  安装kubelet、kubeadm、kubectl
     *  部署Kubernetes Master（master节点）
     *  加入Kubernetes Node（Slave节点）
     *  kubectl常用命令
 *  基于Kubernetes构建Jenkins持续集成平台
 *   *  安装和配置NFS
     *  在Kubernetes安装Jenkins-Master
     *   *  创建NFS client provisioner
         *  安装Jenkins-Master
         *  Jenkins插件管理
     *  Jenkins与Kubernetes整合
     *  构建Jenkins-Slave自定义镜像
     *   *  测试Jenkins-Slave是否可以创建
     *  Jenkins+Kubernetes+Docker完成微服务持续集成
     *   *  拉取代码，构建镜像
         *   *  创建NFS共享目录
             *  创建项目，编写构建Pipeline
             *  记录
     *  微服务部署到k8s
     *   *  完成eureka集群部署到Kubernetes
         *  错误排查
     *  网关集群部署到k8s

# Kubernetes+Docker+Jenkins持续集成架构图 #

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70_pic_center]

1.  构建K8S集群
2.  Jenkins调度K8S API
3.  动态生成 Jenkins Slave pod
4.  Slave pod 拉取 Git 代码／编译／打包镜像
5.  推送到镜像仓库 Harbor
6.  Slave 工作完成，Pod 自动销毁
7.  部署到测试或生产 Kubernetes平台

## Kubernetes+Docker+Jenkins持续集成方案好处 ##

*  服务高可用
    
     *  当 Jenkins Master 出现故障时，Kubernetes 会自动创建一个新的 Jenkins Master容器，并且将 Volume 分配给新创建的容器，保证数据不丢失，从而达到集群服务高可用
 *  动态伸缩，合理使用资源
    
     *  每次运行 Job 时，会自动创建一个 Jenkins Slave，Job 完成后，Slave自动注销并删除容器，资源自动释放
     *  而且 Kubernetes 会根据每个资源的使用情况，动态分配Slave 到空闲的节点上创建，降低出现因某节点资源利用率高，还排队等待在该节点的情况。
 *  扩展性好  
    当 Kubernetes 集群的资源严重不足而导致 Job 排队等待时，可以很容易的添加一个Kubernetes Node 到集群中，从而实现扩展。

# Kubeadm安装Kubernetes #

K8S详细可以参考：[Kubernetes][]

## Kubernetes的架构 ##

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 1]

*  API Server：用于暴露Kubernetes API，任何资源的请求的调用操作都是通过kube-apiserver提供的接口进行的。
 *  Etcd：是Kubernetes提供默认的存储系统，保存所有集群数据，使用时需要为etcd数据提供备份计划。
 *  Controller-Manager：作为集群内部的管理控制中心，负责集群内的Node、Pod副本、服务端点（Endpoint）、命名空间（Namespace）、服务账号（ServiceAccount）、资源定额（ResourceQuota）的管理，当某个Node意外宕机时，Controller Manager会及时发现并执行自动化修复流程，确保集群始终处于预期的工作状态。
 *  Scheduler：监视新创建没有分配到Node的Pod，为Pod选择一个Node。
 *  Kubelet：负责维护容器的生命周期，同时负责Volume和网络的管理
 *  Kube proxy：是Kubernetes的核心组件，部署在每个Node节点上，它是实现Kubernetes Service的通信与负载均衡机制的重要组件

## 安装环境准备 ##

<table> 
 <thead> 
  <tr> 
   <th>主机</th> 
   <th>ip</th> 
   <th>安装的软件</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>k8s-master</td> 
   <td>192.168.188.116</td> 
   <td>kube-apiserver、kube-controller-manager、kube-scheduler、docker、etcd、calico，NFS</td> 
  </tr> 
  <tr> 
   <td>k8s-node1</td> 
   <td>192.168.188.117</td> 
   <td>kubelet、kubeproxy、Docker</td> 
  </tr> 
  <tr> 
   <td>k8s-node2</td> 
   <td>192.168.188.118</td> 
   <td>kubelet、kubeproxy、Docker</td> 
  </tr> 
  <tr> 
   <td>harbor服务器</td> 
   <td>192.168.188.119</td> 
   <td>Harbor</td> 
  </tr> 
  <tr> 
   <td>gitlab服务器</td> 
   <td>192.168.188.120</td> 
   <td>Gitlab</td> 
  </tr> 
 </tbody> 
</table>

**三台k8s服务器都需要完成**

关闭防火墙
    systemctl stop firewalld
    systemctl disable firewalld
    
    关闭selinux
    # 临时关闭
    setenforce 0
    # 永久关闭
    sed -i 's/enforcing/disabled/' /etc/selinux/config  
      
    关闭swap
    # 临时
    swapoff -a 
    # 永久关闭
    sed -ri 's/.*swap.*/#&/' /etc/fstab
    
    # 根据规划设置主机名【master节点上操作】
    hostnamectl set-hostname master
    # 根据规划设置主机名【node1节点操作】
    hostnamectl set-hostname node1
    # 根据规划设置主机名【node2节点操作】
    hostnamectl set-hostname node2
    
    添加ip到hosts
    cat >> /etc/hosts << EOF
    192.168.188.116 master
    192.168.188.117 node1
    192.168.188.118 node2
    EOF
    
    设置系统参数
    设置允许路由转发，不对bridge的数据进行处理
    cat > /etc/sysctl.d/k8s.conf << EOF
    net.bridge.bridge-nf-call-ip6tables = 1
    net.bridge.bridge-nf-call-iptables = 1
    net.ipv4.ip_forward = 1
    vm.swappiness = 0
    EOF
    
    执行文件
    sysctl -p /etc/sysctl.d/k8s.conf
    
    kube-proxy开启ipvs的前置条件
    
    cat > /etc/sysconfig/modules/ipvs.modules <<EOF 
    #!/bin/bash 
    modprobe -- ip_vs 
    modprobe -- ip_vs_rr 
    modprobe -- ip_vs_wrr 
    modprobe -- ip_vs_sh 
    modprobe -- nf_conntrack_ipv4 
    EOF
    
    chmod 755 /etc/sysconfig/modules/ipvs.modules && bash 
    
    /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4

## 安装Docker、kubelet、kubeadm、kubectl ##

所有节点安装Docker、kubeadm、kubelet，Kubernetes默认CRI（容器运行时）为Docker，因此先安装Docker

### 安装Docker ###

首先配置一下Docker的阿里yum源
    安装需要的安装包
    yum install -y yum-utils
    
    设置镜像仓库
    我们用阿里云
    
    yum-config-manager \
        --add-repo \
        http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
    
    更新yum软件包索引
    yum makecache fast
    
    安装docker	docker-ce 社区
    yum -y install docker-ce
    
    查看版本
    docker version
    
    设置开机启动
    systemctl enable docker --now
    
    配置docker的镜像源
    mkdir -p /etc/docker
    
    这个是我自己阿里云的加速 每个人都不一样 可以去阿里云官方查看
    
    tee /etc/docker/daemon.json <<-EOF
     {
       "registry-mirrors": ["https://m0rsqupc.mirror.aliyuncs.com"]
     }
    EOF
    
    验证
    [root@master ~]# cat /etc/docker/daemon.json 
     {
        
       "registry-mirrors": ["https://m0rsqupc.mirror.aliyuncs.com"]
     }
    
    然后重启docker
    systemctl restart docker

### 安装kubelet、kubeadm、kubectl ###

*  kubeadm: 用来初始化集群的指令
 *  kubelet: 在集群中的每个节点上用来启动 pod 和 container 等
 *  kubectl: 用来与集群通信的命令行工具

添加kubernetes软件源
    
    cat > /etc/yum.repos.d/kubernetes.repo << EOF
    [kubernetes]
    name=Kubernetes
    baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
    enabled=1
    gpgcheck=0
    repo_gpgcheck=0
    gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
    EOF
    
    安装kubelet、kubeadm、kubectl，同时指定版本
    
    yum install -y kubelet-1.18.0 kubeadm-1.18.0 kubectl-1.18.0
    
    systemctl enable kubelet --now

## 部署Kubernetes Master（master节点） ##

kubeadm init --kubernetes-version=1.18.0 --apiserver-advertise-address=192.168.188.116 --image-repository registry.aliyuncs.com/google_containers --kubernetes-version v1.18.0 --service-cidr=10.96.0.0/12  --pod-network-cidr=10.244.0.0/16
    
    由于默认拉取镜像地址k8s.gcr.io国内无法访问，这里指定阿里云镜像仓库地址，（执行上述命令会比较慢，因为后台其实已经在拉取镜像了）
    我们 docker images 命令即可查看已经拉取的镜像

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 2]

表示kubernetes的镜像已经安装成功  
红色圈出来的部分 是下面加入从节点需要使用的命令

kubeadm join 192.168.188.116:6443 --token ic49lg.zuwab84r0zfs6bbr \
        --discovery-token-ca-cert-hash sha256:270285cba2080b1e291a3a2b3b21730616b59c95c55ca6f950fecf7b68869b97

使用kubectl工具
    mkdir -p $HOME/.kube
    
    cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    
    chown $(id -u):$(id -g) $HOME/.kube/config
    
    查看运行的节点
    kubectl get nodes
    
    目前有一个master节点已经运行了，但是还处于未准备状态

**安装Calico**  
Calico是一个网络组件，作用是实现master和子节点实现网络通讯功能

mkdir k8s 
    cd k8s 
    
    wget https://docs.projectcalico.org/v3.10/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml
    
    sed -i 's/192.168.0.0/10.244.0.0/g' calico.yaml 
    
    安装calico的组件
    kubectl apply -f calico.yaml
    
    查看pod
    kubectl get pod --all-namespaces
    必须是全部READY 状态必须是Running
    
    查看更详细的信息
    kubectl get pod --all-namespaces -o wide

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 3]

## 加入Kubernetes Node（Slave节点） ##

需要到 node1 和 node2服务器，向集群添加新节点

执行在kubeadm init输出的kubeadm join命令

以下的命令是在master初始化完成后，每个人的都不一样！！！需要复制自己生成的

kubeadm join 192.168.188.116:6443 --token ic49lg.zuwab84r0zfs6bbr \
        --discovery-token-ca-cert-hash sha256:270285cba2080b1e291a3a2b3b21730616b59c95c55ca6f950fecf7b68869b97
    
    查看kubelet是否开启
    systemctl status kubelet
    
    然后查看node信息
    kubectl get nodes
    
    [root@master k8s]# kubectl get nodes
    NAME     STATUS   ROLES    AGE     VERSION
    master   Ready    master   17m     v1.18.0
    node1    Ready    <none>   2m27s   v1.18.0
    node2    Ready    <none>   2m25s   v1.18.0
    
    如果Status全部为Ready，代表集群环境搭建成功

## kubectl常用命令 ##

kubectl get nodes 查看所有主从节点的状态 
    kubectl get ns 获取所有namespace资源 
    kubectl get pods -n {
        $nameSpace} 获取指定namespace的pod 
    kubectl describe pod的名称 -n {
        $nameSpace} 查看某个pod的执行过程 
    kubectl logs --tail=1000 pod的名称 | less 查看日志 
    kubectl create -f xxx.yml 通过配置文件创建一个集群资源对象 
    kubectl delete -f xxx.yml 通过配置文件删除一个集群资源对象 
    kubectl delete pod名称 -n {
        $nameSpace} 通过pod删除集群资源 
    kubectl get service -n {
        $nameSpace} 查看pod的service情况

# 基于Kubernetes构建Jenkins持续集成平台 #

*  在K8S的基础上创建一个Jenkins主节点
 *  在K8S上再创建Jenkins的从节点，来帮助我们进行项目的构建

## 安装和配置NFS ##

NFS（Network File System），它最大的功能就是可以通过网络，让不同的机器、不同的操作系统可以共享彼此的文件。我们可以利用NFS共享Jenkins运行的配置文件、Maven的仓库依赖文件等

我们把NFS服务器安装在K8S主节点上

安装NFS服务（这个需要在所有K8S的节点上安装）
    yum install -y nfs-utils
    
    创建共享目录（这个只需要在master节点）
    mkdir -p /opt/nfs/jenkins
    编写NFS的共享配置
    vim /etc/exports 
    
    /opt/nfs/jenkins	*(rw,no_root_squash) 
    
    *代表对所有IP都开放此目录，rw是读写，no_root_squash不压制root权限
    
    启动服务
    systemctl enable nfs --now
    
    查看NFS共享目录
    showmount -e 192.168.188.116
    
    [root@node1 ~]# showmount -e 192.168.188.116
    Export list for 192.168.188.116:
    /opt/nfs/jenkins *

## 在Kubernetes安装Jenkins-Master ##

### 创建NFS client provisioner ###

nfs-client-provisioner 是一个Kubernetes的简易NFS的外部provisioner，本身不提供NFS，需要现有的NFS服务器提供存储

需要编写yaml了

*  StorageClass.yaml
 *  持久化存储Storageclass
 *  `kubectl explain StorageClass`查看kind和version
 *  定义名称为`managed-nfs-storage`

apiVersion: storage.k8s.io/v1
    kind: StorageClass
    metadata:
      name: managed-nfs-storage
    provisioner: fuseim.pri/ifs # or choose another name, must match deployment's env PROVISIONER_NAME'
    parameters:
      archiveOnDelete: "true"

*  deployment.yaml
 *  部署NFS client provisioner

apiVersion: storage.k8s.io/v1
    kind: ServiceAccount
    metadata:
      name: nfs-client-provisioner
    ---
    kind: Deployment
    apiVersion: apps/v1
    metadata:
      name: nfs-client-provisioner
    spec:
      replicas: 1
      strategy:
        type: Recreate
      selector:
        matchLabels:
          app: nfs-client-provisioner
      template:
        metadata:
          labels:
            app: nfs-client-provisioner
        spec:
          serviceAccountName: nfs-client-provisioner
          containers:
            - name: nfs-client-provisioner
              image: lizhenliang/nfs-client-provisioner:latest
              volumeMounts:
                - name: nfs-client-root
                  mountPath: /persistentvolumes
              env:
                - name: PROVISIONER_NAME
                  value: fuseim.pri/ifs
                - name: NFS_SERVER
                  value: 192.168.188.116
                - name: NFS_PATH
                  value: /opt/nfs/jenkins/
          volumes:
            - name: nfs-client-root
              nfs:
                server: 192.168.188.116
                path: /opt/nfs/jenkins/

最后是一个权限文件rbac.yaml

kind: ServiceAccount
    apiVersion: v1
    metadata:
      name: nfs-client-provisioner
    ---
    kind: ClusterRole
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: nfs-client-provisioner-runner
    rules:
      - apiGroups: [""]
        resources: ["persistentvolumes"]
        verbs: ["get", "list", "watch", "create", "delete"]
      - apiGroups: [""]
        resources: ["persistentvolumeclaims"]
        verbs: ["get", "list", "watch", "update"]
      - apiGroups: ["storage.k8s.io"]
        resources: ["storageclasses"]
        verbs: ["get", "list", "watch"]
      - apiGroups: [""]
        resources: ["events"]
        verbs: ["create", "update", "patch"]
    ---
    kind: ClusterRoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: run-nfs-client-provisioner
    subjects:
      - kind: ServiceAccount
        name: nfs-client-provisioner
        namespace: default
    roleRef:
      kind: ClusterRole
      name: nfs-client-provisioner-runner
      apiGroup: rbac.authorization.k8s.io
    ---
    kind: Role
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: leader-locking-nfs-client-provisioner
    rules:
      - apiGroups: [""]
        resources: ["endpoints"]
        verbs: ["get", "list", "watch", "create", "update", "patch"]
    ---
    kind: RoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: leader-locking-nfs-client-provisioner
    subjects:
      - kind: ServiceAccount
        name: nfs-client-provisioner
        # replace with namespace where provisioner is deployed
        namespace: default
    roleRef:
      kind: Role
      name: leader-locking-nfs-client-provisioner
      apiGroup: rbac.authorization.k8s.io

构建pod

将三个文件都写在同一个目录里面
    kubectl create -f .
    
    [root@master nfs-client]# kubectl create -f .
    storageclass.storage.k8s.io/managed-nfs-storage created
    serviceaccount/nfs-client-provisioner created
    deployment.apps/nfs-client-provisioner created
    clusterrole.rbac.authorization.k8s.io/nfs-client-provisioner-runner created
    clusterrolebinding.rbac.authorization.k8s.io/run-nfs-client-provisioner created
    role.rbac.authorization.k8s.io/leader-locking-nfs-client-provisioner created
    rolebinding.rbac.authorization.k8s.io/leader-locking-nfs-client-provisioner created
    Error from server (AlreadyExists): error when creating "rbac.yaml": serviceaccounts "nfs-client-provisioner" already exists
    
    查看pod
    kubectl get pod
    
    [root@master nfs-client]# kubectl get pod
    NAME                                      READY   STATUS    RESTARTS   AGE
    nfs-client-provisioner-795b4df87d-zchrq   1/1     Running   0          2m4s

### 安装Jenkins-Master ###

还是需要我们自己写Jenkins的Yaml文件

*  rbac.yaml
 *  和权限有关的信息，把jenkins主节点的一些权限加入到k8s的管理下

kind: Role
    apiVersion: rbac.authorization.k8s.io/v1beta1
    metadata:
      name: jenkins
      namespace: kube-ops
    rules:
      - apiGroups: ["extensions", "apps"]
        resources: ["deployments"]
        verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
      - apiGroups: [""]
        resources: ["services"]
        verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
      - apiGroups: [""]
        resources: ["pods"]
        verbs: ["create","delete","get","list","patch","update","watch"]
      - apiGroups: [""]
        resources: ["pods/exec"]
        verbs: ["create","delete","get","list","patch","update","watch"]
      - apiGroups: [""]
        resources: ["pods/log"]
        verbs: ["get","list","watch"]
      - apiGroups: [""]
        resources: ["secrets"]
        verbs: ["get"]
    
    ---
    apiVersion: rbac.authorization.k8s.io/v1beta1
    kind: RoleBinding
    metadata:
      name: jenkins
      namespace: kube-ops
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: Role
      name: jenkins
    subjects:
      - kind: ServiceAccount
        name: jenkins
        namespace: kube-ops
        
    ---
    
    kind: ClusterRole
    apiVersion: rbac.authorization.k8s.io/v1beta1
    metadata:
      name: jenkinsClusterRole
      namespace: kube-ops
    rules:
    - apiGroups: [""]
      resources: ["pods"]
      verbs: ["create","delete","get","list","patch","update","watch"]
    - apiGroups: [""]
      resources: ["pods/exec"]
      verbs: ["create","delete","get","list","patch","update","watch"]
    - apiGroups: [""]
      resources: ["pods/log"]
      verbs: ["get","list","watch"]
    - apiGroups: [""]
      resources: ["secrets"]
      verbs: ["get"]
     
    ---
    apiVersion: rbac.authorization.k8s.io/v1beta1
    kind: RoleBinding
    metadata:
      name: jenkinsClusterRuleBinding
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: jenkinsClusterRole
    subjects:
    - kind: ServiceAccount
      name: jenkins
      namespace: kube-ops

*  ServiceaAcount.yaml
 *  jenkins主节点的一些ServiceaAcount信息

apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: jenkins
      namespace: kube-ops

*  StatefulSet.yaml
 *  是一个有状态应用，里面就定义了之前NFS的信息

apiVersion: apps/v1
    kind: StatefulSet
    metadata:
      name: jenkins
      labels:
        name: jenkins
      namespace: kube-ops
    spec:
      serviceName: jenkins
      selector:
        matchLabels:
          app: jenkins
      replicas: 1
      updateStrategy:
        type: RollingUpdate
      template:
        metadata:
          name: jenkins
          labels:
            app: jenkins
        spec:
          terminationGracePeriodSeconds: 10
          serviceAccountName: jenkins
          containers:
            - name: jenkins
              image: jenkins/jenkins:lts-alpine
              imagePullPolicy: IfNotPresent
              ports:
              - containerPort: 8080
                name: web
                protocol: TCP
              - containerPort: 50000
                name: agent
                protocol: TCP
              resources:
                limits:
                  cpu: 1
                  memory: 1Gi
                requests:
                  cpu: 0.5
                  memory: 500Mi
              env:
                - name: LIMITS_MEMORY
                  valueFrom:
                    resourceFieldRef:
                      resource: limits.memory
                      divisor: 1Mi
                - name: JAVA_OPTS
                  value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85
              volumeMounts:
                - name: jenkins-home
                  mountPath: /var/jenkins_home
              livenessProbe:
                httpGet:
                  path: /login
                  port: 8080
                initialDelaySeconds: 60
                timeoutSeconds: 5
                failureThreshold: 12
              readinessProbe:
                httpGet:
                  path: /login
                  port: 8080
                initialDelaySeconds: 60
                timeoutSeconds: 5
                failureThreshold: 12
          securityContext:
            fsGroup: 1000
      volumeClaimTemplates:
      - metadata:
          name: jenkins-home
        spec:
          storageClassName: "managed-nfs-storage"
          accessModes: [ "ReadWriteOnce" ]
          resources:
            requests:
              storage: 1Gi

*  Service.yaml
 *  对外暴露一些端口

apiVersion: v1
    kind: Service
    metadata:
      name: jenkins
      namespace: kube-ops
      labels:
        app: jenkins
    spec:
      selector:
        app: jenkins
      type: NodePort
      ports:
      - name: web
        port: 8080
        targetPort: web
      - name: agent
        port: 50000
        targetPort: agent

cd jenkins-master/
    [root@master jenkins-master]# ls
    rbac.yaml  ServiceaAcount.yaml  Service.yaml  StatefulSet.yaml
    
    创建一个新的命名空间
    kubectl create namespace kube-ops 
    
    创建jenkins主节点的pod
    kubectl create -f .
    
    
    [root@master jenkins-master]# kubectl create namespace kube-ops
    namespace/kube-ops created
    [root@master jenkins-master]# kubectl create -f .
    service/jenkins created
    serviceaccount/jenkins created
    statefulset.apps/jenkins created
    role.rbac.authorization.k8s.io/jenkins created
    rolebinding.rbac.authorization.k8s.io/jenkins created
    clusterrole.rbac.authorization.k8s.io/jenkinsClusterRole created
    rolebinding.rbac.authorization.k8s.io/jenkinsClusterRuleBinding created
    
    我们可以通过很多个命令来查看pod和service的信息
    查看的时候需要加上我们自己的命名空间
    kubectl get pod --namespace kube-ops
    
    [root@master jenkins-master]# kubectl get pod --namespace kube-ops
    NAME        READY   STATUS    RESTARTS   AGE
    jenkins-0   1/1     Running   0          7m38s
    
    查看服务的信息
    kubectl get service --namespace kube-ops
    
    [root@master jenkins-master]# kubectl get service --namespace kube-ops
    NAME      TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)                          AGE
    jenkins   NodePort   10.101.126.122   <none>        8080:30942/TCP,50000:31796/TCP   8m2s
    
    -o wide查看更详细的信息
    kubectl get service --namespace kube-ops -o wide
    
    [root@master jenkins-master]# kubectl get service --namespace kube-ops -o wide
    NAME      TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)                          AGE     SELECTOR
    jenkins   NodePort   10.101.126.122   <none>        8080:30942/TCP,50000:31796/TCP   8m20s   app=jenkins
    
    查看pod更详细的信息
    kubectl get pod --namespace kube-ops -o wide
    
    [root@master jenkins-master]# kubectl get pod --namespace kube-ops -o wide
    NAME        READY   STATUS    RESTARTS   AGE     IP               NODE    NOMINATED NODE   READINESS GATES
    jenkins-0   1/1     Running   0          8m48s   10.244.166.129   node1   <none>           <none>
    
    通过详细信息发现，我们的jenkins是部署在node1节点上
    我们可以通过浏览器访问
    node1的ip+端口

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 4]  
这里需要解锁jenkins，我们需要去jenkins的共享目录，NFS下寻找

nfs目录在/opt/nfs
    [root@master jenkins-master]# cd /opt//nfs/
    [root@master nfs]# ls
    jenkins
    [root@master nfs]# cd jenkins/
    [root@master jenkins]# ll
    total 4
    drwxrwxrwx 13 root root 4096 May 13 16:26 kube-ops-jenkins-home-jenkins-0-pvc-0d59af09-0339-46eb-a6a0-550c40365efb
    
    这个就是jenkins的运行目录
    [root@master jenkins]# cd kube-ops-jenkins-home-jenkins-0-pvc-0d59af09-0339-46eb-a6a0-550c40365efb/
    [root@master kube-ops-jenkins-home-jenkins-0-pvc-0d59af09-0339-46eb-a6a0-550c40365efb]# ls
    config.xml                     jenkins.telemetry.Correlator.xml  nodes                     secrets      war
    copy_reference_file.log        jobs                              plugins                   updates
    hudson.model.UpdateCenter.xml  logs                              secret.key                userContent
    identity.key.enc               nodeMonitors.xml                  secret.key.not-so-secret  users
    [root@master kube-ops-jenkins-home-jenkins-0-pvc-0d59af09-0339-46eb-a6a0-550c40365efb]# cat secrets/initialAdminPassword 
    5496a0ee3afd449fb65c709d6d721c5d
    
    这个就是管理员密码 将其复制到浏览器解锁jenkins

跳过插件安装，我们一会儿修改下载源之后再安装插件  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 5]  
选择无  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 6]  
创建用户  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 7]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 8]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 9]  
Jenkins在k8s的环境下运行成功

### Jenkins插件管理 ###

Jenkins国外官方插件地址下载速度非常慢，所以可以修改为国内插件地址  
Jenkins->Manage Jenkins->Manage Plugins，点击Available  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 10]  
这样做是为了把Jenkins官方的插件列表下载到本地，接着修改地址文件，替换为国内插件地址

[root@master kube-ops-jenkins-home-jenkins-0-pvc-0d59af09-0339-46eb-a6a0-550c40365efb]# pwd
    /opt/nfs/jenkins/kube-ops-jenkins-home-jenkins-0-pvc-0d59af09-0339-46eb-a6a0-550c40365efb
    [root@master kube-ops-jenkins-home-jenkins-0-pvc-0d59af09-0339-46eb-a6a0-550c40365efb]# cd updates/
    [root@master updates]# ls
    default.json  hudson.tasks.Maven.MavenInstaller
    
    default.json就是插件下载地址
    我们修改插件下载地址
    
    sed -i 's/http:\/\/updates.jenkins-ci.org\/download/https:\/\/mirrors.tuna.tsinghua.edu.cn\/jenkins/g' default.json && sed -i 's/http:\/\/www.google.com/https:\/\/www.baidu.com/g' default.json
    
    sed -i 's/http:\/\/updates.jenkins-ci.org\/download/https:\/\/mirrors.tuna.tsinghua.edu.cn\/jenkins/g'default.json && sed -i 's/http:\/\/www.google.com/https:\/\/www.baidu.com/g' default.json

最后，Manage Plugins点击Advanced，把Update Site改为国内插件下载地址

> https://mirrors.tuna.tsinghua.edu.cn/jenkins/updates/update-center.json

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 11]  
在浏览器ip后面/restart 然后重启jenkins  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 12]  
![在这里插入图片描述][20210513164825484.png]  
安装基本插件

*  Chinese
 *  Git
 *  Pipeline
 *  Extended Choice Parameter

等待安装  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 13]

## Jenkins与Kubernetes整合 ##

安装`Kubernetes`插件  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 14]

安装完成之后前往全局配置最下面  
![在这里插入图片描述][20210513173456285.png]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 15]

*  kubernetes地址采用了kube的服务器发现：https://kubernetes.default.svc.cluster.local
 *  namespace填kube-ops，然后点击Test Connection，如果出现 Connection test successful 的提示信息证明 Jenkins 已经可以和 Kubernetes 系统正常通信
 *  Jenkins URL 地址：http://jenkins.kube-ops.svc.cluster.local:8080

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 16]  
点击连接测试  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 17]

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 18]

## 构建Jenkins-Slave自定义镜像 ##

Jenkins-Master在构建Job的时候，Kubernetes会创建Jenkins-Slave的Pod来完成Job的构建。我们选择运行Jenkins-Slave的镜像为官方推荐镜像：`jenkins/jnlp-slave:latest`，但是这个镜像里面并没有Maven环境，为了方便使用，我们需要自定义一个新的镜像

所以我们要编写Dockerfile自定义一个镜像

vim Dockerfile
    
    FROM jenkins/jnlp-slave:latest
    
    MAINTAINER xiaotian
    
    # 切换到 root 账户进行操作
    USER root
    
    # 安装 maven
    COPY apache-maven-3.6.2-bin.tar.gz .
    
    RUN tar -zxf apache-maven-3.6.2-bin.tar.gz && \
        mv apache-maven-3.6.2 /usr/local && \
        rm -f apache-maven-3.6.2-bin.tar.gz && \
        ln -s /usr/local/apache-maven-3.6.2/bin/mvn /usr/bin/mvn && \
        ln -s /usr/local/apache-maven-3.6.2 /usr/local/apache-maven && \
        mkdir -p /usr/local/apache-maven/repo
    
    COPY settings.xml /usr/local/apache-maven/conf/settings.xml
    
    USER jenkins

还需要一个带有阿里源镜像的settings文件

<?xml version="1.0" encoding="UTF-8"?>
    
    
    
    
    <settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
              xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
              xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">
      
      <localRepository>/usr/local/apache-maven/repo</localRepository>
    
      
    
      
    
      
      <pluginGroups>
        
      </pluginGroups>
    
      
      <proxies>
        
      </proxies>
    
      
      <servers>
        
    
        
      </servers>
    
      
      <mirrors>
        
        <mirror>     
          <id>central</id>     
          <mirrorOf>central</mirrorOf>     
          <name>aliyun maven</name>
          <url>https://maven.aliyun.com/repository/public</url>     
        </mirror>
      </mirrors>
    
      
      <profiles>
        
    
        
      </profiles>
    
      
    </settings>

最后需要一个maven的安装包  
我这里使用 apache-maven-3.6.2-bin.tar

[root@master jenkins-slave]# ls
    apache-maven-3.6.2-bin.tar.gz  Dockerfile  settings.xml
    
    创建镜像
    docker build -t jenkins-slave-maven:latest .

![在这里插入图片描述][2021051317520393.png]

查看镜像
    docker images

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 19]  
然后把镜像上传到Harbor的公共库library中（因为k8s从节点都需要下载此镜像，因此放在公共库最方便）

打标签
    docker tag jenkins-slave-maven:latest 192.168.188.119:85/library/jenkins-slave-maven:latest
    
    vim /etc/docker/daemon.json
    
     {
        
       "registry-mirrors": ["https://m0rsqupc.mirror.aliyuncs.com"],
       "insecure-registries": ["192.168.188.119:85"]
     }
    
    登录
    docker login -u admin -p Harbor12345 192.168.188.119:85
    
    上传镜像
    docker push 192.168.188.119:85/library/jenkins-slave-maven:latest

镜像上传成功  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 20]

### 测试Jenkins-Slave是否可以创建 ###

创建一个Jenkins流水线项目  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 21]

添加gitlab的凭据，一会儿流水线脚本需要使用  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 22]

编写Pipeline，从GItlab拉取代码

def git_address = "http://47.108.13.86:82/maomao_group/tensquare_back.git" 
    def git_auth = "bb5a1b2e-8dfa-4557-a79b-66d0f1f05f5c" 
    
    //创建一个Pod的模板，label为jenkins-slave 
    podTemplate(label: 'jenkins-slave', cloud: 'kubernetes', containers: [ 
        containerTemplate( 
            name: 'jnlp', 
            image: "192.168.188.119:85/library/jenkins-slave-maven:latest"
        ) 
      ]
    )
    {
         
      //引用jenkins-slave的pod模块来构建Jenkins-Slave的pod
      node("jenkins-slave"){
         
          // 第一步 
          stage('拉取代码'){
         
              checkout([$class: 'GitSCM', branches: [[name: 'master']], 
              userRemoteConfigs: [[credentialsId: "${git_auth}", url: "${git_address}"]]])
          } 
      }
    }

*  podTemplate 就是k8s的pod模板
 *  label 是模板名字
 *  cloud 是云名字 在全局配置里的Cloud里自己填写的
 *  containerTemplate 指定在pod里运行的容器

构建成功

![在这里插入图片描述][20210513205236528.png]  
中间遇到的错误  
![在这里插入图片描述][20210513205418356.png]  
所有k8s节点必须授权harbor仓库地址

vi /etc/docker/daemon.json
    
    {
        
    "registry-mirrors": ["https://zydiol88.mirror.aliyuncs.com"],
    "insecure-registries": ["harbor-ip:85"]
    }

## Jenkins+Kubernetes+Docker完成微服务持续集成 ##

### 拉取代码，构建镜像 ###

#### 创建NFS共享目录 ####

让所有Jenkins-Slave构建指向NFS的Maven的共享仓库目录

mkdir -p /opt/nfs/maven
    vi /etc/exports
    
    /opt/nfs/jenkins        *(rw,no_root_squash)
    /opt/nfs/maven          *(rw,no_root_squash)
    
    systemctl restart nfs
    
    showmount -e 192.168.188.116 
    Export list for 192.168.188.116:
    /opt/nfs/maven   *
    /opt/nfs/jenkins *

#### 创建项目，编写构建Pipeline ####

在流水线里添加多选项参数  
![在这里插入图片描述][20210513214315411.png]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 23]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 24]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 25]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 26]  
再添加字符参数  
![在这里插入图片描述][20210513221533408.png]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 27]

保存

接着去添加harbor的凭证  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 28]  
编写流水线脚本

def git_address = "http://192.168.188.120:82/maomao_group/tensquare_back.git"
    def git_auth = "bb5a1b2e-8dfa-4557-a79b-66d0f1f05f5c" 
    //构建版本的名称 
    def tag = "latest" 
    //Harbor私服地址 
    def harbor_url = "192.168.188.119:85" 
    //Harbor的项目名称 
    def harbor_project_name = "tensquare" 
    //Harbor的凭证 
    def harbor_auth = "4fe90544-8b7b-4f81-b282-d20a2eb6e437" 
    
    podTemplate(label: 'jenkins-slave', cloud: 'kubernetes', containers: [ 
    	containerTemplate( 
    		name: 'jnlp',
    		image: "192.168.188.119:85/library/jenkins-slave-maven:latest"
    	),
    	containerTemplate(
    		name: 'docker',
    		image: "docker:stable",
    		ttyEnabled: true,
    		command: 'cat'
    	),
      ],
      volumes: [
        hostPathVolume(mountPath: '/var/run/docker.sock', hostPath:'/var/run/docker.sock'),
    	nfsVolume(mountPath: '/usr/local/apache-maven/repo', serverAddress: '192.168.188.116' , serverPath: '/opt/nfs/maven'),
    	],
    )
    {
         
      node("jenkins-slave"){
        
    	   // 第一步
    	   stage('拉取代码'){
        
    		  checkout([$class: 'GitSCM', branches: [[name: '${branch}']],
    		  userRemoteConfigs: [[credentialsId: "${git_auth}", url: "${git_address}"]]])
    	   }
    	   // 第二步
    	   stage('公共工程代码编译'){
        
    			//编译并安装公共工程
    		  sh "mvn -f tensquare_common clean install"
    	   }
    	   // 第三步
    	   stage('构建镜像，部署项目'){
        
    			 //把选择的项目信息转为数组
    			 def selectedProjects = "${project_name}".split(',')
    			 
    			 for(int i=0;i<selectedProjects.size();i++){
        
    				//取出每个项目的名称和端口 
    				def currentProject = selectedProjects[i];
    				//项目名称
    				def currentProjectName = currentProject.split('@')[0]
    				//项目启动端口
    				def currentProjectPort = currentProject.split('@')[1]
    				
    				//定义镜像名称
    				def imageName = "${currentProjectName}:${tag}"
    				
    				//编译，构建本地镜像
    				sh "mvn -f ${currentProjectName} clean package dockerfile:build"
    				
    				container('docker') {
        
    				
    					//给镜像打标签
    					sh "docker tag ${imageName} ${harbor_url}/${harbor_project_name}/${imageName}"
    					
    					//登录Harbor，并上传镜像
    					withCredentials([usernamePassword(credentialsId:"${harbor_auth}", passwordVariable: 'password', usernameVariable: 'username')])
    				{
        
    						  //登录
    						  sh "docker login -u ${username} -p ${password} ${harbor_url}"
    						  
    						  //上传镜像
    						  sh "docker push ${harbor_url}/${harbor_project_name}/${imageName}"
    					}
    					//删除本地镜像
    					sh "docker rmi -f ${imageName}"
    					sh "docker rmi -f ${harbor_url}/${harbor_project_name}/${imageName}"
    				}
    			}
    		}
    	}
    }

注意：在构建过程会发现无法创建仓库目录，是因为NFS共享目录权限不足，需更改权限

只用在k8s主节点

chmod -R 777 /opt/nfs/maven

如果出现这种错误  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 29]

是Docker命令执行权限问题，所有k8s服务器都需要执行

chmod 777 /var/run/docker.sock

需要手动上传父工程依赖到NFS的Maven共享仓库目录中

#### 记录 ####

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 30]  
脚本写好之后，都能错这么多次，真的太难了

错了只能看日志一步一步进行排错

因为这一步实在是太容易错了，我记录一下构建过程，以后遇到问题可以查看

Started by user maomao
    Running in Durability level: MAX_SURVIVABILITY
    [Pipeline] Start of Pipeline
    [Pipeline] podTemplate
    [Pipeline] {
        
    [Pipeline] node
    Created Pod: kubernetes kube-ops/jenkins-slave-qv8g5-p9qlr
    Agent jenkins-slave-qv8g5-p9qlr is provisioned from template jenkins-slave-qv8g5
    ---
    apiVersion: "v1"
    kind: "Pod"
    metadata:
      annotations:
        buildUrl: "http://jenkins.kube-ops.svc.cluster.local:8080/job/tensquare__back/15/"
        runUrl: "job/tensquare__back/15/"
      labels:
        jenkins: "slave"
        jenkins/label-digest: "5059d2cd0054f9fe75d61f97723d98ab1a42d71a"
        jenkins/label: "jenkins-slave"
      name: "jenkins-slave-qv8g5-p9qlr"
    spec:
      containers:
      - env:
        - name: "JENKINS_SECRET"
          value: "********"
        - name: "JENKINS_AGENT_NAME"
          value: "jenkins-slave-qv8g5-p9qlr"
        - name: "JENKINS_NAME"
          value: "jenkins-slave-qv8g5-p9qlr"
        - name: "JENKINS_AGENT_WORKDIR"
          value: "/home/jenkins/agent"
        - name: "JENKINS_URL"
          value: "http://jenkins.kube-ops.svc.cluster.local:8080/"
        image: "192.168.188.119:85/library/jenkins-slave-maven:latest"
        imagePullPolicy: "IfNotPresent"
        name: "jnlp"
        resources:
          limits: {
        }
          requests: {
        }
        tty: false
        volumeMounts:
        - mountPath: "/usr/local/apache-maven/repo"
          name: "volume-1"
          readOnly: false
        - mountPath: "/var/run/docker.sock"
          name: "volume-0"
          readOnly: false
        - mountPath: "/home/jenkins/agent"
          name: "workspace-volume"
          readOnly: false
      - command:
        - "cat"
        image: "docker:stable"
        imagePullPolicy: "IfNotPresent"
        name: "docker"
        resources:
          limits: {
        }
          requests: {
        }
        tty: true
        volumeMounts:
        - mountPath: "/usr/local/apache-maven/repo"
          name: "volume-1"
          readOnly: false
        - mountPath: "/var/run/docker.sock"
          name: "volume-0"
          readOnly: false
        - mountPath: "/home/jenkins/agent"
          name: "workspace-volume"
          readOnly: false
      nodeSelector:
        kubernetes.io/os: "linux"
      restartPolicy: "Never"
      volumes:
      - hostPath:
          path: "/var/run/docker.sock"
        name: "volume-0"
      - name: "volume-1"
        nfs:
          path: "/opt/nfs/maven"
          readOnly: false
          server: "192.168.188.116"
      - emptyDir:
          medium: ""
        name: "workspace-volume"
    
    Running on jenkins-slave-qv8g5-p9qlr in /home/jenkins/agent/workspace/tensquare__back
    [Pipeline] {
        
    [Pipeline] stage
    [Pipeline] {
         (拉取代码)
    [Pipeline] checkout
    The recommended git tool is: NONE
    using credential bb5a1b2e-8dfa-4557-a79b-66d0f1f05f5c
    Cloning the remote Git repository
    Cloning repository http://192.168.188.120:82/maomao_group/tensquare_back.git
     > git init /home/jenkins/agent/workspace/tensquare__back # timeout=10
    Fetching upstream changes from http://192.168.188.120:82/maomao_group/tensquare_back.git
     > git --version # timeout=10
     > git --version # 'git version 2.20.1'
    using GIT_ASKPASS to set credentials gitlab-http-auth
     > git fetch --tags --force --progress -- http://192.168.188.120:82/maomao_group/tensquare_back.git +refs/heads/*:refs/remotes/origin/* # timeout=10
    Avoid second fetch
    Checking out Revision dba0faf11591dc9aa572e43bb0b5134b3ebf195e (origin/master)
     > git config remote.origin.url http://192.168.188.120:82/maomao_group/tensquare_back.git # timeout=10
     > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
     > git rev-parse origin/master^{
        commit} # timeout=10
     > git config core.sparsecheckout # timeout=10
     > git checkout -f dba0faf11591dc9aa572e43bb0b5134b3ebf195e # timeout=10
    Commit message: "牛"
     > git rev-list --no-walk dba0faf11591dc9aa572e43bb0b5134b3ebf195e # timeout=10
    [Pipeline] }
    [Pipeline] // stage
    [Pipeline] stage
    [Pipeline] {
         (公共工程代码编译)
    [Pipeline] sh
    + mvn -f tensquare_common clean install
    [INFO] Scanning for projects...
    [INFO] 
    [INFO] -------------------< com.tensquare:tensquare_common >-------------------
    [INFO] Building tensquare_common 1.0-SNAPSHOT
    [INFO] --------------------------------[ jar ]---------------------------------
    [INFO] 
    [INFO] --- maven-clean-plugin:3.0.0:clean (default-clean) @ tensquare_common ---
    [INFO] Deleting /home/jenkins/agent/workspace/tensquare__back/tensquare_common/target
    [INFO] 
    [INFO] --- maven-resources-plugin:3.0.1:resources (default-resources) @ tensquare_common ---
    [INFO] Using 'UTF-8' encoding to copy filtered resources.
    [INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/tensquare__back/tensquare_common/src/main/resources
    [INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/tensquare__back/tensquare_common/src/main/resources
    [INFO] 
    [INFO] --- maven-compiler-plugin:3.7.0:compile (default-compile) @ tensquare_common ---
    [INFO] Changes detected - recompiling the module!
    [INFO] Compiling 5 source files to /home/jenkins/agent/workspace/tensquare__back/tensquare_common/target/classes
    [INFO] 
    [INFO] --- maven-resources-plugin:3.0.1:testResources (default-testResources) @ tensquare_common ---
    [INFO] Using 'UTF-8' encoding to copy filtered resources.
    [INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/tensquare__back/tensquare_common/src/test/resources
    [INFO] 
    [INFO] --- maven-compiler-plugin:3.7.0:testCompile (default-testCompile) @ tensquare_common ---
    [INFO] No sources to compile
    [INFO] 
    [INFO] --- maven-surefire-plugin:2.21.0:test (default-test) @ tensquare_common ---
    [INFO] No tests to run.
    [INFO] 
    [INFO] --- maven-jar-plugin:3.0.2:jar (default-jar) @ tensquare_common ---
    [INFO] Building jar: /home/jenkins/agent/workspace/tensquare__back/tensquare_common/target/tensquare_common-1.0-SNAPSHOT.jar
    [INFO] 
    [INFO] --- maven-install-plugin:2.5.2:install (default-install) @ tensquare_common ---
    [INFO] Installing /home/jenkins/agent/workspace/tensquare__back/tensquare_common/target/tensquare_common-1.0-SNAPSHOT.jar to /usr/local/apache-maven/repo/com/tensquare/tensquare_common/1.0-SNAPSHOT/tensquare_common-1.0-SNAPSHOT.jar
    [INFO] Installing /home/jenkins/agent/workspace/tensquare__back/tensquare_common/pom.xml to /usr/local/apache-maven/repo/com/tensquare/tensquare_common/1.0-SNAPSHOT/tensquare_common-1.0-SNAPSHOT.pom
    [INFO] ------------------------------------------------------------------------
    [INFO] BUILD SUCCESS
    [INFO] ------------------------------------------------------------------------
    [INFO] Total time:  4.162 s
    [INFO] Finished at: 2021-05-13T15:27:00Z
    [INFO] ------------------------------------------------------------------------
    [Pipeline] }
    [Pipeline] // stage
    [Pipeline] stage
    [Pipeline] {
         (构建镜像，部署项目)
    [Pipeline] sh
    + mvn -f tensquare_eureka_server clean package dockerfile:build
    [INFO] Scanning for projects...
    [INFO] 
    [INFO] ---------------< com.tensquare:tensquare_eureka_server >----------------
    [INFO] Building tensquare_eureka_server 1.0-SNAPSHOT
    [INFO] --------------------------------[ jar ]---------------------------------
    [INFO] 
    [INFO] --- maven-clean-plugin:3.0.0:clean (default-clean) @ tensquare_eureka_server ---
    [INFO] Deleting /home/jenkins/agent/workspace/tensquare__back/tensquare_eureka_server/target
    [INFO] 
    [INFO] --- maven-resources-plugin:3.0.1:resources (default-resources) @ tensquare_eureka_server ---
    [INFO] Using 'UTF-8' encoding to copy filtered resources.
    [INFO] Copying 1 resource
    [INFO] Copying 0 resource
    [INFO] 
    [INFO] --- maven-compiler-plugin:3.7.0:compile (default-compile) @ tensquare_eureka_server ---
    [INFO] Changes detected - recompiling the module!
    [INFO] Compiling 1 source file to /home/jenkins/agent/workspace/tensquare__back/tensquare_eureka_server/target/classes
    [INFO] 
    [INFO] --- maven-resources-plugin:3.0.1:testResources (default-testResources) @ tensquare_eureka_server ---
    [INFO] Using 'UTF-8' encoding to copy filtered resources.
    [INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/tensquare__back/tensquare_eureka_server/src/test/resources
    [INFO] 
    [INFO] --- maven-compiler-plugin:3.7.0:testCompile (default-testCompile) @ tensquare_eureka_server ---
    [INFO] No sources to compile
    [INFO] 
    [INFO] --- maven-surefire-plugin:2.21.0:test (default-test) @ tensquare_eureka_server ---
    [INFO] No tests to run.
    [INFO] 
    [INFO] --- maven-jar-plugin:3.0.2:jar (default-jar) @ tensquare_eureka_server ---
    [INFO] Building jar: /home/jenkins/agent/workspace/tensquare__back/tensquare_eureka_server/target/tensquare_eureka_server-1.0-SNAPSHOT.jar
    [INFO] 
    [INFO] --- spring-boot-maven-plugin:2.0.1.RELEASE:repackage (default) @ tensquare_eureka_server ---
    [INFO] 
    [INFO] --- dockerfile-maven-plugin:1.3.6:build (default-cli) @ tensquare_eureka_server ---
    [INFO] Building Docker context /home/jenkins/agent/workspace/tensquare__back/tensquare_eureka_server
    [INFO] 
    [INFO] Image will be built as tensquare_eureka_server:latest
    [INFO] 
    [INFO] Step 1/5 : FROM openjdk:8-jdk-alpine
    [INFO] 
    [INFO] Pulling from library/openjdk
    [INFO] Digest: sha256:94792824df2df33402f201713f932b58cb9de94a0cd524164a0f2283343547b3
    [INFO] Status: Image is up to date for openjdk:8-jdk-alpine
    [INFO]  ---> a3562aa0b991
    [INFO] Step 2/5 : ARG JAR_FILE
    [INFO] 
    [INFO]  ---> Using cache
    [INFO]  ---> a2a3b3df4f15
    [INFO] Step 3/5 : COPY ${JAR_FILE} app.jar
    [INFO] 
    [INFO]  ---> 656a595f07ab
    [INFO] Step 4/5 : EXPOSE 10086
    [INFO] 
    [INFO]  ---> Running in 868cfdbdd284
    [INFO] Removing intermediate container 868cfdbdd284
    [INFO]  ---> 4fa7a8297ad1
    [INFO] Step 5/5 : ENTRYPOINT ["java","-jar","/app.jar"]
    [INFO] 
    [INFO]  ---> Running in bee52b92749b
    [INFO] Removing intermediate container bee52b92749b
    [INFO]  ---> d0ec04357240
    [INFO] Successfully built d0ec04357240
    [INFO] Successfully tagged tensquare_eureka_server:latest
    [INFO] 
    [INFO] Detected build of image with id d0ec04357240
    [INFO] Building jar: /home/jenkins/agent/workspace/tensquare__back/tensquare_eureka_server/target/tensquare_eureka_server-1.0-SNAPSHOT-docker-info.jar
    [INFO] Successfully built tensquare_eureka_server:latest
    [INFO] ------------------------------------------------------------------------
    [INFO] BUILD SUCCESS
    [INFO] ------------------------------------------------------------------------
    [INFO] Total time:  27.874 s
    [INFO] Finished at: 2021-05-13T15:27:29Z
    [INFO] ------------------------------------------------------------------------
    [Pipeline] container
    [Pipeline] {
        
    [Pipeline] sh
    + docker tag tensquare_eureka_server:latest 192.168.188.119:85/tensquare/tensquare_eureka_server:latest
    [Pipeline] withCredentials
    Masking supported pattern matches of $username or $password
    [Pipeline] {
        
    [Pipeline] sh
    Warning: A secret was passed to "sh" using Groovy String interpolation, which is insecure.
    		 Affected argument(s) used the following variable(s): [password, username]
    		 See https://jenkins.io/redirect/groovy-string-interpolation for details.
    + docker login -u **** -p **** 192.168.188.119:85
    WARNING! Using --password via the CLI is insecure. Use --password-stdin.
    WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
    Configure a credential helper to remove this warning. See
    https://docs.docker.com/engine/reference/commandline/login/#credentials-store
    
    Login Succeeded
    [Pipeline] sh
    + docker push 192.168.188.119:85/tensquare/tensquare_eureka_server:latest
    The push refers to repository [192.168.188.119:85/tensquare/tensquare_eureka_server]
    c8a7e30e2667: Preparing
    ceaf9e1ebef5: Preparing
    9b9b7f3d56a0: Preparing
    f1b5933fe4b5: Preparing
    ceaf9e1ebef5: Layer already exists
    9b9b7f3d56a0: Layer already exists
    f1b5933fe4b5: Layer already exists
    c8a7e30e2667: Pushed
    latest: digest: sha256:320a88f1b5efa46bc74643c156652ca724211fff07578c8733ac7208f2ffa8a9 size: 1159
    [Pipeline] }
    [Pipeline] // withCredentials
    [Pipeline] sh
    + docker rmi -f tensquare_eureka_server:latest
    Untagged: tensquare_eureka_server:latest
    [Pipeline] sh
    + docker rmi -f 192.168.188.119:85/tensquare/tensquare_eureka_server:latest
    Untagged: 192.168.188.119:85/tensquare/tensquare_eureka_server:latest
    Untagged: 192.168.188.119:85/tensquare/tensquare_eureka_server@sha256:320a88f1b5efa46bc74643c156652ca724211fff07578c8733ac7208f2ffa8a9
    Deleted: sha256:d0ec043572402382bd3610e7fe9c63642c838edb43f7a16d822c374d231f9e9a
    Deleted: sha256:4fa7a8297ad170d46a9b5774b6f6adacd6276e9acae9a4cbbb918286a7617c67
    Deleted: sha256:656a595f07ab8eb40c24f787cdff50cbf0082e450251cc007a6bb055e67824e2
    Deleted: sha256:caa525b07e2d84ac4c0ef38c69f193cd72d094c285d2971b5017359b9c9cb1d5
    [Pipeline] }
    [Pipeline] // container
    [Pipeline] }
    [Pipeline] // stage
    [Pipeline] }
    [Pipeline] // node
    [Pipeline] }
    [Pipeline] // podTemplate
    [Pipeline] End of Pipeline
    Finished: SUCCESS

## 微服务部署到k8s ##

*  需要把构建好的微服务镜像 交给k8s的环境来进行运行，因此需要一个`Kubernetes Continuous Deploy`插件
 *  这是一个持续集成的插件

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 31]  
修改后的流水线脚本

*  `deploy.yml`描述了 在k8s里怎么部署不同微服务
 *  kubeconfigId 是证书，需要在jenkins环境里创建出来

添加k8s证书，添加凭据  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 32]  
这个证书在k8s主节点

cd /root/.kube
    cat config
    
    显示的内容就是证书，我们需要把内容原封不动的全部复制过来

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 33]  
然后再次查看 会产生一个id，这个id添加到脚本里  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 34]  
![在这里插入图片描述][20210513235432534.png]

def git_address = "http://192.168.188.120:82/maomao_group/tensquare_back.git"
    def git_auth = "bb5a1b2e-8dfa-4557-a79b-66d0f1f05f5c" 
    //构建版本的名称 
    def tag = "latest" 
    //Harbor私服地址 
    def harbor_url = "192.168.188.119:85" 
    //Harbor的项目名称 
    def harbor_project_name = "tensquare" 
    //Harbor的凭证 
    def harbor_auth = "2ec3c8b6-f9b6-4ef6-b4bc-fdba74f99420" 
    //k8s的凭证
    def k8s_auth = ""9565b450-3899-4892-8aed-d15b6f26f8fd
    
    podTemplate(label: 'jenkins-slave', cloud: 'kubernetes', containers: [ 
    	containerTemplate( 
    		name: 'jnlp',
    		image: "192.168.188.119:85/library/jenkins-slave-maven:latest"
    	),
    	containerTemplate(
    		name: 'docker',
    		image: "docker:stable",
    		ttyEnabled: true,
    		command: 'cat'
    	),
      ],
      volumes: [
        hostPathVolume(mountPath: '/var/run/docker.sock', hostPath:'/var/run/docker.sock'),
    	nfsVolume(mountPath: '/usr/local/apache-maven/repo', serverAddress: '192.168.188.116' , serverPath: '/opt/nfs/maven'),
    	],
    )
    {
         
      node("jenkins-slave"){
        
    	   // 第一步
    	   stage('拉取代码'){
        
    		  checkout([$class: 'GitSCM', branches: [[name: '${branch}']],
    		  userRemoteConfigs: [[credentialsId: "${git_auth}", url: "${git_address}"]]])
    	   }
    	   // 第二步
    	   stage('公共工程代码编译'){
        
    			//编译并安装公共工程
    		  sh "mvn -f tensquare_common clean install"
    	   }
    	   // 第三步
    	   stage('构建镜像，部署项目'){
        
    			 //把选择的项目信息转为数组
    			 def selectedProjects = "${project_name}".split(',')
    			 
    			 for(int i=0;i<selectedProjects.size();i++){
        
    				//取出每个项目的名称和端口 
    				def currentProject = selectedProjects[i];
    				//项目名称
    				def currentProjectName = currentProject.split('@')[0]
    				//项目启动端口
    				def currentProjectPort = currentProject.split('@')[1]
    				
    				//定义镜像名称
    				def imageName = "${currentProjectName}:${tag}"
    				
    				//编译，构建本地镜像
    				sh "mvn -f ${currentProjectName} clean package dockerfile:build"
    				
    				container('docker') {
        
    				
    					//给镜像打标签
    					sh "docker tag ${imageName} ${harbor_url}/${harbor_project_name}/${imageName}"
    					
    					//登录Harbor，并上传镜像
    					withCredentials([usernamePassword(credentialsId:"${harbor_auth}", passwordVariable: 'password', usernameVariable: 'username')])
    				{
        
    						  //登录
    						  sh "docker login -u ${username} -p ${password} ${harbor_url}"
    						  
    						  //上传镜像
    						  sh "docker push ${harbor_url}/${harbor_project_name}/${imageName}"
    					}
    					//删除本地镜像
    					sh "docker rmi -f ${imageName}"
    					sh "docker rmi -f ${harbor_url}/${harbor_project_name}/${imageName}"
    				}
    				def deploy_image_name = "${harbor_url}/${harbor_project_name}/${imageName}"
    				
    				//部署到K8S
    				sh """
    						sed -i 's#\$IMAGE_NAME#${deploy_image_name}#' ${currentProjectName}/deploy.yml 
    						sed -i 's#\$SECRET_NAME#${secret_name}#' ${currentProjectName}/deploy.yml 
    				   """
    
    				   kubernetesDeploy configs: "${currentProjectName}/deploy.yml", kubeconfigId: "${k8s_auth}"
    				
    			}
    		}
    	}
    }

### 完成eureka集群部署到Kubernetes ###

我们在eureka项目下创建一个`deploy.yml`文件

---
    apiVersion: v1
    kind: Service
    metadata:
      name: eureka
      labels:
        app: eureka
    spec:
      type: NodePort
      ports:
        - port: 10086
          name: eureka
          targetPort: 10086
      selector:
        app: eureka
    ---
    apiVersion: apps/v1
    kind: StatefulSet
    metadata:
      name: eureka
    spec:
      serviceName: "eureka"
      replicas: 2
      selector:
        matchLabels:
          app: eureka
      template:
        metadata:
          labels:
            app: eureka
        spec:
          imagePullSecrets:
            - name: $SECRET_NAME
          containers:
            - name: eureka
              image: $IMAGE_NAME
              ports:
                - containerPort: 10086
              env:
                - name: MY_POD_NAME
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.name
                - name: EUREKA_SERVER
                  value: "http://eureka-0.eureka:10086/eureka/,http://eureka-1.eureka:10086/eureka/"
                - name: EUREKA_INSTANCE_HOSTNAME
                  value: ${
        MY_POD_NAME}.eureka
      podManagementPolicy: "Parallel"

接着更改eruka的配置文件`application.yml`

server:
      port: ${
        PORT:10086}
    spring:
      application:
        name: eureka
    
    eureka:
      server:
        # 续期时间，即扫描失效服务的间隔时间（缺省为60*1000ms）
        eviction-interval-timer-in-ms: 5000
        enable-self-preservation: false
        use-read-only-response-cache: false
      client:
        # eureka client间隔多久去拉取服务注册信息 默认30s
        registry-fetch-interval-seconds: 5
        serviceUrl:
          defaultZone: ${
        EUREKA_SERVER:http://127.0.0.1:${
        server.port}/eureka/}
      instance:
        # 心跳间隔时间，即发送一次心跳之后，多久在发起下一次（缺省为30s）
        lease-renewal-interval-in-seconds: 5
        # 在收到一次心跳之后，等待下一次心跳的空档时间，大于心跳间隔即可，即服务续约到期时间（缺省为90s）
        lease-expiration-duration-in-seconds: 10
        instance-id: ${
        EUREKA_INSTANCE_HOSTNAME:${
        spring.application.name}}:${
        server.port}@${
        random.long(1000000,9999999)}
        hostname: ${
        EUREKA_INSTANCE_HOSTNAME:${
        spring.application.name}}

提交代码到仓库  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 35]  
尝试构建eruka服务器，但报错显示找不到`secret_name`这个属性  
![在这里插入图片描述][20210514000950195.png]  
修改脚本定义一个secret\_name的变量  
![在这里插入图片描述][20210514001357731.png]  
然后我们去制作凭证

首先在k8s主节点登录harbor仓库
    docker login -u maomao -p Xiaotian123 192.168.188.119:85
    
    然后使用一串命令生成证书
    kubectl create secret docker-registry registry-auth-secret --docker-server=192.168.188.119:85 --docker-username=maomao --docker-password=Xiaotian123 --docker-email=123@qq.com

![在这里插入图片描述][20210514001747109.png]  
查看证书

kubectl get secret

![在这里插入图片描述][20210514001821477.png]  
然后再次构建eruka服务器  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 36]  
构建成功之后，可以在主节点查看信息

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 37]

kubectl get pods
    
    查看端口
    kubectl get svc

![在这里插入图片描述][20210514004103917.png]  
![在这里插入图片描述][20210514004135836.png]  
![在这里插入图片描述][20210514004529763.png]

我们可以通过端口访问 所有k8s服务器ip

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 38]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 39]![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 40]

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 41]

### 错误排查 ###

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 42]  
经过我的深思熟虑，发现是密钥的问题  
因为直接复制，导致有些地方本来是一行的结果变成了两行  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 43]  
将换行删除就行了  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 44]

结果又错了  
![在这里插入图片描述][20210514003255262.png]  
我干脆直接把密钥文件拷贝到自己电脑上进行复制，这样格式就不会错了

记录一下构建过程

Started by user maomao
    Running in Durability level: MAX_SURVIVABILITY
    [Pipeline] Start of Pipeline
    [Pipeline] podTemplate
    [Pipeline] {
        
    [Pipeline] node
    Created Pod: kubernetes kube-ops/jenkins-slave-dlxsx-7pxm1
    Agent jenkins-slave-dlxsx-7pxm1 is provisioned from template jenkins-slave-dlxsx
    ---
    apiVersion: "v1"
    kind: "Pod"
    metadata:
      annotations:
        buildUrl: "http://jenkins.kube-ops.svc.cluster.local:8080/job/tensquare__back/20/"
        runUrl: "job/tensquare__back/20/"
      labels:
        jenkins: "slave"
        jenkins/label-digest: "5059d2cd0054f9fe75d61f97723d98ab1a42d71a"
        jenkins/label: "jenkins-slave"
      name: "jenkins-slave-dlxsx-7pxm1"
    spec:
      containers:
      - env:
        - name: "JENKINS_SECRET"
          value: "********"
        - name: "JENKINS_AGENT_NAME"
          value: "jenkins-slave-dlxsx-7pxm1"
        - name: "JENKINS_NAME"
          value: "jenkins-slave-dlxsx-7pxm1"
        - name: "JENKINS_AGENT_WORKDIR"
          value: "/home/jenkins/agent"
        - name: "JENKINS_URL"
          value: "http://jenkins.kube-ops.svc.cluster.local:8080/"
        image: "192.168.188.119:85/library/jenkins-slave-maven:latest"
        imagePullPolicy: "IfNotPresent"
        name: "jnlp"
        resources:
          limits: {
        }
          requests: {
        }
        tty: false
        volumeMounts:
        - mountPath: "/usr/local/apache-maven/repo"
          name: "volume-1"
          readOnly: false
        - mountPath: "/var/run/docker.sock"
          name: "volume-0"
          readOnly: false
        - mountPath: "/home/jenkins/agent"
          name: "workspace-volume"
          readOnly: false
      - command:
        - "cat"
        image: "docker:stable"
        imagePullPolicy: "IfNotPresent"
        name: "docker"
        resources:
          limits: {
        }
          requests: {
        }
        tty: true
        volumeMounts:
        - mountPath: "/usr/local/apache-maven/repo"
          name: "volume-1"
          readOnly: false
        - mountPath: "/var/run/docker.sock"
          name: "volume-0"
          readOnly: false
        - mountPath: "/home/jenkins/agent"
          name: "workspace-volume"
          readOnly: false
      nodeSelector:
        kubernetes.io/os: "linux"
      restartPolicy: "Never"
      volumes:
      - hostPath:
          path: "/var/run/docker.sock"
        name: "volume-0"
      - name: "volume-1"
        nfs:
          path: "/opt/nfs/maven"
          readOnly: false
          server: "192.168.188.116"
      - emptyDir:
          medium: ""
        name: "workspace-volume"
    
    Running on jenkins-slave-dlxsx-7pxm1 in /home/jenkins/agent/workspace/tensquare__back
    [Pipeline] {
        
    [Pipeline] stage
    [Pipeline] {
         (拉取代码)
    [Pipeline] checkout
    The recommended git tool is: NONE
    using credential bb5a1b2e-8dfa-4557-a79b-66d0f1f05f5c
    Cloning the remote Git repository
    Cloning repository http://192.168.188.120:82/maomao_group/tensquare_back.git
     > git init /home/jenkins/agent/workspace/tensquare__back # timeout=10
    Fetching upstream changes from http://192.168.188.120:82/maomao_group/tensquare_back.git
     > git --version # timeout=10
     > git --version # 'git version 2.20.1'
    using GIT_ASKPASS to set credentials gitlab-http-auth
     > git fetch --tags --force --progress -- http://192.168.188.120:82/maomao_group/tensquare_back.git +refs/heads/*:refs/remotes/origin/* # timeout=10
    Avoid second fetch
    Checking out Revision a35a2f630f44b425c52aa483aef1b7dc64539940 (origin/master)
     > git config remote.origin.url http://192.168.188.120:82/maomao_group/tensquare_back.git # timeout=10
     > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
     > git rev-parse origin/master^{
        commit} # timeout=10
     > git config core.sparsecheckout # timeout=10
     > git checkout -f a35a2f630f44b425c52aa483aef1b7dc64539940 # timeout=10
    Commit message: "K8S"
     > git rev-list --no-walk a35a2f630f44b425c52aa483aef1b7dc64539940 # timeout=10
    [Pipeline] }
    [Pipeline] // stage
    [Pipeline] stage
    [Pipeline] {
         (公共工程代码编译)
    [Pipeline] sh
    + mvn -f tensquare_common clean install
    [INFO] Scanning for projects...
    [INFO] 
    [INFO] -------------------< com.tensquare:tensquare_common >-------------------
    [INFO] Building tensquare_common 1.0-SNAPSHOT
    [INFO] --------------------------------[ jar ]---------------------------------
    [INFO] 
    [INFO] --- maven-clean-plugin:3.0.0:clean (default-clean) @ tensquare_common ---
    [INFO] Deleting /home/jenkins/agent/workspace/tensquare__back/tensquare_common/target
    [INFO] 
    [INFO] --- maven-resources-plugin:3.0.1:resources (default-resources) @ tensquare_common ---
    [INFO] Using 'UTF-8' encoding to copy filtered resources.
    [INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/tensquare__back/tensquare_common/src/main/resources
    [INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/tensquare__back/tensquare_common/src/main/resources
    [INFO] 
    [INFO] --- maven-compiler-plugin:3.7.0:compile (default-compile) @ tensquare_common ---
    [INFO] Changes detected - recompiling the module!
    [INFO] Compiling 5 source files to /home/jenkins/agent/workspace/tensquare__back/tensquare_common/target/classes
    [INFO] 
    [INFO] --- maven-resources-plugin:3.0.1:testResources (default-testResources) @ tensquare_common ---
    [INFO] Using 'UTF-8' encoding to copy filtered resources.
    [INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/tensquare__back/tensquare_common/src/test/resources
    [INFO] 
    [INFO] --- maven-compiler-plugin:3.7.0:testCompile (default-testCompile) @ tensquare_common ---
    [INFO] No sources to compile
    [INFO] 
    [INFO] --- maven-surefire-plugin:2.21.0:test (default-test) @ tensquare_common ---
    [INFO] No tests to run.
    [INFO] 
    [INFO] --- maven-jar-plugin:3.0.2:jar (default-jar) @ tensquare_common ---
    [INFO] Building jar: /home/jenkins/agent/workspace/tensquare__back/tensquare_common/target/tensquare_common-1.0-SNAPSHOT.jar
    [INFO] 
    [INFO] --- maven-install-plugin:2.5.2:install (default-install) @ tensquare_common ---
    [INFO] Installing /home/jenkins/agent/workspace/tensquare__back/tensquare_common/target/tensquare_common-1.0-SNAPSHOT.jar to /usr/local/apache-maven/repo/com/tensquare/tensquare_common/1.0-SNAPSHOT/tensquare_common-1.0-SNAPSHOT.jar
    [INFO] Installing /home/jenkins/agent/workspace/tensquare__back/tensquare_common/pom.xml to /usr/local/apache-maven/repo/com/tensquare/tensquare_common/1.0-SNAPSHOT/tensquare_common-1.0-SNAPSHOT.pom
    [INFO] ------------------------------------------------------------------------
    [INFO] BUILD SUCCESS
    [INFO] ------------------------------------------------------------------------
    [INFO] Total time:  4.069 s
    [INFO] Finished at: 2021-05-13T16:34:40Z
    [INFO] ------------------------------------------------------------------------
    [Pipeline] }
    [Pipeline] // stage
    [Pipeline] stage
    [Pipeline] {
         (构建镜像，部署项目)
    [Pipeline] sh
    + mvn -f tensquare_eureka_server clean package dockerfile:build
    [INFO] Scanning for projects...
    [INFO] 
    [INFO] ---------------< com.tensquare:tensquare_eureka_server >----------------
    [INFO] Building tensquare_eureka_server 1.0-SNAPSHOT
    [INFO] --------------------------------[ jar ]---------------------------------
    [INFO] 
    [INFO] --- maven-clean-plugin:3.0.0:clean (default-clean) @ tensquare_eureka_server ---
    [INFO] Deleting /home/jenkins/agent/workspace/tensquare__back/tensquare_eureka_server/target
    [INFO] 
    [INFO] --- maven-resources-plugin:3.0.1:resources (default-resources) @ tensquare_eureka_server ---
    [INFO] Using 'UTF-8' encoding to copy filtered resources.
    [INFO] Copying 1 resource
    [INFO] Copying 0 resource
    [INFO] 
    [INFO] --- maven-compiler-plugin:3.7.0:compile (default-compile) @ tensquare_eureka_server ---
    [INFO] Changes detected - recompiling the module!
    [INFO] Compiling 1 source file to /home/jenkins/agent/workspace/tensquare__back/tensquare_eureka_server/target/classes
    [INFO] 
    [INFO] --- maven-resources-plugin:3.0.1:testResources (default-testResources) @ tensquare_eureka_server ---
    [INFO] Using 'UTF-8' encoding to copy filtered resources.
    [INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/tensquare__back/tensquare_eureka_server/src/test/resources
    [INFO] 
    [INFO] --- maven-compiler-plugin:3.7.0:testCompile (default-testCompile) @ tensquare_eureka_server ---
    [INFO] No sources to compile
    [INFO] 
    [INFO] --- maven-surefire-plugin:2.21.0:test (default-test) @ tensquare_eureka_server ---
    [INFO] No tests to run.
    [INFO] 
    [INFO] --- maven-jar-plugin:3.0.2:jar (default-jar) @ tensquare_eureka_server ---
    [INFO] Building jar: /home/jenkins/agent/workspace/tensquare__back/tensquare_eureka_server/target/tensquare_eureka_server-1.0-SNAPSHOT.jar
    [INFO] 
    [INFO] --- spring-boot-maven-plugin:2.0.1.RELEASE:repackage (default) @ tensquare_eureka_server ---
    [INFO] 
    [INFO] --- dockerfile-maven-plugin:1.3.6:build (default-cli) @ tensquare_eureka_server ---
    [INFO] Building Docker context /home/jenkins/agent/workspace/tensquare__back/tensquare_eureka_server
    [INFO] 
    [INFO] Image will be built as tensquare_eureka_server:latest
    [INFO] 
    [INFO] Step 1/5 : FROM openjdk:8-jdk-alpine
    [INFO] 
    [INFO] Pulling from library/openjdk
    [INFO] Digest: sha256:94792824df2df33402f201713f932b58cb9de94a0cd524164a0f2283343547b3
    [INFO] Status: Image is up to date for openjdk:8-jdk-alpine
    [INFO]  ---> a3562aa0b991
    [INFO] Step 2/5 : ARG JAR_FILE
    [INFO] 
    [INFO]  ---> Using cache
    [INFO]  ---> a2a3b3df4f15
    [INFO] Step 3/5 : COPY ${JAR_FILE} app.jar
    [INFO] 
    [INFO]  ---> 1b476689026f
    [INFO] Step 4/5 : EXPOSE 10086
    [INFO] 
    [INFO]  ---> Running in 68e029b5bdac
    [INFO] Removing intermediate container 68e029b5bdac
    [INFO]  ---> f76e827eb058
    [INFO] Step 5/5 : ENTRYPOINT ["java","-jar","/app.jar"]
    [INFO] 
    [INFO]  ---> Running in 92ad4dcb6596
    [INFO] Removing intermediate container 92ad4dcb6596
    [INFO]  ---> ce5f4598f452
    [INFO] Successfully built ce5f4598f452
    [INFO] Successfully tagged tensquare_eureka_server:latest
    [INFO] 
    [INFO] Detected build of image with id ce5f4598f452
    [INFO] Building jar: /home/jenkins/agent/workspace/tensquare__back/tensquare_eureka_server/target/tensquare_eureka_server-1.0-SNAPSHOT-docker-info.jar
    [INFO] Successfully built tensquare_eureka_server:latest
    [INFO] ------------------------------------------------------------------------
    [INFO] BUILD SUCCESS
    [INFO] ------------------------------------------------------------------------
    [INFO] Total time:  28.550 s
    [INFO] Finished at: 2021-05-13T16:35:10Z
    [INFO] ------------------------------------------------------------------------
    [Pipeline] container
    [Pipeline] {
        
    [Pipeline] sh
    + docker tag tensquare_eureka_server:latest 192.168.188.119:85/tensquare/tensquare_eureka_server:latest
    [Pipeline] withCredentials
    Masking supported pattern matches of $username or $password
    [Pipeline] {
        
    [Pipeline] sh
    Warning: A secret was passed to "sh" using Groovy String interpolation, which is insecure.
    		 Affected argument(s) used the following variable(s): [password, username]
    		 See https://jenkins.io/redirect/groovy-string-interpolation for details.
    + docker login -u **** -p **** 192.168.188.119:85
    WARNING! Using --password via the CLI is insecure. Use --password-stdin.
    WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
    Configure a credential helper to remove this warning. See
    https://docs.docker.com/engine/reference/commandline/login/#credentials-store
    
    Login Succeeded
    [Pipeline] sh
    + docker push 192.168.188.119:85/tensquare/tensquare_eureka_server:latest
    The push refers to repository [192.168.188.119:85/tensquare/tensquare_eureka_server]
    6b8a8e1926e4: Preparing
    ceaf9e1ebef5: Preparing
    9b9b7f3d56a0: Preparing
    f1b5933fe4b5: Preparing
    f1b5933fe4b5: Layer already exists
    9b9b7f3d56a0: Layer already exists
    ceaf9e1ebef5: Layer already exists
    6b8a8e1926e4: Pushed
    latest: digest: sha256:584ab9d6bd684636dfef55c2a2ac3d36d445b287f5a0a89a6694823655f909b1 size: 1159
    [Pipeline] }
    [Pipeline] // withCredentials
    [Pipeline] sh
    + docker rmi -f tensquare_eureka_server:latest
    Untagged: tensquare_eureka_server:latest
    [Pipeline] sh
    + docker rmi -f 192.168.188.119:85/tensquare/tensquare_eureka_server:latest
    Untagged: 192.168.188.119:85/tensquare/tensquare_eureka_server:latest
    Untagged: 192.168.188.119:85/tensquare/tensquare_eureka_server@sha256:584ab9d6bd684636dfef55c2a2ac3d36d445b287f5a0a89a6694823655f909b1
    Deleted: sha256:ce5f4598f452e300f537deacab64ee958f93a7c39ced0ff71f360f9c4d5d7572
    Deleted: sha256:f76e827eb0580d99d60f8709608850589404666611fae1068e4e2231d100e6a3
    Deleted: sha256:1b476689026f842092261a4d524043b6c4aa754e6b7b68c25501269fa1e17dce
    Deleted: sha256:731b8259434bd270843c034ed36011581a2371daf28f682a6e3d0fa8b713545a
    [Pipeline] }
    [Pipeline] // container
    [Pipeline] sh
    + sed -i s#$IMAGE_NAME#192.168.188.119:85/tensquare/tensquare_eureka_server:latest# tensquare_eureka_server/deploy.yml
    + sed -i s#$SECRET_NAME#registry-auth-secret# tensquare_eureka_server/deploy.yml
    [Pipeline] kubernetesDeploy
    Starting Kubernetes deployment
    Loading configuration: /home/jenkins/agent/workspace/tensquare__back/tensquare_eureka_server/deploy.yml
    Finished Kubernetes deployment
    [Pipeline] }
    [Pipeline] // stage
    [Pipeline] }
    [Pipeline] // node
    [Pipeline] }
    [Pipeline] // podTemplate
    [Pipeline] End of Pipeline
    Finished: SUCCESS

## 网关集群部署到k8s ##

更改zull的配置，将地址换成这个

defaultZone: http://eureka-0.eureka:10086/eureka/,http://eureka-1.eureka:10086/eureka/

编写一个deploy.yml文件

---
    apiVersion: v1
    kind: Service
    metadata:
      name: zuul
      labels:
        app: zuul
    spec:
      type: NodePort
      ports:
        - port: 10020
          name: zuul
          targetPort: 10020
      selector:
        app: zuul
    ---
    apiVersion: apps/v1
    kind: StatefulSet
    metadata:
      name: zuul
    spec:
      serviceName: "zuul"
      replicas: 2
      selector:
        matchLabels:
          app: zuul
      template:
        metadata:
          labels:
            app: zuul
        spec:
          imagePullSecrets:
            - name: $SECRET_NAME
          containers:
            - name: zuul
              image: $IMAGE_NAME
              ports:
                - containerPort: 10020
      podManagementPolicy: "Parallel"

去进行任务构建  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 45]  
报错了  
![在这里插入图片描述][20210514010434852.png]  
原来时需要 手动上传父工程依赖到NFS的Maven共享仓库目录中

[root@master tensquare]# pwd
    /opt/nfs/maven/com/tensquare
    
    [root@master tensquare]# mv /root/tensquare_parent/ ./
    [root@master tensquare]# ls
    tensquare_common  tensquare_parent

上传完父工程依赖之后再次进行构建  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 46]

已经有两个网关了  
![在这里插入图片描述][2021051401175525.png]  
![在这里插入图片描述][20210514011812456.png]

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 47]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70]: /images/20221021/a6831fba88b2441ca049c95bba215d7d.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70_pic_center]: /images/20221021/3fb3db67717341e18c634d1f1c46f5fb.png
[Kubernetes]: https://blog.csdn.net/Cantevenl/article/details/116481831
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 1]: /images/20221021/1dc5af7edd374c11be3762b77e7982b2.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 2]: /images/20221021/2e8147c77e384a588f64ca417293878d.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 3]: /images/20221021/88fd66ffc245478eb9e067516519d590.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 4]: /images/20221021/3c6d8f63d9b1427189323d37e61e50f4.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 5]: /images/20221021/1eb4b11b5a88483eb49c9af66821b1cc.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 6]: /images/20221021/e16b9eb2194e4e99a0488ca0152ce102.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 7]: /images/20221021/85ada264dfd34105ba889a4a129971c9.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 8]: /images/20221021/24d86191b45644c28fc8650ba4fab152.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 9]: /images/20221021/77ae1388153d41c3a92b6ff27a256c8a.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 10]: /images/20221021/80f10ddc190a483da5d790dfbe15b1b6.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 11]: /images/20221021/6a73318019984047a5c94e751fd52a11.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 12]: /images/20221021/51172cdce54e4f929332bda5219683be.png
[20210513164825484.png]: /images/20221021/5d535f8d14f545fda4664e1a2bd833ad.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 13]: /images/20221021/6d30e380508f46579a275cafdc17baa2.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 14]: /images/20221021/6f0c76bbe34a4dd2bd5db0f8b68e70ff.png
[20210513173456285.png]: /images/20221021/c47e0fcb73314604a644a5e8fe8e1651.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 15]: /images/20221021/7c32b24e8e974a0a818c569fa8a96ac3.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 16]: /images/20221021/14c24a401f7f4bd1a28cf97543ce867b.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 17]: /images/20221021/c797b5bc6ffb4a6290cafcd77e686fa0.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 18]: /images/20221021/f9df4e9784864ff3a422b74d51d27827.png
[2021051317520393.png]: /images/20221021/2afd4430ef0c4dbfaeb8b937a8649ec9.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 19]: /images/20221021/a6ad351686014c29ba145fa0b97d6413.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 20]: /images/20221021/73408a2ee54c4c66bad726e74c948c2f.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 21]: /images/20221021/2994793aaffb4e329f330c64084cead5.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 22]: /images/20221021/bb399ff8845f4bd1a0a503656d66fe5e.png
[20210513205236528.png]: /images/20221021/be743f4185184d7792c4b2de0e6244f6.png
[20210513205418356.png]: /images/20221021/d18767f183b8490b8d3f71cdffae28e2.png
[20210513214315411.png]: /images/20221021/ae0a8b3ab5254714a0463b874cb7a7ff.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 23]: /images/20221021/e48128764af24f17bc158095c6c2b701.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 24]: /images/20221021/61f4d62a11de4398ac6d710374550378.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 25]: /images/20221021/b43c16ab8d9b43c2950a1a9abe22c7c8.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 26]: /images/20221021/a04ad29507bf4099b61d12addf2149b8.png
[20210513221533408.png]: /images/20221021/777a0fb81b324ada8cf8ab4193d754a0.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 27]: /images/20221021/40a109b1ac904a198a1aca7b358669d5.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 28]: /images/20221021/0642999635634b6e8b310324150b30c7.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 29]: /images/20221021/82da762a6bd54e53a43b27bda4271432.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 30]: /images/20221021/9420d208eba64c89a1f2520d3a63d314.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 31]: /images/20221021/84bc91d11aee45008d5e532526c6c285.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 32]: /images/20221021/c4d083be2a7a4308a248ea13ed4884e8.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 33]: /images/20221021/cb12a7e4d8f644f18ab79b5bddea0744.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 34]: /images/20221021/1a9fe17b2862409ba8fdb34e17e6084d.png
[20210513235432534.png]: /images/20221021/50848396d3b140e7b70dd798eaf7ba45.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 35]: /images/20221021/c2514cda2b354ad592f27ff5ed6635a8.png
[20210514000950195.png]: /images/20221021/f5062d32027e4558bd094e043bb00fd5.png
[20210514001357731.png]: /images/20221021/ae55de58a76146d68f943087bbae6ded.png
[20210514001747109.png]: /images/20221021/f9951a4ddeae4661a7f7c7155e18301d.png
[20210514001821477.png]: /images/20221021/84e76c0d1815416da29899b5ced47532.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 36]: /images/20221021/424295f297dd4c00a8ea4a5b265dcf94.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 37]: /images/20221021/f92a3392afc04e4382df84fd1bad41b1.png
[20210514004103917.png]: /images/20221021/3e1e16077de5419ba1e2db1dfb118281.png
[20210514004135836.png]: /images/20221021/b6f6b598e23a4307b785ec2a82a99025.png
[20210514004529763.png]: /images/20221021/32f9b6aaf6754e50887ee503f45cf85d.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 38]: /images/20221021/5ead18bd6c3645fc94809253ccbc461b.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 39]: https://img-blog.csdnimg.cn/20210514005156196.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 40]: /images/20221021/59b8e566790a4d0b84950e9dbc263191.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 41]: /images/20221021/92ae5fab4ad940ea819e215f4f28d8af.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 42]: /images/20221021/2906d8370f664a789234f25f151dd217.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 43]: /images/20221021/84fef36089d8495e8fca94805b66e2a3.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 44]: /images/20221021/919cc958857c4949aa8ec04e19911549.png
[20210514003255262.png]: /images/20221021/a023be78b45f4f24b1101d742abaffd4.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 45]: /images/20221021/b6a113b4e4f6460eba228ff9467825cf.png
[20210514010434852.png]: /images/20221021/e4758869ce3b42e1afad078cbcdc2f1f.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 46]: /images/20221021/b54f9e8445de4fc2bef0645a2d43cf22.png
[2021051401175525.png]: /images/20221021/5303f5a7a76245ddab35ae512c3601b7.png
[20210514011812456.png]: /images/20221021/7c255ea4b88e45c6b51c11bf69e7bb77.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NhbnRldmVubA_size_16_color_FFFFFF_t_70 47]: /images/20221021/ed19f96352dd469eb3dbc424197ea081.png