kubernetes上部署jenkins并使用ingress-nginx提供域名访问

素颜马尾好姑娘i 2023-06-21 03:54 203阅读 0赞

## 在 K8S 中部署 jenkins ##

github上有一个1.5k star的项目：[https://github.com/jenkinsci/kubernetes-plugin][https_github.com_jenkinsci_kubernetes-plugin]

上面提供了jenkins在kubernetes中容器化的部署方式，以及yaml文件，现在直接 git clone 下来。

[root@master ~]# cd /work/
    [root@master work]# git clone https://github.com/jenkinsci/kubernetes-plugin.git

克隆完之后进入到这个目录。

[root@master work]# cd kubernetes-plugin/src/main/kubernetes/
    [root@master kubernetes]# ls
    jenkins.yml  service-account.yml

**直接克隆下来的文件，可能在某些k8s版本上会报错，请根据报错提示修改；**

> 下面的yaml在k8s v1.16.3版本上做过了测试。

可以看到有两个文件，service-account.yml 文件是创建 RBAC 授权相关的东西，这个不要动，主要看一下 jenkins.yml，需要修改一下，修改后的文件如下：

# jenkins
    
    ---
    apiVersion: apps/v1
    kind: StatefulSet
    metadata:
      name: jenkins
      labels:
        name: jenkins
    spec:
      selector:
        matchLabels:
          name: jenkins
      serviceName: jenkins
      replicas: 1
      updateStrategy:
        type: RollingUpdate
      template:
        metadata:
          name: jenkins
          labels:
            name: jenkins
        spec:
          terminationGracePeriodSeconds: 10
          serviceAccountName: jenkins
          containers:
            - name: jenkins
              command: 
              - /sbin/tini 
              - /usr/local/bin/jenkins.sh
              - --prefix=/jenkins
              image: jenkins/jenkins:lts-alpine
              imagePullPolicy: Always
              ports:
                - containerPort: 8080
                - containerPort: 50000
              resources:
                limits:
                  cpu: 1
                  memory: 1Gi
                requests:
                  cpu: 0.5
                  memory: 500Mi
              env:
                - name: LIMITS_MEMORY
                  valueFrom:
                    resourceFieldRef:
                      resource: limits.memory
                      divisor: 1Mi
                - name: JAVA_OPTS
                  # value: -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=1 -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85
                  value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85 -D--prefix=/jenkins
              volumeMounts:
                - name: jenkins-home
                  mountPath: /var/jenkins_home
              livenessProbe:
                httpGet:
                  path: /jenkins/login
                  port: 8080
                initialDelaySeconds: 60
                timeoutSeconds: 5
                failureThreshold: 12 # ~2 minutes
              readinessProbe:
                httpGet:
                  path: /jenkins/login
                  port: 8080
                initialDelaySeconds: 60
                timeoutSeconds: 5
                failureThreshold: 12 
          securityContext:
            fsGroup: 1000
          volumes:
          - hostPath:
              path: /data/jenkins
            name: jenkins-home
    
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: jenkins
    spec:
      selector:
        name: jenkins
      ports:
        -
          name: http
          port: 80
          targetPort: 8080
          protocol: TCP
        -
          name: agent
          port: 50000
          protocol: TCP
    
    ---
    apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
      name: jenkins
      annotations:
        nginx.ingress.kubernetes.io/ssl-redirect: "true"
        kubernetes.io/tls-acme: "true"
        # "413 Request Entity Too Large" uploading plugins, increase client_max_body_size
        nginx.ingress.kubernetes.io/proxy-body-size: 50m
        nginx.ingress.kubernetes.io/proxy-request-buffering: "off"
        # For nginx-ingress controller < 0.9.0.beta-18
        ingress.kubernetes.io/ssl-redirect: "true"
        # "413 Request Entity Too Large" uploading plugins, increase client_max_body_size
        ingress.kubernetes.io/proxy-body-size: 50m
        ingress.kubernetes.io/proxy-request-buffering: "off"
    spec:
      rules:
      - http:
          paths:
          - path: /jenkins
            backend:
              serviceName: jenkins
              servicePort: 80
        host: example.cn
      tls:
      - hosts:
        - example.cn
        secretName: litemall-admin-secret

注意到，上面yaml中主要修改ingress中的host为自己的域名，我的域名是在阿里备案过的域名（如果有国外的机器，不用备案也可以），而且申请了阿里的免费https证书，将证书配置到ingress-nginx中（通过创建一个secret，secret中包含了证书的信息）；

还有需要修改ingress中的path为带前缀的：`/jenkins`，因为对于入口网关:`ingress-nginx`，有很多路径为`/`的服务创建`ingress`去暴露，为了区分，所以通过带前缀的方式。这样就需要在Statefulset的podTemplate中配置jenkins的参数`--prefix=/jenkins`，这样Pod启动执行command：`/sbin/tini -- /usr/local/bin/jenkins.sh --prefix=/jenkins`  
后，`jenkins.sh`脚本中就会调用`java -jar jenkins.war --prefix=/jenkins`将参数传入。

默认`path`加了`/jenkins`前缀后，同时对应的修改健康检查中的`path`为`/jenkins/login`；

另外，jenkins的数据目录需要做持久化，也就是`/var/jenkins_home`，`jenkins` 所有的数据都是存在这个目录下面的。

如果没有公网域名，那就没必要创建Ingress，直接通过Service的NodePort方式去暴露，最后通过`{k8s集群任一主机IP}:{nodePort端口}/jenkins`方式去访问。

最后顺便贴出`Service Account`的配置，这里对github上的原有yaml没做修改：

# In GKE need to get RBAC permissions first with
    # kubectl create clusterrolebinding cluster-admin-binding --clusterrole=cluster-admin [--user=<user-name>|--group=<group-name>]
    
    ---
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: jenkins
    
    ---
    kind: Role
    apiVersion: rbac.authorization.k8s.io/v1beta1
    metadata:
      name: jenkins
    rules:
    - apiGroups: [""]
      resources: ["pods"]
      verbs: ["create","delete","get","list","patch","update","watch"]
    - apiGroups: [""]
      resources: ["pods/exec"]
      verbs: ["create","delete","get","list","patch","update","watch"]
    - apiGroups: [""]
      resources: ["pods/log"]
      verbs: ["get","list","watch"]
    - apiGroups: [""]
      resources: ["secrets"]
      verbs: ["get"]
    
    ---
    apiVersion: rbac.authorization.k8s.io/v1beta1
    kind: RoleBinding
    metadata:
      name: jenkins
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: Role
      name: jenkins
    subjects:
    - kind: ServiceAccount
      name: jenkins

最后通过域名去访问：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9saWFiaW8uYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70]

[https_github.com_jenkinsci_kubernetes-plugin]: https://github.com/jenkinsci/kubernetes-plugin
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9saWFiaW8uYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20191211104437325.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9saWFiaW8uYmxvZy5jc2RuLm5ldA==,size_16,color_FFFFFF,t_70