k8s上使用operator部署redis集群

小灰灰 2022-11-02 13:21 826阅读 0赞

## 通过operator部署redis集群 ##

operator部署有状态的应用会简单很多，GitHub已有很多写好的operator：https://github.com/operator-framework/awesome-operators  
这里直接参考ucloud的operator部署redis集群：https://github.com/ucloud/redis-cluster-operator  
Redis Cluster Operator在Kubernetes上管理Redis-Cluster集群  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0hscm9saXU_size_16_color_FFFFFF_t_70]  
每个主节点及其从节点都由statefulSet管理，为每个statefulSet创建无头svc，并为所有节点创建clusterIP服务。  
每个有状态集都使用PodAntiAffinity来确保主节点和从节点分散在不同的节点上。同时，当操作员在每个有状态集中选择主节点时，它会优先选择具有不同k8s节点的容器作为主节点。  
从GitHub上下载项目  
git clone https://github.com/ucloud/redis-cluster-operator.git  
或者直接通过浏览器下载压缩包，上传到k8s集群![在这里插入图片描述][20210225160820352.png]  
在名称空间：redis-cluster下部署Redis集群，注意修改yaml文件的namespace参数  
创建名称空间Namespace：redis-cluster

]# cat ns-redis-cluster.yaml
    apiVersion: v1
    kind: Namespace
    metadata:
      name: redis-cluster
      
    ]# kubectl apply -f ns-redis-cluster.yaml
    namespace/redis-cluster created
    kubectl get ns
    'NAME                   STATUS   AGE
    default                Active   1d
    ingress-nginx          Active   1d
    kube-node-lease        Active   1d
    kube-public            Active   1d
    kube-system            Active   1d
    kubernetes-dashboard   Active   1d
    redis-cluster          Active   18s

创建自定义资源（CRD）

]# cd redis-cluster-operator/ 
    kubectl apply -f deploy/crds/

创建operator

]# cd deploy/
    ]# kubectl apply -f service_account.yaml 
    ]# cd cluster/
    ]# kubectl apply -f cluster_role.yaml 
    ]# kubectl apply -f cluster_role_binding.yaml 
    ]# kubectl apply -f operator.yaml 
    ]# kubectl get pod -n redis-cluster 
    NAME                                     READY   STATUS    RESTARTS   AGE
    redis-cluster-operator-d4b695fd6-9vc85   1/1     Running   0          33s

创建redis集群，预先配置了动态PV存储，并设置资源限制  
默认创建三主三从集群

]# cat redis-cluster.yaml 
    apiVersion: redis.kun/v1alpha1
    kind: DistributedRedisCluster
    metadata:
      annotations:
        # if your operator run as cluster-scoped, add this annotations
        redis.kun/scope: cluster-scoped
      name: example-distributedrediscluster
      namespace: redis-cluster
    spec:
      image: redis:5.0.4-alpine
      imagePullPolicy: IfNotPresent
      masterSize: 3			#master节点数量
      clusterReplicas: 1	#每个master节点的从节点数量
      serviceName: redis-svc
      # resources config
      resources:
        limits:
          cpu: 300m
          memory: 200Mi
        requests:
          cpu: 200m
          memory: 150Mi
      # pv storage
      storage:
        type: persistent-claim
        size: 2Gi
        class: nfs-storage
        deleteClaim: true
    ]# kubectl apply -f redis-cluster.yaml

查看创建的pod，svc，pvc，pv

]# kubectl get pod,svc,pvc,pv -n redis-cluster 
    NAME                                          READY   STATUS    RESTARTS   AGE
    pod/drc-example-distributedrediscluster-0-0   1/1     Running   0          11m
    pod/drc-example-distributedrediscluster-0-1   1/1     Running   0          10m
    pod/drc-example-distributedrediscluster-1-0   1/1     Running   0          11m
    pod/drc-example-distributedrediscluster-1-1   1/1     Running   0          10m
    pod/drc-example-distributedrediscluster-2-0   1/1     Running   0          11m
    pod/drc-example-distributedrediscluster-2-1   1/1     Running   0          10m
    pod/redis-cluster-operator-d4b695fd6-9vc85    1/1     Running   0          17m
    
    NAME                                     TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)              AGE
    service/redis-cluster-operator-metrics   ClusterIP   10.108.239.105   <none>        8383/TCP,8686/TCP    17m
    service/redis-svc                        ClusterIP   10.104.55.186    <none>        6379/TCP,16379/TCP   11m
    service/redis-svc-0                      ClusterIP   None             <none>        6379/TCP,16379/TCP   11m
    service/redis-svc-1                      ClusterIP   None             <none>        6379/TCP,16379/TCP   11m
    service/redis-svc-2                      ClusterIP   None             <none>        6379/TCP,16379/TCP   11m
    
    NAME                                                                       STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
    persistentvolumeclaim/redis-data-drc-example-distributedrediscluster-0-0   Bound    pvc-8da269a1-d704-40ee-9034-59c242229c42   2Gi        RWO            nfs-storage    11m
    persistentvolumeclaim/redis-data-drc-example-distributedrediscluster-0-1   Bound    pvc-2f90ccb0-87d2-4152-a292-d672941aa27a   2Gi        RWO            nfs-storage    10m
    persistentvolumeclaim/redis-data-drc-example-distributedrediscluster-1-0   Bound    pvc-e34aeac8-a555-42e8-bd6f-d6e84e417780   2Gi        RWO            nfs-storage    11m
    persistentvolumeclaim/redis-data-drc-example-distributedrediscluster-1-1   Bound    pvc-22a46a79-965b-41e7-9c5b-e314b158488a   2Gi        RWO            nfs-storage    10m
    persistentvolumeclaim/redis-data-drc-example-distributedrediscluster-2-0   Bound    pvc-78640340-27a9-45a9-ae3c-de68405e00d4   2Gi        RWO            nfs-storage    11m
    persistentvolumeclaim/redis-data-drc-example-distributedrediscluster-2-1   Bound    pvc-0ef4b886-38c2-43b6-aef3-e78bdd0c71de   2Gi        RWO            nfs-storage    10m
    
    NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                                              STORAGECLASS   REASON   AGE
    persistentvolume/pvc-0ef4b886-38c2-43b6-aef3-e78bdd0c71de   2Gi        RWO            Retain           Bound    redis-cluster/redis-data-drc-example-distributedrediscluster-2-1   nfs-storage             10m
    persistentvolume/pvc-22a46a79-965b-41e7-9c5b-e314b158488a   2Gi        RWO            Retain           Bound    redis-cluster/redis-data-drc-example-distributedrediscluster-1-1   nfs-storage             10m
    persistentvolume/pvc-2f90ccb0-87d2-4152-a292-d672941aa27a   2Gi        RWO            Retain           Bound    redis-cluster/redis-data-drc-example-distributedrediscluster-0-1   nfs-storage             10m
    persistentvolume/pvc-78640340-27a9-45a9-ae3c-de68405e00d4   2Gi        RWO            Retain           Bound    redis-cluster/redis-data-drc-example-distributedrediscluster-2-0   nfs-storage             11m
    persistentvolume/pvc-8da269a1-d704-40ee-9034-59c242229c42   2Gi        RWO            Retain           Bound    redis-cluster/redis-data-drc-example-distributedrediscluster-0-0   nfs-storage             11m
    persistentvolume/pvc-e34aeac8-a555-42e8-bd6f-d6e84e417780   2Gi        RWO            Retain           Bound    redis-cluster/redis-data-drc-example-distributedrediscluster-1-0   nfs-storage             11m

验证redis集群

]# kubectl exec -it -n redis-cluster drc-example-distributedrediscluster-0-0 -- sh
    /data # redis-cli -c -h redis-svc
    redis-svc:6379> cluster info
    cluster_state:ok
    cluster_slots_assigned:16384
    cluster_slots_ok:16384
    cluster_slots_pfail:0
    cluster_slots_fail:0
    cluster_known_nodes:6
    cluster_size:3
    cluster_current_epoch:5
    cluster_my_epoch:5
    cluster_stats_messages_ping_sent:225
    cluster_stats_messages_pong_sent:237
    cluster_stats_messages_sent:462
    cluster_stats_messages_ping_received:232
    cluster_stats_messages_pong_received:225
    cluster_stats_messages_meet_received:5
    cluster_stats_messages_received:462
    redis-svc:6379> set a 100
    -> Redirected to slot [15495] located at 10.244.166.135:6379
    OK
    10.244.166.135:6379> get a
    "100"

删除redis集群

]# cd redis-cluster-operator/ 
    ]# kubectl delete -f redis-cluster.yaml
    ]# cd cluster/
    ]# kubectl delete -f operator.yaml 
    ]# kubectl delete -f cluster_role_binding.yaml 
    ]# kubectl delete -f cluster_role.yaml 
    ]# kubectl delete-f service_account.yaml 
    ]# kubectl delete -f deploy/crds/
    ]# kubectl delete -f ns-redis-cluster.yaml

# yaml文件 #

## ns-redis-cluster.yaml ##

]# cat ns-redis-cluster.yaml 
    apiVersion: v1
    kind: Namespace
    metadata:
      name: redis-cluster

## redis.kun\_distributedredisclusters\_crd.yaml ##

]# cat redis.kun_distributedredisclusters_crd.yaml 
    apiVersion: apiextensions.k8s.io/v1beta1
    kind: CustomResourceDefinition
    metadata:
      name: distributedredisclusters.redis.kun
      namespace: redis-cluster
    spec:
      group: redis.kun
      names:
        kind: DistributedRedisCluster
        listKind: DistributedRedisClusterList
        plural: distributedredisclusters
        singular: distributedrediscluster
        shortNames:
        - drc
      scope: Namespaced
      additionalPrinterColumns:
      - JSONPath: .spec.masterSize
        description: The number of redis master node in the ensemble
        name: MasterSize
        type: integer
      - JSONPath: .status.status
        description: The status of redis cluster
        name: Status
        type: string
      - JSONPath: .metadata.creationTimestamp
        name: Age
        type: date
      - JSONPath: .status.numberOfMaster
        priority: 1
        description: The current master number of redis cluster
        name: CurrentMasters
        type: integer
      - JSONPath: .spec.image
        priority: 1
        description: The image of redis cluster
        name: Images
        type: string
      subresources:
        status: { }
      validation:
        openAPIV3Schema:
          description: DistributedRedisCluster is the Schema for the distributedredisclusters
            API
          properties:
            apiVersion:
              description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources'
              type: string
            kind:
              description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds'
              type: string
            metadata:
              type: object
            spec:
              description: DistributedRedisClusterSpec defines the desired state of
                DistributedRedisCluster
              properties:
                masterSize:
                  format: int32
                  type: integer
                  minimum: 3
                  maximum: 10
                clusterReplicas:
                  format: int32
                  type: integer
                  minimum: 1
                  maximum: 3
                serviceName:
                  type: string
                  pattern: '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*'
              type: object
            status:
              description: DistributedRedisClusterStatus defines the observed state
                of DistributedRedisCluster
              type: object
          type: object
      version: v1alpha1
      versions:
      - name: v1alpha1
        served: true
        storage: true

## redis.kun\_redisclusterbackups\_crd.yaml ##

]# cat redis.kun_redisclusterbackups_crd.yaml 
    apiVersion: apiextensions.k8s.io/v1beta1
    kind: CustomResourceDefinition
    metadata:
      name: redisclusterbackups.redis.kun
      namespace: redis-cluster
    spec:
      group: redis.kun
      names:
        kind: RedisClusterBackup
        listKind: RedisClusterBackupList
        plural: redisclusterbackups
        singular: redisclusterbackup
        shortNames:
          - drcb
      scope: Namespaced
      additionalPrinterColumns:
        - JSONPath: .metadata.creationTimestamp
          name: Age
          type: date
        - JSONPath: .status.phase
          description: The phase of redis cluster backup
          name: Phase
          type: string
      subresources:
        status: { }
      versions:
        - name: v1alpha1
          # Each version can be enabled/disabled by Served flag.
          served: true
          # One and only one version must be marked as the storage version.
          storage: true
      validation:
        openAPIV3Schema:
          description: RedisClusterBackup is the Schema for the redisclusterbackups
            API
          properties:
            apiVersion:
              description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources'
              type: string
            kind:
              description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds'
              type: string
            metadata:
              type: object
            spec:
              description: RedisClusterBackupSpec defines the desired state of RedisClusterBackup
              type: object
            status:
              description: RedisClusterBackupStatus defines the observed state of RedisClusterBackup
              type: object
          type: object

## service\_account.yaml ##

]# cat service_account.yaml 
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: redis-cluster-operator
      namespace: redis-cluster

## cluster\_role.yaml ##

]# cat cluster_role.yaml 
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      name: redis-cluster-operator
    rules:
      - apiGroups:
          - ""
        resources:
          - pods
          - secrets
          - endpoints
          - persistentvolumeclaims
        verbs:
          - get
          - list
          - watch
          - delete
      - apiGroups:
          - ""
        resources:
          - configmaps
          - pods/exec
          - secrets
          - services
          - events
          - persistentvolumeclaims
        verbs:
          - create
          - get
          - list
          - patch
          - update
          - watch
          - delete
      - apiGroups:
          - ""
        resources:
          - namespaces
        verbs:
          - get
          - list
          - watch
      - apiGroups:
        - batch
        resources:
          - jobs
        verbs:
          - create
          - get
          - list
          - patch
          - update
          - watch
          - delete
      - apiGroups:
          - apps
        resources:
          - deployments
          - replicasets
          - statefulsets
        verbs:
          - create
          - get
          - list
          - patch
          - update
          - watch
          - delete
      - apiGroups:
          - policy
        resources:
          - poddisruptionbudgets
        verbs:
          - create
          - get
          - list
          - patch
          - update
          - watch
          - delete
      - apiGroups:
          - apps
        resourceNames:
          - redis-operator
        resources:
          - deployments/finalizers
        verbs:
          - update
      - apiGroups:
          - redis.kun
        resources:
          - '*'
          - redisclusterbackups
        verbs:
          - delete
          - deletecollection
          - get
          - list
          - patch
          - update
          - watch

## cluster\_role\_binding.yaml ##

]# cat cluster_role_binding.yaml 
    kind: ClusterRoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: redis-cluster-operator
    subjects:
    - kind: ServiceAccount
      name: redis-cluster-operator
      namespace: redis-cluster
    roleRef:
      kind: ClusterRole
      name: redis-cluster-operator
      apiGroup: rbac.authorization.k8s.io

## operator.yaml ##

]# cat operator.yaml 
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: redis-cluster-operator
      namespace: redis-cluster
    spec:
      replicas: 1
      selector:
        matchLabels:
          name: redis-cluster-operator
      template:
        metadata:
          labels:
            name: redis-cluster-operator
        spec:
          serviceAccountName: redis-cluster-operator
          containers:
            - name: redis-cluster-operator
              # Replace this with the built image name
              image: fishu/redis-cluster-operator:latest
              imagePullPolicy: IfNotPresent
              command:
              - redis-cluster-operator
              args:
              - --rename-command-path=/etc/redisconf
              - --rename-command-file=redis.conf
              imagePullPolicy: Always
              env:
                - name: WATCH_NAMESPACE
                  value: ""
                - name: POD_NAME
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.name
                - name: OPERATOR_NAME
                  value: "redis-cluster-operator"
              volumeMounts:
                - name: redisconf
                  mountPath: /etc/redisconf
          volumes:
            - name: redisconf
              configMap:
                name: redis-admin
    ---
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: redis-admin
      namespace: redis-cluster
    data:
      redis.conf: |-
        rename-command CONFIG lni07z1p
        rename-command BGSAVE pp14qluk
        rename-command DEBUG 8a4insyv
        rename-command SAVE 6on30p6z
        rename-command SHUTDOWN dvui0opr
        rename-command SLAVEOF xwxvcw36
        rename-command BGREWRITEAOF www07fko

## redis-cluster.yaml ##

]# cat redis-cluster.yaml 
    ---
    apiVersion: redis.kun/v1alpha1
    kind: DistributedRedisCluster
    metadata:
      annotations:
        # if your operator run as cluster-scoped, add this annotations
        redis.kun/scope: cluster-scoped
      name: example-distributedrediscluster
      namespace: redis-cluster
    spec:
      image: redis:5.0.4-alpine
      imagePullPolicy: IfNotPresent
      masterSize: 3
      clusterReplicas: 1
      serviceName: redis-svc
      # resources config
      resources:
        limits:
          cpu: 300m
          memory: 200Mi
        requests:
          cpu: 200m
          memory: 150Mi
      # pv storage
      storage:
        type: persistent-claim
        size: 2Gi
        class: nfs-storage
        deleteClaim: true

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0hscm9saXU_size_16_color_FFFFFF_t_70]: /images/20221024/6e735330ff374c35aac25b90ef571871.png
[20210225160820352.png]: /images/20221024/0dde9012fc4a4895a4f10589887d16ef.png