如何在K8S中部署redis集群并使用ceph作为持久化存储

拼搏现实的明天。 2023-05-22 06:35 10阅读 0赞

### 1、环境准备 ###

请安装完成K8S环境，此示例中我们有一个master：10.41.10.61，3个node：10.41.10.71-73，1个registry镜像仓库：10.41.10.81，1个ceph分布式存储集群：10.41.10.81-83

请在K8S所有节点上配置/etc/docker/daemon.json，添加：“insecure-registries”: \[“10.41.10.81:5000”\]

### 2、在镜像仓库服务上下载Percona\_xtradb\_cluster镜像 ###

docker pull redis
    docker tag docker.io/redis 10.41.10.81:5000/redis
    docker push 10.41.10.81:5000/redis #提交到镜像仓库

### 3、在master上部署2个服务 ###

cat <<eof >redis-master-service.yaml
    apiVersion: v1
    kind: Service
    metadata:
      name: redis-master-svc
    spec:
      selector:
        node: redis-master
      ports:
      - name: redis
        port: 6379
      type: NodePort
      externalIPs:
      - 10.41.10.60
    eof
    cat <<eof >redis-slave-service.yaml
    apiVersion: v1
    kind: Service
    metadata:
      name: redis-slave-svc
    spec:
      selector:
        node: redis-slave
      ports:
      - name: redis
        port: 6379
    eof
    kubectl apply -f redis-master-service.yaml #生效服务 
    kubectl apply -f redis-slave-service.yaml
    kubectl get svc #查看服务，这一步需要得到redis-master-svc的内部集群地址，后面添加slave节点时需要它作为master的IP地址

![在这里插入图片描述][20200416144155165.png]

### 4、添加两个节点的持久化存储 ###

cat <<eof >redis-master-pv-pvc.yaml
    apiVersion: v1
    kind: PersistentVolume
    metadata:
      name: redis-master-pv
    spec:
      capacity:
        storage: 20Gi
      accessModes:
        - ReadWriteMany
      cephfs:
        monitors:
          - 10.41.10.81:6789,10.41.10.82:6789,10.41.10.83:6789
        path: /redis/master
        user: admin
        readOnly: false
        secretRef:
          name: ceph-secret
      persistentVolumeReclaimPolicy: Recycle
    ---
    apiVersion: v1
    metadata:
      name: redis-master-pvc
    spec:
      accessModes:
        - ReadWriteMany
      volumeName: redis-master-pv
      resources:
        requests:
          storage: 20Gi
    eof
    cat <<eof >redis-slave-pv-pvc.yaml
    apiVersion: v1
    kind: PersistentVolume
    metadata:
      name: redis-slave-pv
    spec:
      capacity:
        storage: 20Gi
      accessModes:
        - ReadWriteMany
      cephfs:
        monitors:
          - 10.41.10.81:6789,10.41.10.82:6789,10.41.10.83:6789
        path: /redis/slave
        user: admin
        readOnly: false
        secretRef:
          name: ceph-secret
      persistentVolumeReclaimPolicy: Recycle
    ---
    kind: PersistentVolumeClaim
    apiVersion: v1
    metadata:
      name: redis-slave-pvc
    spec:
      accessModes:
        - ReadWriteMany
      volumeName: redis-slave-pv
      resources:
        requests:
          storage: 20Gi
    eof
    kubectl apply -f redis-master-pv-pvc.yaml #生效存储
    kubectl apply -f redis-slave-pv-pvc.yaml
    kubectl get pv #查看存储

![在这里插入图片描述][20200416144254772.png]

### 5、在存储中添加redis配置文件 ###

[我的下载里找名称：“k8s部署redis所需要的配置文件”][k8s_redis]

上传以上两个文件分别到存储目录下的master与slave文件下，并分别修改文件名为redis.conf  
![在这里插入图片描述][20200416144350864.png]  
这个dump.rdb只有已经进行数据快照才会产生，这里不用管它。

修改slave文件夹下面的redis.conf，主要关注以下几项

requirepass #这里配置redis的密码
    masterauth #这里配置slave的复制密码
    slaveof 10.111.80.141 6379 #这里的主节点就是要用上面我们查看到的k8s的redis-master-svc集群地址

修改master文件夹下面的redis.conf，注意slaveof这一项需要注释掉，其它同上。

### 6、K8S中添加部署 ###

cat <<eof >redis-master-deployment.yaml
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: redis-master
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: redis-master
      template:
        metadata:
          labels:
            app: redis-master
            node: redis-master
            unit: redis-cluster
        spec:
          containers:
          - name: redis-master
            image: 10.41.10.81:5000/redis
            volumeMounts:
            - mountPath: "/data"
              name: data
            ports:
            - containerPort: 6379
            command: ["redis-server"]
            args: ["/data/redis.conf"]
          volumes:
          - name: data
            persistentVolumeClaim:
              claimName: redis-master-pvc
    eof
    cat <<eof >redis-slave-deployment.yaml
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: redis-slave
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: redis-slave
      template:
        metadata:
          labels:
            app: redis-slave
            node: redis-slave
            unit: redis-cluster
        spec:
          containers:
          - name: redis-slave
            image: 10.41.10.81:5000/redis
            command: ["redis-server"]
            args: ["/data/redis.conf"]
            volumeMounts:
            - mountPath: "/data"
              name: data
            ports:
            - containerPort: 6379
          volumes:
          - name: data
            persistentVolumeClaim:
              claimName: redis-slave-pvc
    eof
    kubectl apply -f redis-master-deployment.yaml #部署主节点
    kubectl apply -f redis-slave-deployment.yaml #部署次节点
    kubectl get pods -o wide #查看部署情况

### 7、验证 ###

分别进入pod，查看集群状态

kubectl exec -it redis-master-75fdfdb8f8-bqwmx /bin/bash
    redis-cli
    auth redis@123 #输入密码
    info replication #查看集群

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2Noc2NvbWZhbmVy_size_16_color_FFFFFF_t_70]  
进入slave节点

kubectl exec -it redis-slave-7b764f878b-tfrdp /bin/bash  
redis-cli  
auth redis@123 \#输入密码  
info replication \#查看集群  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2Noc2NvbWZhbmVy_size_16_color_FFFFFF_t_70 1]  
大功告成！

[20200416144155165.png]: /images/20230521/a5166d50443a49f6864c77dfc1a3aff1.png
[20200416144254772.png]: /images/20230521/42bd08daceea4409beb4d4d22990e481.png
[k8s_redis]: https://download.csdn.net/download/chscomfaner/12331302
[20200416144350864.png]: /images/20230521/ad58122a96d743d9a9d8fe0299bfba09.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2Noc2NvbWZhbmVy_size_16_color_FFFFFF_t_70]: /images/20230521/fcf36870a19f49769fa1858358e1d139.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2Noc2NvbWZhbmVy_size_16_color_FFFFFF_t_70 1]: /images/20230521/cc83f4d53a3c4599ba6322c2ab5928c2.png