[kubernetes]-kubernetes+nfs创建高可用mysql

迈不过友情╰ 2022-12-09 04:51 292阅读 0赞

安装mysql主从玩一下。

1 创建rabc rabc-1.yaml

  1. ---
  2. # 唯一需要修改的地方只有namespace,根据实际情况定义
  3. apiVersion: v1
  4. kind: ServiceAccount
  5. metadata:
  6.   name: nfs-client-provisioner
  7.   # replace with namespace where provisioner is deployed
  8.   namespace: default        #根据实际环境设定namespace,下面类同
  9. ---
  10. kind: ClusterRole
  11. apiVersion: rbac.authorization.k8s.io/v1
  12. metadata:
  13.   name: nfs-client-provisioner-runner
  14. rules:
  15.   - apiGroups: [""]
  16.     resources: ["persistentvolumes"]
  17.     verbs: ["get", "list", "watch", "create", "delete"]
  18.   - apiGroups: [""]
  19.     resources: ["persistentvolumeclaims"]
  20.     verbs: ["get", "list", "watch", "update"]
  21.   - apiGroups: ["storage.k8s.io"]
  22.     resources: ["storageclasses"]
  23.     verbs: ["get", "list", "watch"]
  24.   - apiGroups: [""]
  25.     resources: ["events"]
  26.     verbs: ["create", "update", "patch"]
  27. ---
  28. kind: ClusterRoleBinding
  29. apiVersion: rbac.authorization.k8s.io/v1
  30. metadata:
  31.   name: run-nfs-client-provisioner
  32. subjects:
  33.   - kind: ServiceAccount
  34.     name: nfs-client-provisioner
  35.     # replace with namespace where provisioner is deployed
  36.     namespace: default
  37. roleRef:
  38.   kind: ClusterRole
  39.   name: nfs-client-provisioner-runner
  40.   apiGroup: rbac.authorization.k8s.io
  41. ---
  42. kind: Role
  43. apiVersion: rbac.authorization.k8s.io/v1
  44. metadata:
  45.   name: leader-locking-nfs-client-provisioner
  46.     # replace with namespace where provisioner is deployed
  47.   namespace: default
  48. rules:
  49.   - apiGroups: [""]
  50.     resources: ["endpoints"]
  51.     verbs: ["get", "list", "watch", "create", "update", "patch"]
  52. ---
  53. kind: RoleBinding
  54. apiVersion: rbac.authorization.k8s.io/v1
  55. metadata:
  56.   name: leader-locking-nfs-client-provisioner
  57. subjects:
  58.   - kind: ServiceAccount
  59.     name: nfs-client-provisioner
  60.     # replace with namespace where provisioner is deployed
  61.     namespace: default
  62. roleRef:
  63.   kind: Role
  64.   name: leader-locking-nfs-client-provisioner
  65.   apiGroup: rbac.authorization.k8s.io

2 创建nfsStorageClass nfs-StorageClass-2.yaml

  1. apiVersion: storage.k8s.io/v1
  2. kind: StorageClass
  3. metadata:
  4.   name: managed-nfs-storage
  5. provisioner: mysql-nfs-storage #这里的名称要和provisioner配置文件中的环境变量PROVISIONER_NAME保持一致parameters:  archiveOnDelete: "false"

这里需要先创建好nfs server

3 创建nfs-provisioner nfs-provisioner-3.yaml

  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata:
  4.   name: nfs-client-provisioner
  5.   labels:
  6.     app: nfs-client-provisioner
  7.   # replace with namespace where provisioner is deployed
  8.   namespace: default  #与RBAC文件中的namespace保持一致
  9. spec:
  10.   replicas: 1
  11.   selector:
  12.     matchLabels:
  13.       app: nfs-client-provisioner
  14.   strategy:
  15.     type: Recreate
  16.   selector:
  17.     matchLabels:
  18.       app: nfs-client-provisioner
  19.   template:
  20.     metadata:
  21.       labels:
  22.         app: nfs-client-provisioner
  23.     spec:
  24.       serviceAccountName: nfs-client-provisioner
  25.       containers:
  26.         - name: nfs-client-provisioner
  27.           image: quay.io/external_storage/nfs-client-provisioner:latest
  28.           volumeMounts:
  29.             - name: nfs-client-root
  30.               mountPath: /persistentvolumes
  31.           env:
  32.             - name: PROVISIONER_NAME
  33.               value: mysql-nfs-storage  #provisioner名称,请确保该名称与 nfs-StorageClass.yaml文件中的provisioner名称保持一致
  34.             - name: NFS_SERVER
  35.               value: 192.168.1.171   #NFS Server IP地址
  36.             - name: NFS_PATH
  37.               value: /srv/data    #NFS挂载卷
  38.       volumes:
  39.         - name: nfs-client-root
  40.           nfs:
  41.             server: 192.168.1.171  #NFS Server IP地址
  42.             path: /srv/data     #NFS 挂载卷

4 测试一下 创建test-claim-4.yaml

  1. kind: PersistentVolumeClaim
  2. apiVersion: v1
  3. metadata:
  4.   name: test-claim
  5.   annotations:
  6.     volume.beta.kubernetes.io/storage-class: "managed-nfs-storage"   #与nfs-StorageClass.yaml metadata.name保持一致
  7. spec:
  8.   accessModes:
  9.     - ReadWriteMany
  10.   resources:
  11.     requests:
  12.       storage: 1Mi

5 测试一下 创建test-pod-5.yaml

  1. kind: Pod
  2. apiVersion: v1
  3. metadata:
  4.   name: test-pod
  5. spec:
  6.   containers:
  7.   - name: test-pod
  8.     image: busybox:1.24
  9.     command:
  10.       - "/bin/sh"
  11.     args:
  12.       - "-c"
  13.       - "touch /mnt/SUCCESS && exit 0 || exit 1"   #创建一个SUCCESS文件后退出
  14.     volumeMounts:
  15.       - name: nfs-pvc
  16.         mountPath: "/mnt"
  17.   restartPolicy: "Never"
  18.   volumes:
  19.     - name: nfs-pvc
  20.       persistentVolumeClaim:
  21.         claimName: test-claim  #与PVC名称保持一致

36ff84ba73b14b2cb6fa98a5663b4a0a.png

在nfs 生成了SUCCESS文件

8250b4be621d58be3f0d47e8d815953a.png

6 测试一下 nginx-statefulset-6.yaml

  1. # 这个我自己没测试
  2. ---
  3. apiVersion: v1
  4. kind: Service
  5. metadata:
  6.   name: nginx-headless
  7.   labels:
  8.     app: nginx
  9. spec:
  10.   ports:
  11.   - port: 80
  12.     name: web
  13.   clusterIP: None   #注意此处的值,None表示无头服务
  14.   selector:
  15.     app: nginx
  16. ---
  17. apiVersion: apps/v1beta1
  18. kind: StatefulSet
  19. metadata:
  20.   name: web
  21. spec:
  22.   serviceName: "nginx"
  23.   replicas: 2  #两个副本
  24.   template:
  25.     metadata:
  26.       labels:
  27.         app: nginx
  28.     spec:
  29.       containers:
  30.       - name: nginx
  31.         image: ikubernetes/myapp:v1
  32.         ports:
  33.         - containerPort: 80
  34.           name: web
  35.         volumeMounts:
  36.         - name: www
  37.           mountPath: /usr/share/nginx/html
  38.   volumeClaimTemplates:
  39.   - metadata:
  40.       name: www
  41.       annotations:
  42.         volume.beta.kubernetes.io/storage-class: "managed-nfs-storage"   #managed-nfs-storage为我们创建的storage-class名称
  43.     spec:
  44.       accessModes: [ "ReadWriteOnce" ]
  45.       resources:
  46.         requests:
  47.           storage: 1Gi

d25b56dfa3791b27ee947923ecfc312a.png

7 创建mysql配置文件 cm-mysql-7.yaml

  1. apiVersion: v1
  2. kind: ConfigMap
  3. metadata:
  4.   name: mysql
  5.   labels:
  6.     app: mysql
  7. data:
  8.   master.cnf: |
  9.     # Apply this config only on the master.
  10.     [mysqld]
  11.     log-bin
  12.     log_bin_trust_function_creators=1
  13.     lower_case_table_names=1
  14.   slave.cnf: |
  15.     # Apply this config only on slaves.
  16.     [mysqld]
  17.     super-read-only
  18.     log_bin_trust_function_creators=1

8 创建mysql 的svc svc-mysql-8.yaml

  1. # Headless service for stable DNS entries of StatefulSet members.
  2. apiVersion: v1
  3. kind: Service
  4. metadata:
  5.   name: mysql
  6.   labels:
  7.     app: mysql
  8. spec:
  9.   ports:
  10.   - name: mysql
  11.     port: 3306
  12.   clusterIP: None
  13.   selector:
  14.     app: mysql
  15. ---
  16. # Client service for connecting to any MySQL instance for reads.
  17. # For writes, you must instead connect to the master: mysql-0.mysql.
  18. apiVersion: v1
  19. kind: Service
  20. metadata:
  21.   name: mysql-read
  22.   labels:
  23.     app: mysql
  24. spec:
  25.   ports:
  26.   - name: mysql
  27.     port: 3306
  28.   selector:
  29.     app: mysql

可以看到，这两个 Service 都代理了所有携带 app=mysql 标签的 Pod，也就是所有的 MySQL Pod。端口映射都是用 Service 的 3306 端口对应 Pod 的 3306 端口。

不同的是，第一个名叫“mysql”的 Service 是一个 Headless Service（即：clusterIP= None）。所以它的作用，是通过为 Pod 分配 DNS 记录来固定它的拓扑状态，比如“mysql-0.mysql”和“mysql-1.mysql”这样的 DNS 名字。其中，编号为 0 的节点就是我们的主节点。

而第二个名叫“mysql-read”的 Service，则是一个常规的 Service。

并且我们规定，所有用户的读请求，都必须访问第二个 Service 被自动分配的 DNS 记录，即：“mysql-read”（当然，也可以访问这个 Service 的 VIP）。这样，读请求就可以被转发到任意一个 MySQL 的主节点或者从节点上。

备注：Kubernetes 中的所有 Service、Pod 对象，都会被自动分配同名的 DNS 记录。并且读请求应该是分配到所有mysql节点，不是网上博客说的读是到read的slave是那个。

而所有用户的写请求，则必须直接以 DNS 记录的方式访问到 MySQL 的主节点，也就是：“mysql-0.mysql“这条 DNS 记录。

9 创建mysql的sts sts-mysql-9.yaml

  1. apiVersion: apps/v1
  2. kind: StatefulSet
  3. metadata:
  4.   name: mysql
  5. spec:
  6.   selector:
  7.     matchLabels:
  8.       app: mysql
  9.   serviceName: mysql
  10.   replicas: 3
  11.   volumeClaimTemplates:
  12.   - metadata:
  13.       name: data
  14.       annotations:
  15.         volume.beta.kubernetes.io/storage-class: "managed-nfs-storage"
  16.     spec:
  17.       accessModes: [ "ReadWriteOnce" ]
  18.       resources:
  19.         requests:
  20.           storage: 10Gi
  21.   template:
  22.     metadata:
  23.       labels:
  24.         app: mysql
  25.     spec:
  26.       initContainers:
  27.       - name: init-mysql
  28.         image: registry.cn-hangzhou.aliyuncs.com/xjm_public/mysql:5.7
  29.         command:
  30.         - bash
  31.         - "-c"
  32.         - |
  33.           set -ex
  34.           [[ `hostname` =~ -([0-9]+)$ ]] || exit 1
  35.           ordinal=${BASH_REMATCH[1]}
  36.           echo [mysqld] > /mnt/conf.d/server-id.cnf
  37.           echo server-id=$((100 + $ordinal)) >> /mnt/conf.d/server-id.cnf
  38.           if [[ $ordinal -eq 0 ]]; then
  39.             cp /mnt/config-map/master.cnf /mnt/conf.d/
  40.           else
  41.             cp /mnt/config-map/slave.cnf /mnt/conf.d/
  42.           fi
  43.         volumeMounts:
  44.         - name: conf
  45.           mountPath: /mnt/conf.d
  46.         - name: config-map
  47.           mountPath: /mnt/config-map
  48.       - name: clone-mysql
  49.         image: registry.cn-hangzhou.aliyuncs.com/xjm_public/xtrabackup:1.0
  50.         command:
  51.         - bash
  52.         - "-c"
  53.         - |
  54.           set -ex
  55.           [[ -d /var/lib/mysql/mysql ]] && exit 0
  56.           [[ `hostname` =~ -([0-9]+)$ ]] || exit 1
  57.           ordinal=${BASH_REMATCH[1]}
  58.           [[ $ordinal -eq 0 ]] && exit 0
  59.           ncat --recv-only mysql-$(($ordinal-1)).mysql 3307 | xbstream -x -C /var/lib/mysql
  60.           xtrabackup --prepare --target-dir=/var/lib/mysql
  61.         volumeMounts:
  62.         - name: data
  63.           mountPath: /var/lib/mysql
  64.           subPath: mysql
  65.         - name: conf
  66.           mountPath: /etc/mysql/conf.d
  67.       containers:
  68.       - name: mysql
  69.         image: registry.cn-hangzhou.aliyuncs.com/xjm_public/mysql:5.7
  70.         env:
  71.         - name: MYSQL_ALLOW_EMPTY_PASSWORD
  72.           value: "1"
  73.         ports:
  74.         - name: mysql
  75.           containerPort: 3306
  76.         volumeMounts:
  77.         - name: data
  78.           mountPath: /var/lib/mysql
  79.           subPath: mysql
  80.         - name: conf
  81.           mountPath: /etc/mysql/conf.d
  82.         resources:
  83.           requests:
  84.             cpu: 500m
  85.             memory: 1Gi
  86.         livenessProbe:
  87.           exec:
  88.             command: ["mysqladmin", "ping"]
  89.           initialDelaySeconds: 30
  90.           periodSeconds: 10
  91.           timeoutSeconds: 5
  92.         readinessProbe:
  93.           exec:
  94.             command: ["mysql", "-h", "127.0.0.1", "-e", "SELECT 1"]
  95.           initialDelaySeconds: 5
  96.           periodSeconds: 2
  97.           timeoutSeconds: 1
  98.       - name: xtrabackup
  99.         image: registry.cn-hangzhou.aliyuncs.com/xjm_public/xtrabackup:1.0
  100.         ports:
  101.         - name: xtrabackup
  102.           containerPort: 3307
  103.         command:
  104.         - bash
  105.         - "-c"
  106.         - |
  107.           set -ex
  108.           cd /var/lib/mysql
  109.           if [[ -f xtrabackup_slave_info ]]; then
  110.             mv xtrabackup_slave_info change_master_to.sql.in
  111.             rm -f xtrabackup_binlog_info
  112.           elif [[ -f xtrabackup_binlog_info ]]; then
  113.             [[ `cat xtrabackup_binlog_info` =~ ^(.*?)[[:space:]]+(.*?)$ ]] || exit 1
  114.             rm xtrabackup_binlog_info
  115.             echo "CHANGE MASTER TO MASTER_LOG_FILE='${BASH_REMATCH[1]}',\
  116.                   MASTER_LOG_POS=${BASH_REMATCH[2]}" > change_master_to.sql.in
  117.           fi
  118.           if [[ -f change_master_to.sql.in ]]; then
  119.             echo "Waiting for mysqld to be ready (accepting connections)"
  120.             until mysql -h 127.0.0.1 -e "SELECT 1"; do sleep 1; done
  121.             echo "Initializing replication from clone position"
  122.             mv change_master_to.sql.in change_master_to.sql.orig
  123.             mysql -h 127.0.0.1 <<EOF
  124.           $(<change_master_to.sql.orig),
  125.             MASTER_HOST='mysql-0.mysql',
  126.             MASTER_USER='root',
  127.             MASTER_PASSWORD='',
  128.             MASTER_CONNECT_RETRY=10;
  129.           START SLAVE;
  130.           EOF
  131.           fi
  132.           exec ncat --listen --keep-open --send-only --max-conns=1 3307 -c \
  133.             "xtrabackup --backup --slave-info --stream=xbstream --host=127.0.0.1 --user=root"
  134.         volumeMounts:
  135.         - name: data
  136.           mountPath: /var/lib/mysql
  137.           subPath: mysql
  138.         - name: conf
  139.           mountPath: /etc/mysql/conf.d
  140.         resources:
  141.           requests:
  142.             cpu: 100m
  143.             memory: 100Mi
  144.       volumes:
  145.       - name: conf
  146.         emptyDir: {}
  147.       - name: config-map
  148.         configMap:
  149.           name: mysql

5589dded241eb6049008332e11233b90.png

测试服务 启动可能比较慢

  1. kubectl run mysql-client --image=registry.cn-hangzhou.aliyuncs.com/xjm_public/mysql:5.7 -it --rm --restart=Never -- mysql -h mysql-0.mysql.default

dad76e0e209b344b084acae1ddcce5c0.png

  1. CREATE DATABASE demo;
  2. CREATE TABLE demo.messages (message VARCHAR(250));
  3. INSERT INTO demo.messages VALUES ('hello');

查看主机名为mysql-read的数据

  1. kubectl run mysql-client2 --image=registry.cn-hangzhou.aliyuncs.com/xjm_public/mysql:5.7 -i -t --rm --restart=Never -- mysql -h mysql-read.default
  3. select * from demo.messages;

70f424dcfa8389516a272982a8d81945.png

最后参考博客里的有些错误描述 需要纠正一下

1 读请求 应该是0-2这3台机子都会请求。因为svc设置的是app:mysql 主从并没有区分

2 删除master节点 似乎并不会由slave 升级为master 依旧是重新生成了一个新的mysql-0

8019914403d46a445a4ee9587840e60b.png

测试继续写入数据没有问题 后续进行更多测试

但是slave不会升级成master的话 可能可用性还有待提高 更换为operater 主从用不同的sts来部署可能会好些

参考

mysql

https://my.oschina.net/u/3627276/blog/4528810

nfs storage

https://www.cnblogs.com/panwenbin-logs/p/12196286.html

发表评论

表情：
评论列表 （有 0 条评论，292人围观）

还没有评论，来说两句吧...

相关阅读

    相关 mysql高可用

    中小型企业mysql一般使用什么l高可用方案 传感器是一种检测装置，能感受到被测量的信息，并能将感受到的信息，按一定规律变换成为电信号或其他所需形式的信息输出，以满足信息

    忘是亡心i忘是亡心i/ 0 赞/ 181 阅读

    相关 高可用

    海恩法则：大事都是小事积累导致的 墨菲定律：不好的事，该发的一定会发生 早发现、响应快 发现：概率、范围、处理难度 术 设计 AKF扩展术：x:服务无状态、水平

    忘是亡心i忘是亡心i/ 0 赞/ 370 阅读

    相关 【Mysql】怎么保证高可用

    几种造成主从延迟的情况: 1.主库DML语句并发大,从库qps高 2.从库服务器配置差或者一台服务器上几台从库(资源竞争激烈,特别是io) 3.主库和从库的参数配置

    墨蓝墨蓝/ 0 赞/ 391 阅读

    相关 MySQL高可用

    [MySQL高可用][MySQL] MySQL数据库简单介绍 MySQL作为世界上使用最为广泛的数据库之一，免费是其原因之一。但不可忽略的是它本身的功能的确很强大。随着

    淡淡的烟草味﹌淡淡的烟草味﹌/ 0 赞/ 491 阅读

    相关 高可用

    一、什么是高可用 高可用HA（High Availability）是分布式系统架构设计中必须考虑的因素之一，它通常是指，通过设计减少系统不能提供服务的时间。 假设系统一直能

    本是古典 何须时尚本是古典 何须时尚/ 0 赞/ 668 阅读