【OSS】——阿里云OSS作为k8s存储(pv/pvc)实践

太过爱你忘了你带给我的痛 2023-02-20 06:53 31阅读 0赞

### 一、前言 ###

关于k8s的存储知识，大家可以参考博文[应用存储和持久化数据卷：核心知识][Link 1]和[Kubernetes 存储架构及插件使用][Kubernetes]，我这就不科普了。阿里云官网上的文档都是基于阿里云自己的k8s的，找了半天资料也没有文章好好介绍一下自己通过kubeadm搭建的k8s集群该怎么用阿里云的oss作为pv/pvc使用，今天这篇文章就是我踩坑之后写下来作为记录的。

说明：

*  OSS 数据卷是使用 OSSFS 文件进行挂载的 FUSE 文件系统，适合于读文件场景。例如：读配置文件、视频、图片文件等场景。
 *  OSSFS 不擅长于写文件的应用场景。

### 二、准备工作 ###

1、k8s集群，我是通过kubeadm搭建的三节点集群。搭建步骤参见我的另外一篇博文:[搭建k8s多节点集群][k8s]

NAME    STATUS   ROLES    AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
    node1   Ready    master   51d   v1.16.2   10.53.5.94    <none>        Ubuntu 16.04.1 LTS   4.4.0-127-generic   docker://19.3.4
    node2   Ready    <none>   51d   v1.16.2   10.53.6.185   <none>        Ubuntu 16.04.1 LTS   4.4.0-127-generic   docker://19.3.4
    node3   Ready    <none>   51d   v1.16.2   10.53.7.37    <none>        Ubuntu 16.04.1 LTS   4.4.0-127-generic   docker://19.3.4

2、阿里云oss账号，并创建bucket。这个没什么好说的，参见[阿里云OSS官方文档][OSS]

3、在每台机器上安装阿里云ossfs软件。这个软件必须要安装，因为pv/pvc如果想要用阿里云的oss的话，这是必须的软件。安装步骤参见[阿里云oss安装文档][oss]，我的机器是三节点ubuntu 16.04,下边我就写一下ubuntu 16.04该怎么安装ossfs。

# 下载安装包
    $ wget http://gosspublic.alicdn.com/ossfs/ossfs_1.80.6_ubuntu16.04_amd64.deb
    
    #安装
    $ sudo apt-get update
    $ sudo apt-get install gdebi-core
    #sudo gdebi <your_ossfs_package>
    $ sudo gdebi ossfs_1.80.6_ubuntu16.04_amd64.deb

### 三、yaml文件准备 ###

**1.rbac.yaml**

# This YAML file contains all RBAC objects that are necessary to run external
    # CSI provisioner.
    #
    # In production, each CSI driver deployment has to be customized:
    # - to avoid conflicts, use non-default namespace and different names
    #   for non-namespaced entities like the ClusterRole
    # - decide whether the deployment replicates the external CSI
    #   provisioner, in which case leadership election must be enabled;
    #   this influences the RBAC setup, see below
    
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: admin
      # replace with the same namespace name with plugin
      namespace: kube-system
    
    ---
    kind: ClusterRole
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: alicloud-csi-plugin
    rules:
      - apiGroups: [""]
        resources: ["secrets"]
        verbs: ["get", "list"]
      - apiGroups: [""]
        resources: ["persistentvolumes"]
        verbs: ["get", "list", "watch", "update", "create", "delete"]
      - apiGroups: [""]
        resources: ["persistentvolumeclaims"]
        verbs: ["get", "list", "watch", "update"]
      - apiGroups: ["storage.k8s.io"]
        resources: ["storageclasses"]
        verbs: ["get", "list", "watch"]
      - apiGroups: ["storage.k8s.io"]
        resources: ["csinodes"]
        verbs: ["get", "list", "watch"]
      - apiGroups: [""]
        resources: ["events"]
        verbs: ["get", "list", "watch", "create", "update", "patch"]
      - apiGroups: [""]
        resources: ["endpoints"]
        verbs: ["get", "watch", "list", "delete", "update", "create"]
      - apiGroups: [""]
        resources: ["configmaps"]
        verbs: ["get", "watch", "list", "delete", "update", "create"]
      - apiGroups: [""]
        resources: ["nodes"]
        verbs: ["get", "list", "watch"]
      - apiGroups: ["csi.storage.k8s.io"]
        resources: ["csinodeinfos"]
        verbs: ["get", "list", "watch"]
      - apiGroups: ["storage.k8s.io"]
        resources: ["volumeattachments"]
        verbs: ["get", "list", "watch", "update"]
      - apiGroups: ["snapshot.storage.k8s.io"]
        resources: ["volumesnapshotclasses"]
        verbs: ["get", "list", "watch"]
      - apiGroups: ["snapshot.storage.k8s.io"]
        resources: ["volumesnapshotcontents"]
        verbs: ["create", "get", "list", "watch", "update", "delete"]
      - apiGroups: ["snapshot.storage.k8s.io"]
        resources: ["volumesnapshots"]
        verbs: ["get", "list", "watch", "update"]
      - apiGroups: ["apiextensions.k8s.io"]
        resources: ["customresourcedefinitions"]
        verbs: ["create", "list", "watch", "delete"]
    
    ---
    kind: ClusterRoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: alicloud-csi-plugin
    subjects:
      - kind: ServiceAccount
        name: admin
        namespace: kube-system
    roleRef:
      kind: ClusterRole
      name: alicloud-csi-plugin
      apiGroup: rbac.authorization.k8s.io

**2.oss-plugin.yaml**

apiVersion: storage.k8s.io/v1beta1
    kind: CSIDriver
    metadata:
      name: ossplugin.csi.alibabacloud.com
    spec:
      attachRequired: false
    ---
    # This YAML defines all API objects to create RBAC roles for csi node plugin.
    kind: DaemonSet
    apiVersion: apps/v1
    metadata:
      name: csi-ossplugin
      namespace: kube-system
    spec:
      selector:
        matchLabels:
          app: csi-ossplugin
      template:
        metadata:
          labels:
            app: csi-ossplugin
        spec:
          tolerations:
          - operator: Exists
          priorityClassName: system-node-critical
          serviceAccount: admin
          hostNetwork: true
          hostPID: true
          containers:
          - name: driver-registrar
            image: registry.cn-hangzhou.aliyuncs.com/acs/csi-node-driver-registrar:v1.1.0
            imagePullPolicy: Always
            lifecycle:
              preStop:
                exec:
                  command: ["/bin/sh", "-c", "rm -rf /registration/ossplugin.csi.alibabacloud.com /registration/ossplugin.csi.alibabacloud.com-reg.sock"]
            args:
            - "--v=5"
            - "--csi-address=/var/lib/kubelet/plugins/ossplugin.csi.alibabacloud.com/csi.sock"
            - "--kubelet-registration-path=/var/lib/kubelet/plugins/ossplugin.csi.alibabacloud.com/csi.sock"
            env:
            - name: KUBE_NODE_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: spec.nodeName
            volumeMounts:
            - name: kubelet-dir
              mountPath: /var/lib/kubelet/
            - name: registration-dir
              mountPath: /registration
    
          - name: csi-ossplugin
            securityContext:
              privileged: true
              capabilities:
                add: ["SYS_ADMIN"]
              allowPrivilegeEscalation: true
            image: registry.cn-hangzhou.aliyuncs.com/acs/csi-plugin:v1.14.8.32-c77e277b-aliyun
            imagePullPolicy: "Always"
            args:
            - "--endpoint=$(CSI_ENDPOINT)"
            - "--v=5"
            - "--driver=ossplugin.csi.alibabacloud.com"
            - "--nodeid=$(KUBE_NODE_NAME)"
            env:
            - name: CSI_ENDPOINT
              value: unix://var/lib/kubelet/plugins/ossplugin.csi.alibabacloud.com/csi.sock
            - name: KUBE_NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            volumeMounts:
            - name: kubelet-dir
              mountPath: /var/lib/kubelet/
              mountPropagation: "Bidirectional"
            - name: etc
              mountPath: /host/etc
            - mountPath: /var/log/
              name: host-log
            - mountPath: /host/usr/
              name: flexvolumedir
          volumes:
          - name: kubelet-dir
            hostPath:
              path: /var/lib/kubelet/
              type: Directory
          - name: registration-dir
            hostPath:
              path: /var/lib/kubelet/plugins_registry
              type: DirectoryOrCreate
          - name: etc
            hostPath:
              path: /etc
          - name: flexvolumedir
            hostPath:
              path: /usr/
          - name: host-log
            hostPath:
              path: /var/log/
      updateStrategy:
        type: RollingUpdate

**3.pv.yaml**

apiVersion: v1
    kind: PersistentVolume
    metadata:
      name: oss-csi-pv
      labels:
        alicloud-pvname: oss-csi-pv
    spec:
      capacity:
        storage: 5Gi
      accessModes:
        - ReadWriteOnce
      persistentVolumeReclaimPolicy: Retain
      csi:
        driver: ossplugin.csi.alibabacloud.com
        # set volumeHandle same value pv name
        volumeHandle: oss-csi-pv
        volumeAttributes:
          bucket: "*****" #重要
          url: "******" #重要
          otherOpts: "-o max_stat_cache_size=0 -o allow_other"
          akId: "****" #重要
          akSecret: "*******" #重要
          path: "/"

说明：

*  bucket：目前只支持挂载Bucket，不支持挂载Bucket下面的子目录或文件。
 *  url：OSS endpoint，挂载OSS的接入域名，挂载节点和bucket相同region时，可使用内网地址。
 *  akId：用户的access id值。
 *  akSecret：用户的access secret值。
 *  otherOpts：挂载OSS时支持定制化参数输入，格式为：`-o *** -o ***`。

** 4.pvc.yaml**

apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: oss-pvc
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 5Gi
      selector:
        matchLabels:
          alicloud-pvname: oss-csi-pv

**5. deploy.yaml**

apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: deployment-oss
      labels:
        app: nginx
    spec:
      selector:
        matchLabels:
          app: nginx
      template:
        metadata:
          labels:
            app: nginx
        spec:
          containers:
          - name: nginx
            image: nginx:1.7.9
            ports:
            - containerPort: 80
            volumeMounts:
              - name: oss-pvc
                mountPath: "/data"
          volumes:
            - name: oss-pvc
              persistentVolumeClaim:
                claimName: oss-pvc

### 四、部署 ###

至此，部署阶段就比较简单了。按照上边准备的文件顺序知行yaml文件：

#创建rbac权限
    $ kubectl create -f ./rbac.yaml 
    serviceaccount/admin created
    clusterrole.rbac.authorization.k8s.io/alicloud-csi-plugin created
    clusterrolebinding.rbac.authorization.k8s.io/alicloud-csi-plugin created
    
    
    #创建oss-plugin
    $ kubectl create -f ./oss-plugin.yaml
    
    #检查创建情况
    $ kubectl get pod -n kube-system | grep csi-oss
    kube-system             csi-ossplugin-9jdhw                                  2/2     Running             0          55m
    kube-system             csi-ossplugin-f7n5f                                  2/2     Running             0          55m
    kube-system             csi-ossplugin-vgkcp                                  2/2     Running             0          55m
    
    #查验CSIDriver安装情况
    $ kubectl get CSIDriver
    NAME                             CREATED AT
    ossplugin.csi.alibabacloud.com   2020-06-23T14:48:18Z
    
    #创建pv
    $ kubectl create -f ./pv.yaml
    
    #创建pvc
    $ kubectl create -f ./pvc.yaml
    
    #检验一下阿里云oss是否可以成功挂载到k8s集群中做pv使用
    $ kubectl create -f ./deploy.yaml

### 五、验证 ###

$ kubectl get pod
    NAME                              READY   STATUS              RESTARTS   AGE
    deployment-oss-795894886d-lhpsx   1/1     Running             0          11h
    
    #pod成功后通过kubectl exec 进入到pod中，你能看到你账号下bucket里边的所有文件。样例如下：
    $ kubectl exec -it deployment-oss-795894886d-lhpsx -- sh
    $ ls
    bin  boot  data  dev  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  selinux  srv  sys	tmp  usr  var
    $ cd data	
    $ ls
    osstest

### 六、参考文章 ###

1、[K8S有状态服务-OSS存储使用最佳实践][K8S_-OSS]；

2、[阿里云oss CSI安装步骤][oss CSI]

3、[阿里云oss官方文档][oss 1]

[Link 1]: https://www.cnblogs.com/passzhang/p/12542607.html
[Kubernetes]: https://www.cnblogs.com/passzhang/p/12563690.html
[k8s]: https://blog.csdn.net/weixin_40449300/article/details/104758475
[OSS]: https://help.aliyun.com/document_detail/31883.html
[oss]: https://partners-intl.aliyun.com/help/doc-detail/153892.htm?spm=a2c63.p38356.b99.184.212865d08TydXr
[K8S_-OSS]: https://developer.aliyun.com/article/640212
[oss CSI]: https://github.com/kubernetes-sigs/alibaba-cloud-csi-driver/blob/master/docs/oss.md
[oss 1]: https://help.aliyun.com/document_detail/31883.html?spm=a2c4g.11186623.6.580.24106ab31sFaxK